(Resend; list post size limit reached)
The question is whether you consider the trailing 1 bits in the high value to be limited to the 34 (or 30) bits that are available, or whether you let the 1 bits continue indefinitely. The +1 favours the latter interpretation (and avoids any 4mm gaps).
Ultimately, this is rather academic. It's less than 4mm (in both cases). Given the other limitations of the representation, that's nothing. Let's just add the 1 to the mask and keep everything nice and simple.
highvalue = (value | mask) + 1
(or highvalue = lowvalue + scale)
(Single precision floating point will likely lose this detail, but doubles have space aplenty :)
--Martin
From: Richard L. Barnes [mailto:rbarnes@bbn.com]
Sent: Thursday, 11 February 2010 10:37 AM
To: Thomson, Martin
Cc: Bernard Aboba; geopriv@ietf.org
Subject: Re: [Geopriv] Proposed text for 3825bis
Are you referring to the quantization error?
highvalue = value | mask
versus:
highvalue = ( value | mask ) + 1 (== lowvalue + scale)
(the all-0xFF high value vs. the next low value)
If that's the concern, then we might as well just adjust the fixed-point version to use the "+1" version.
I guess I'll agree that you're unlikely to get errors simply due to the fact that you're doing floating-point arithmetic in this case.
--Richard
…
