79707578

Date: 2025-07-19 21:28:38
Score: 1
Natty:
Report link

You need not do this in double precision: single precision is enough! The trouble is what you're using the precision for, and by trying to represent cosines near 1 rather than near 0 you're throwing away all the precision available to you. Navigators in days of yoreβ€”with much less precision than modern IEEE 754 binary32 single precision!β€”had the same problem, and what did they do?

Let's write this as a formula without all the unit conversionsβ€”you have latitudes πœ‘β‚ and πœ‘β‚‚ and longitudes πœ†β‚ and πœ†β‚‚ and you want the great-circle angle πœƒ, which you've decided to approach with the relation (probably derived from converting the spherical law of cosines naively to a longitude/latitude coordinate system):

cos πœƒ = sin πœ‘β‚ sin πœ‘β‚‚ + cos πœ‘β‚ cos πœ‘β‚‚ cos(πœ†β‚‚ βˆ’ πœ†β‚).

The crux of the problem is that you're trying to find an angle whose cosine is very close to 1, and you're doing it by computing that cosine. In your example, sin πœ‘β‚ sin πœ‘β‚‚ + cos πœ‘β‚ cos πœ‘β‚‚ cos(πœ†β‚‚ βˆ’ πœ†β‚) comes out to roughly 0.999999937 = 1 βˆ’ 6.3Γ—10⁻⁸, but the single-precision floating-point numbers in [Β½, 1] can only discern increments of about 6Γ—10⁻⁸, so you've essentially wasted all your precision on those leading nines. Navigators in days of yore didn't even have eight digits (for example, the 1913 tables of E.R. Hedrick and the 1800 tables of Josef de Mendoza y RΓ­os provide only around five digits), so how did they do it?

You probably grew up on the three trig functions sine, cosine, and tangent if you're much less than a century old, but there are others like haversine (hav πœƒ = Β½ (1 βˆ’ cos πœƒ)) and exsecant (exsec πœƒ = sec πœƒ βˆ’ 1 = 1/cos πœƒ βˆ’ 1)β€”invented not to torture students in school, but because they made calculations feasible with limited precision! The trick is not to compute the cosine 1 βˆ’ 𝛿 for very small 𝛿, but instead compute the versine 𝛿, or the haversine ½𝛿, so you can use the full precision available to you to represent it.

The venerable haversine formula relates the latitudes πœ‘β‚ and πœ‘β‚‚ and longitudes πœ†β‚ and πœ†β‚‚ to the great-circle angle πœƒ by:

hav πœƒ = hav(πœ‘β‚‚ βˆ’ πœ‘β‚) + cos πœ‘β‚ cos πœ‘β‚‚ hav(πœ†β‚‚ βˆ’ πœ†β‚).

This is better for nearby angles because for small angle differences, haversine preserves most of the precision of the input to return a result near 0, while cosine throws it away to return a result near 1.

Of course, the Android math library might be missing a haversine function (pity!), but you can recover it from the trigonometric identity:

hav πœƒ = Β½ (1 βˆ’ cos πœƒ) = sinΒ²(πœƒ/2).

You can always divide by two exactly in floating-point arithmetic (unless it underflows), and squaring won't amplify the error much, so the formula sinΒ²(πœƒ/2) will work reliably, and you can recover the inverse archav π‘₯ = 2 arcsin(√π‘₯), using functions that probably are in the Android math library. And this identityβ€”together with coordinate transformation between longitude/latitude and spherical triangle anglesβ€”is how you can derive the haversine formula from (e.g.) the spherical law of cosines, as the Wikipedia article shows.

Once you use the haversine formula, even if you compute everything in single precision, you get within about 30cm of the exact result in your example, or <0.014% relative errorβ€”which is smaller than the error of about 60cm arising from using a spherical approximation to the oblate spheroid of Earth!

Reasons:
  • Whitelisted phrase (-1.5): you can use
  • Contains signature (1):
  • Long answer (-1):
  • No code block (0.5):
  • Contains question mark (0.5):
  • Unregistered user (0.5):
  • Low reputation (1):
Posted by: Josef de Mendoza y RΓ­os