I am coming at this from a math/physics angle rather than programming, so forgive me if I am focusing on the wrong thing, but I need some clarification on what exactly you are trying to transform here. Are you trying to preserve the ratio of the scale length to the point-scale distance?

By point-scale distance, I mean the length of the perpendicular line/arc from the point to the closest point on the scale. My understanding is that you are trying to satisfy:

It's known from differential geometry (see Theorema Egregium) that you cannot project from a sphere to a plane while preserving both shapes (or angles) and areas, which I suspect is very likely to be the root cause of your problems. I am not really sure if what you are trying to achieve can be done by only preserving one or the other or if you're trying to do something impossible, but it's probably worthwhile to actually carry out the math in 3D rather than a 2D projection. The (two ends of the) scale and the point together form a triangle on the Earth's surface, so you're really trying to transform (rotate, scale, translate) a spherical triangle, which I am not sure would work. Spherical trigonometry might help you here.

Likely why your 2D approximation does not work

The transformations you're composing are:

• translation (cyan -> magenta)

• scaling (ends up in green point)

• rotation (takes the point in question from green point to pink point)

Using regular 2D/3D cartesian coordinates, these operations do not commute. Chiefly, the reason for this is that translation is not a linear transformation in Cartesian coordinates (Tiny proof sketch: The 0 vector does not map back to 0 under a translation). In other words, you'll get a different result if you change the order. In general, you'll apparently need to use homogenous coordinates, under which translation is linear, to avoid this problem; however, you might end up working with points/lines/areas off the surface of the Earth if you directly convert Cartesian coordinates to homogenous coordinates in this case. I cannot guess offhand if the approximation would work better with homogenous coordinates or not.