To answer your question very directly, the distance is always measured as the distance between the world camera position and the origin of the object. Nothing else should have any influence, did you double check the origins? Other than that I'm no help.

I've had a similar performance issue recently with meshes but the unfortunate truth is that other than LOD management, there isn't much to do. Three.js likes less objects in general so try to merge objects where you can but I don't think that applies to what you want to do.