This took me way longer than I should have found.

You NEED to set transform-origin: 0 0; on the container element for the map.

EVEN if you don't have a transform value set, for SOME reason, setting this explicitly fixed the problem

Stay golden pony boy