Not sure if this is going to help you but if the map is static you could render it using MKMapSnapshotter and display the image instead. That helps. I do this in my app as well.

One other idea could be to try UIKit's MKMapView instead of SwiftUI's Map. I haven't tried that myself.