I know this is pretty late answer but if you're targeting iOS 15.0 or above
image.preparingForDisplay()!
This does the magic I needed, for this kind of problem.