I couldn't reproduce the images not loading issue, but if you are having trouble with the view resizing, have you considered delegating the same frame size (particularly the height) to the text as well?

As in,

if isImageVisible {
            Image(imageName)
                .resizable()
                .scaledToFit()
                .frame(width: 100, height: 100)
                .background(Color.gray.opacity(0.2))
        } else {
            Text("Image is hidden")
                .frame(width: 200, height: 100)
        }