I'm trying to display a custom SwiftUI view inside ARKit in a 3D scene as a texture on a plane. My view contains a blurred image and a translucent effect, but I am facing issues with getting the SwiftUI view to show correctly in the AR scene. Here is what I have so far:

I created a custom SwiftUI view, CustomBlurTranslucentView, which includes an image and some opacity and blur effects. I rendered the SwiftUI view to an image (UIImage), then converted it into a texture for the ARView. I am using this rendered image as a texture on a 3D plane entity in ARKit. However, I am unable to get it to display as intended and have some concerns regarding the image rendering part, since the image is generated from a UIView and not an actual ImageView component. Here’s the code I’m using:

// Anchor and plane creation
let anchor = AnchorEntity(world: [0, 0, -1])

// Create a custom SwiftUI view and convert it to a texture
let swiftUIView = CustomBlurTranslucentView()
let modelEntity = createPlaneEntity(from: swiftUIView, size: CGSize(width: 200, height: 100))

// Add the model entity to the anchor
anchor.addChild(modelEntity)

// Add the anchor to the AR scene
arView.scene.addAnchor(anchor)

// Function to create a plane entity with the custom view texture
private func createPlaneEntity(from view: some View, size: CGSize) -> ModelEntity {
    let image = renderImage(from: view, size: size) // Render the SwiftUI view to an image
    let adjustedImage = image.adjustOpacity(opacity: 0.4) // Adjust opacity of the rendered image
    let texture = try! TextureResource(image: adjustedImage.cgImage!, options: .init(semantic: .none))
    
    var material = UnlitMaterial()
    material.baseColor = MaterialColorParameter.texture(texture)

    let planeMesh = MeshResource.generatePlane(width: Float(size.width / 1000), height: Float(size.height / 1000))
    let planeEntity = ModelEntity(mesh: planeMesh, materials: [material])
    
    return planeEntity
}

// Function to render a SwiftUI view as a UIImage
private func renderImage(from view: some View, size: CGSize) -> UIImage {
    let hostingController = UIHostingController(rootView: view)
    hostingController.view.bounds = CGRect(origin: .zero, size: size)
    hostingController.view.backgroundColor = .clear
    
    let renderer = UIGraphicsImageRenderer(size: size)
    return renderer.image { _ in
        hostingController.view.drawHierarchy(in: hostingController.view.bounds, afterScreenUpdates: true)
    }
}

// Custom SwiftUI view with blur and opacity
struct CustomBlurTranslucentView: View {
    var body: some View {
        ZStack {
            Image("photo5") // Replace with your image
                .resizable()
                .scaledToFill()
                .edgesIgnoringSafeArea(.all)
                .blur(radius: 15)
                .opacity(0.3)
            
            VStack {
                Text("Custom Under Effect")
                    .font(.headline)
                    .foregroundColor(.white)
                    .padding()
            }
        }
    }
}

In ARKit with SwiftUI, it's a bit tricky to display a custom SwiftUI view as a texture on a 3D object, since ARKit generally works with textures created from UIImage objects, and not SwiftUI views directly. However, you can work around this by rendering the SwiftUI view to a UIImage first, and then using that image as a texture for your AR model.

Here’s how you can approach this:

Rendering SwiftUI view to UIImage: We use the UIHostingController to host the SwiftUI view inside a UIView and then render that view as an image using UIGraphicsImageRenderer. The rendered image can be used as a texture in ARKit.

Applying opacity and blur: The custom SwiftUI view, CustomBlurTranslucentView, has an image with blur and opacity effects applied. After rendering it as an image, we can adjust the opacity if needed before applying it as a texture.

Texture application in ARKit: Once the image is rendered and processed, we convert it to a TextureResource and assign it to the material of the ModelEntity, which is then placed on a plane mesh in ARKit.

Key Notes: UIImage Rendering: The image here is not directly from an ImageView, but from a UIView (UIHostingController wrapping a SwiftUI view). This means you have full flexibility to create complex views like the one with a blur and opacity effect before rendering it. Performance Consideration: Rendering images in this way for each frame can be computationally expensive. For complex views or frequent updates, you might want to cache the result or limit the updates. UnlitMaterial: This material is used to avoid any lighting effects and display the image as-is, which works well for UI elements like this.