Anyway, thank you for the help. I've found the root cause, in case anyone got similar issue.

So, notice that I'm only assign the cameraNode to sceneView.pointOfView, turns out you should also insert the camera to the rootNode.

As for what does it's only work after I tapped the screen, I had no idea.

sceneView.pointOfView = cameraNode
sceneView.scene.rootNode.addChild(cameraNode)