The solution is that you need to additionally install the lima machine and start a qemu driven machine instead of the default rusetta machine and then everything will go smoothly. https://github.com/containers/podman/discussions/23201#discussioncomment-11094909