I added
set -o vi
to ~/.zshrc and it worked a treat. I hit esc and can navigate and edit with vi keys. Hit i and I'm back in typing mode.