You could set the environment variable PIPX_DEFAULT_PYTHON to use python3.11 (use pipx environment to list the available environment variables)

e.g on macOS on Apple silicon

export PIPX_DEFAULT_PYTHON=/opt/homebrew/bin/python3.11