May you have tried to set BAZELISK_HOME to a path that suits your needs, like the repo root? Be aware also the downloads during the setup of bazel will be placed there.

This can be achieved by using a .bazeliskrc file in the root of your repo

BAZELISK_HOME=./.repo_local_cache/bazelisk