After years of pain I'm replying to my old question after I found the definitive solution, namely get rid of conda.

I have chosen Pixi because it can use conda-forge as channel for the packages as well as pypi, check it out.

So far I don't see one reason to get back to conda while I can list so many why I migrated.

So uninstall conda, python and everything... then install Pixi and just "pixi init" + "pixi add python=<desired-version>" etc