As of 2024, the Natim's answer does not seem to work. I ran

pip install torch==2.5.0
pip install torchvision

and this automatically upgraded torch to 2.5.1. The correct way now is to pass the --no-deps argument to pip:

pip install torch==2.5.0
pip install torchvision --no-deps