I used PyTorch-suggested pip3 instead of conda and had no trouble on a Windows 11 machine.

First of all, even though conda is a package manager, when you activate an environment and conda install python, it would install pip as well. So, after creating and activating an environment, you would go to https://pytorch.org/get-started/locally/, do your selection, and copy the command recommended there. Lastly, while still in a conda-activated environment, paste and issue the recommended command, which uses pip3 instead of conda.

I tested the above in a PowerShell terminal and in a terminal in VS code.