You should always use a virtual env with python especially for versions >= 3.12.
If it doesn't work even with a virtual env i recommend trying poetry. If it still fails to build even with poetry you should probably switch to a lesser version like 3.11