In the end, I just used a workaround using python3 -c 'import torch;print(torch.utils.cmake_prefix_path)', linking manually seems to be not possible.
python3 -c 'import torch;print(torch.utils.cmake_prefix_path)'