It seems like a bug in cmake https://gitlab.kitware.com/cmake/cmake/-/issues/24163 . When using `PRIVATE`, outer scope will not link torch lib and thus will not trigger the problem.

Updating to newest version of cmake will solve it.