This issue is solved by running the published executable as Admin. My visual studio always runs as admin, turns out it makes a difference.
I am not sure why it matters, maybe windows defender by default scanning the executable while running that makes it slower, or like Guru Stron said that it has anything to do with DOTNET_TC_QuickJitForLoops, but I haven't got time to test it any further.
Maybe when I have enough time to test, I will update my answer.
For now, I will close this issue.