It turned out that I was missing Asp Net Core Hosting Bundle:

"The .NET Core Hosting bundle is an installer for the .NET Core Runtime and the ASP.NET Core Module. The bundle allows ASP.NET Core apps to run with IIS."

https://learn.microsoft.com/en-us/aspnet/core/host-and-deploy/iis/hosting-bundle?view=aspnetcore-9.0

Once I installed that and restarted IIS suddenly everything starts to work!