Okay holy shit so I just figured this out!

So when launching multiple projects there's this new setting called "Debug Target" seen here:

The two first in that list is the project itself, and IIS Express. Since it's blank it seems to default to starting IIS Express, and when I remove the entries from my launchsettings I can start it without. But if I set the "Debug Target" to my project, it never adds IIS Express!

I think I never noticed it because I've only really opened that menu like once before "Debug Target" was a thing!