In my case it was a mistake in the .sln file. There was a mapping from x64 to Win32 like this:

{29B3DBB1-C22B-4366-B257-AFA436F24871}.Release|x64.ActiveCfg = Release|Win32

Which needs to be

{29B3DBB1-C22B-4366-B257-AFA436F24871}.Release|x64.ActiveCfg = Release|x64

This was caused by me manually editing those files and making mistakes ... I probably shouldn't do that.