I have no idea what changed, but after a few hours of rebuild attempts and recreating git history it just started working. There were no changes to code, dependencies or solution/project files.

The only explanation I can think of is that dotnet has some per-project-guid cache somewhere that cleared at some point