Turns out this was a compiler bug in the version of Clang I was using.

You can see in this godbolt example how the minimally reproducible example compiles in Clang 20.1.0 but not Clang 19.1.10.

The issue was caused by having the code contained in a template inside a module.