Obviously right after I created this question:
I find this question. The answer I came up with is this:
A workaround to fix this without changing anything else is to insert the following in your WebClient project file:
<PropertyGroup>
<_ExtraTrimmerArgs>--keep-metadata parametername</_ExtraTrimmerArgs>
</PropertyGroup>
The real issue should be fixed in https://github.com/dotnet/runtime/issues/81979