Ok after some more research I found that this might have been fixed with CommunityToolkit.Maui
version 11.0.0
https://github.com/CommunityToolkit/Maui/releases/tag/11.0.0
or
https://github.com/CommunityToolkit/Maui/pull/2309
This requires .NET 9
I am currently still bringing my .NET 8 Project to .NET 9 so I could not test this