It turns out this is a bug in WindowsAppSdk as mentioned by @Domo and @AndrewKeepCoding.
A link to the existing bug report can be found here: https://github.com/microsoft/microsoft-ui-xaml/issues/10009
Reverting to WindowsAppSdk 1.5 seems to solve the issue though.