I actually found what it is. I was regenerating the OIDC application registration on every restart. In the past with OpenIdDict pre version 6 this would work, but apparantly with the version 6 this means that also all stored tokens are invalidated.