While Zhang's answer would likely work fine using that library, I'm using AspNetCore3 due to being pointed in that direction via the current tutorial and guides. I did some digging into the AspNetCore3 source code and managed to put together a solution following what it does to manage its AccessTokens.

Here's a link to the source file and the relevant line already marked https://github.com/googleapis/google-api-dotnet-client/blob/main/Src/Support/Google.Apis.Auth.AspNetCore3/GoogleAuthProvider.cs#L62 With that it's fairly straightforward to persist what you need.