I had to add the user email from the Google Cloud service account to the credential, like this:

credential = GoogleCredential.FromStream(stream)
    .CreateScoped(scopes)
    .CreateWithUser("[email protected]");