I found an alternative solution that does not directly answer my question but still provides the same behavior I needed. Perhaps this is also me not fully understanding how authorization and custom domains works earlier.

I came across this video from a few years ago that explicitly mentioned making a second site in your Firebase project and linking a custom auth domain to it. This is what I did:

1. Inside my-project-staging Firebase project, I created a new site on the Hosting page and called it my-project-staging-auth. There is no code or custom app deployed to this project.
2. I added a custom domain to my-project-staging-auth called auth.staging.my-custom-domain.app
3. Added DNS config recommended by Firebase when going through the domain connection wizard.
4. Added https://auth.staging.my-custom-domain/__/auth/handlers to my Google client ID
5. Updated Firebase config on my app to use auth.staging.my-custom-domain for its authDomain property

This resolved the popup being handled by my main app. I'm still open to other alternatives if anyone has any, but this setup works well for my needs. Perhaps another way is to exclude certain routes in my React app so that Firebase handles them...