I assume validating the invitation code before user sign up and potentially preventing it on the client side is not an option. Perhaps you need to know their email address first, correct?

If that's the case, you could go one of the following routes:

1. You can indeed block user sign-up https://firebase.google.com/docs/auth/extend-with-blocking-functions?gen=1st. Here you would need to store the invitation code beforehand with some way of matching it with the user later (for ex. using Anonymous authentication https://firebase.google.com/docs/auth/web/anonymous-auth) and validate it inside beforeCreate cloud function.
2. Arguably a simpler solution - you could keep the current flow - user signs up with Google > you verify their invitation code > if it's invalid you immediately delete their account on the back-end (using Firebase Admin SDK) and in the client app you automatically sign them out. You could combine this with custom user claims which would be set on the back-end only when invitation code is validated to enable full access to your app (to stop attackers who might avoid the code validation on the client)