I've setup Credential Manager through Firebase and the steps that helped me solve similar issues in the past were:

1. Make sure you're signed into your Google account on the device you're testing
2. Make sure that you've added your signing certificate fingerprints in the Firebase Project settings (you have to use SHA-1, SHA-256 doesn't work)
3. Make sure that in the Google Cloude Console, under APIs & Services > Credentials > OAuth 2.0 Client IDs the appropriate clients have been added and that the fingerprints match (in case of using Firebase, the clients will be named "Android client for com.example.appId (auto created by Google Service)".

The code is rarely the issue (I can't spot anything wrong with the code you provided), but it's easy to mess up the setup.