After reviewing my code, I realize that I had an unused parent widget which was causing the issue.

GestureDetector(
    onTap: () => FocusScope.of(context).unfocus(),
    ...
        // GoogleSignInButton() is placed as a children of GestureDetector
    ...
)

By removing the GestureDetector, it no longer rebuilds on tapping/unfocus, thus the twitching is gone! Thank you @Niyam Prassanna Kunder for the heads up.