The nonce is in the ID token, not in the refresh token.
nonce
As it fails on the ID token validation, I guess an ID token is returned upon the Successful Refresh Response, right?
Successful Refresh Response
Does this ID token contains a nonce?