The user info endpoint was incorrect. The correct one is shown below, and it works fine after the change.
Before - User info endpoint: https://login.microsoftonline.com/xxxxxxxxxxxxxxxxxx/.well-known/openid-configuration
Now - userinfo_endpoint":"https://graph.microsoft.com/oidc/userinfo