They do hit the IdP (Identity Provider) every time you pass a token. How else can you validate the user? In the JWT header you have a field in which you can pass a public key and then you fetch the public key and then you validate against the same public key. The public key will be stored in the domain controlled by Microsoft.

Of course if someone changes the header field then the JWT will not be a valid one.

You fetch the key using JWKS.