Following below solved the issue,

1. Created a jks file with public and private key.
2. Returned JwkSource by setting a common kid.

Even in clustered environment, on refresh of jwks uri returns the same value and the intermittent issue is no more observed.