So, I really really really hate this... When debugging we found that we needed to set the baseURL as our Keycloak client name. Then it worked.

After this we've upgraded our production environment and get this, the stupid problem didn't occur... Everything is still the same as it was before.

I have no idea what went wrong or why it works now but it does...