I am answering my own question for people who would face the same issue in future. It is not related about IdentityServer4 configurations or axa-fr/react-oidc library.

Error caused from calling app.UseIdentityServer() in wrong order. I can't tell why this happens but if you call anything before app.UseIdentityServer() it causes this kind of weird problem.

var app = builder.Build();
app.UseIdentityServer(); ->Call this first. 
app.UseStaticFiles();