In my case, the reason for HTTP 500 in the callback was due to the Enterprise Application not having permissions granted. After I granted the permissions and restarted the app (and it took some time too), callback started to work.

You can grant the permissions here: Azure Entra ID > Enterprise Applications > Clear filter for Application Type > (Your App) > Permissions > Click "Grant admin consent".