In my specific case, the issue came down to network routing.

I investigated the azure app configuration again more in-depth after I posted this question and some of the network configuration. It turns out that I had some network restrictions in place on my identity provider and needed to add this app to the vnet. After doing this the app is now able to successfully go through the authentication flow!