Questions: Why would this CSP issue appear only in production?

Because on your dev env you either do not have a CSP specification at all, or the domain was already handled.

What is the best way to configure the CSP to allow this token request without compromising security?

I will forget about "best" and will answer the "how". CSP whitelists domains you trust. So if you trust ogin.microsoftonline.com - and you trust it with the login -, then whitelist it in CSP.

Is explicitly setting connect-src in the CSP header sufficient to fix this?

It could be. Set it, whitelist the domain(s) that you trust and see whether there are further issues.

Could a CDN or production web server (e.g., nginx, Apache, etc.) be altering or overriding the CSP?

In some systems they are overriden. If you are unsure, either ask someone who knows or look into the configuration.

Any help or experience with similar production-only CSP issues would be greatly appreciated!

You could do well to reproduce the issue locally, that is, have the same (wrong) CSP on your local temporarily to reproduce the issue and then fix it on your local. Once you succeed, it should work on live too. BUT: back up your settings, especially the CSP directives from live before you do any change.