For anyone in 2025:
The only solution really working is to log out of the account, then press Option+Ctr+E (to delete the cache on safari) and then re-login.

After that you should be able to just download the .p8 file as normal.