As always it is about the correct permissions. The documentation does not show all the permissions required. It was required to add the .Shared versions as well.

Got it to finally work by adding the .Shared consent. so not only Mail.Read and Mail.ReadBasic, but also Mail.Read.Shared and Mail.ReadBasic.Shared is required.