After doing some more research I realized that I was actually dealing with 2 different API's. The first one is my custom API and the second is the Microsoft Graph API. So, essentially it's one token per API. So, here is what I did:

1. Get the access token from the SPA.

2. Use that access token to request another token from the API authority (openId, etc..), being sure to request the scopes needed for Microsft Graph. It's best to use the default, which get's all scopes available - "https://graph.microsoft.com/.default"

3. Pass the new token to a Microsoft Graph endpoint, such as https://graph.microsoft.com/v1.0/me

4. That will get you a json string response.