After a long day (and not the first) of searching I finally arrived at a solution. @Jason Pan is correct. During the sign-out process you have to call the Microsoft logout URI. My code ended up something like the following:

accountGroup.MapPost("/Logout", async (
    ClaimsPrincipal user,
    SignInManager<ApplicationUser> signInManager,
    [FromForm] string returnUrl,
    HttpContext httpcontext) =>
{
    // Clear the existing browser cookie
    await signInManager.SignOutAsync();
    //If there isn't a return URL, redirect to the login page
    returnUrl = string.IsNullOrEmpty(returnUrl) ? "/Account/Login" : returnUrl;

    if (user.Claims.Any(c => c.ToString().Contains("Microsoft")))
    {
        //If the user is authenticated with a Microsoft account, redirect to the Microsoft logout page
        var redirectUrl = $@"{httpcontext.Request.Scheme}://{httpcontext.Request.Host}{returnUrl}";
        string url = $@"https://login.microsoftonline.com/common/oauth2/v2.0/logout?post_logout_redirect_uri={redirectUrl}";
        return TypedResults.Redirect(url);
    }
    //Otherwise, redirect to the return URL
    return TypedResults.LocalRedirect(returnUrl);
});

If you have a single MS account logged in (for that browser) it will sign you out and redirect to the provided post_logout_redirect_uri. If you have multiple MS log ins it will ask which one you want to sign out of. In this case I find it does NOT reliably work (i.e. your still signed in w/Microsoft). But I've always had issues with a browser dealing with multiple MS logins.

I don't know if the order of sign out is important or not. If you want to call the MS log out URI first then you'll have to do the signInManager.SignoutAsync() on the callback URI.

I think there is a way to configure this when setting up the external provider but haven't found the right syntax yet. For now this works for my prototype.

It's also worth noting that EACH external provider will have their own logout URI. Somewhere a long time ago I found one for Github. Why oh why do they make it so hard to find? I guess you can check in but never leave :)