Solved the issue. I was passing access token instead of identity token.

LogoutRequest logoutRequest = new LogoutRequest
{
    IdTokenHint = token // has to be identity token
};
LogoutResult? logoutResult = await OidcClient.LogoutAsync(logoutRequest);