The solution can be found at

JWT bearer token Authorization not working asp net core web api

Further down in Program.cs I had

// This is the correct order
app.UseAuthentication();
app.UseAuthorization();

In the wrong order.