I think I may have sorted this out... I had:

authority: "https://login.microsoftonline.com/common",

And I've changed it to include the tenant / directory id:

authority: "https://login.microsoftonline.com/XXXXX-XXXXX-XXXX-XXXXX",

Not sure why having it set to common vs. the tenant / directory id would ignore the other settings and allow anyone to login, but that's what it's looking like.