What I get is you are facing a limitation in SSMS where the default annotation font size is not adjustable through the "Fonts and Colors" settings. It can be due to an application that is properly respecting your configurations if you are using Windows 11 with scaling options as enabled.

I recommend you check your display setting for any scaling issue, and infrom Microsoft about this potential bug. Creating new diagrams can help, but if the problem remains the same, you can explore SSMS updates or alternative ways to customize your setup.

Regards