I've tried all suggestions, nothing helped. Than it struck me. I have an external screen that I am using with is regular density, while the built in screen of my MacBook is indeed retina.

Then I grabbed the Safari window, put it on the retina screen, and boom, every font became clear and just the weight as it was supposed to be.

So what happens is that Safari renders fonts different for high DPI screens and regular screens. But when you have both a high-dpi retina screen and a regular screen attached at the same time, Safari activates the retina font rendering, but the retina font renderer somehow messes up the antialiasing when the actual rendering happens on a regular screen and suddenly every font looks just way too bold.

Moral of the story: don't try to fix it in CSS, because it is a lower level issue with how Safari and macOS handles the font rendering. In other words it is an edge case apple bug that happens when you use different resolution screens together.