You're doing what many start with: RGB/HSV distance. The issue you're hitting is a classic — perceived color and numberic proximity in RGB...

Color math ain't mathin...

A couple breadcrumbs:

There's a color space with a name that starts with L and ends in ab — look into it.

Once there, don't just calculate raw distance — search for a metric named deltaE, specifically the one from the 2000s.

If your algorithm thinks dark blue is silver, you might want to investigate how saturation and luminance interact in low-light colors.

You're not far off. Just need to switch dimensions (literally).