Is there a reason you can't use the WCAG contrast ratio definition? Assuming your colours are within sRGB, you can calculate the relative luminance of the background colour from RGB values. You could check which foreground colour has a greater contrast with the background colour, and use that.

(It's the approach I use for the RGB text at the bottom of my colour picker)

Note that HSB's Brightness and HSL's Lightness don't represent what the eye perceives, they are just a reshaping of RGB.