You might want to look into HSV (Hue, Saturation, Value) color space or similar. In those spaces you might be able to filter out the gray pixels (e.g., low saturation and low value).