Answer from @Axeman solved this for me:

stat_sf_coordinates(aes(geometry = geometry, color = 'my label'), size = 0.9, position = position_jitter(), show.legend = "geometry") and + scale_color_manual(values = 'red').