You could use

scale_color_fermenter(limits = c(0,3), breaks = 1:3)

and shift the key labels down a bit with

+
  theme(
    legend.text = element_text(vjust = 2.5)
  )