The problem is that by having fill aesthetics in ggplot(), you are telling stat_pvalue_manual that fill = Temperature, which it doesn't know what to do with because Temperature doesn't exist in contrasts.means.species.temp. Move the fill aesthetic to geom_boxplot instead.