I don't exactly know how your error is occurring, as I cannot seem to replicate it myself. I'm getting your desired dataframe as output even with your code. That said, the below works as well, and relies on dplyr from the tidyverse

library(tidyverse)
library(vegan)

caract_sites <- caract_sites |> #|> is the same as %>%, but preferred by the r style guide
  mutate(specnumber = specnumber(site_sp_ok))

When adding or changing rows in a dataframe I find dplyr::mutate() works well for me.