Use the option discrete = TRUE:

ggplot(df, aes(x = x, y = y)) +
  geom_density_2d_filled() +
  scale_fill_viridis(direction = -1, discrete = TRUE)

geom_density_2d_filled expects a discrete color scale since it bins the density into contours, so we need to tell scale_fill_viridis to provide a discrete, rather than continuous color scale.