Including the line
+ coord_sf(expand = FALSE)
should fix your issue and still let you scale the axis as you prefer.