I discovered the answer, in case anybody finds this. The problem is in the font-family. If I specify
pdf("font_plot.pdf", family="Lexend Deca Medium", width=4, height=4)
for example, then it works like a charm!