I took a closer look and it looks like the issue was happening for me at the display() part, so I removed that and saved the results to a csv and that worked!

        .load() \
        .write \
        .format("csv") \
        .option("header", "true") \
        .mode("overwrite") \
        .save(output_path)