Enabling pyarrow in spark config resolved this problem. Below is the setting used. pyarrow seamlessly converts pandas dataframe types to spark dataframe types.

"spark.sql.execution.arrow.pyspark.enabled": "true"