this error because of NaN presents in your Dataframe. I got resolved this by following
df=df.fillna('')
df1 = spark.createDataFrame(df)