79414404

Date: 2025-02-05 10:26:16
Score: 1.5
Natty:
Report link

With thanks to @Paulo Marques for the suggestion: I converted the latitude and longitude fields to str type while the data were still in a pandas dataframe.\

The block that converts to Spark and writes to table now looks like this:

basic_df = pd.DataFrame(basic_data_list)
basic_df.latitude = basic_df.latitude.astype(str)
basic_df.longitude = basic_df.longitude.astype(str)

    if not basic_df.empty:
            basic_spark_df = spark.createDataFrame(basic_df, schema=basic_schema)
            basic_spark_df.write.mode("append").option("mergeSchema", "true").format("delta").saveAsTable("api_test")          

And all the data now pull through without any errors.

Reasons:
  • Blacklisted phrase (0.5): thanks
  • Long answer (-0.5):
  • Has code block (-0.5):
  • User mentioned (1): @Paulo
  • Self-answer (0.5):
  • Low reputation (0.5):
Posted by: CrowsNose