79138528

Date: 2024-10-29 18:38:36
Score: 0.5
Natty:
Report link

If I understand well, this code will do the job simply and quickly by using the partitionBy option :

df = spark.read.format('avro').load("/user/test/data/data_backlog")
df.write.partitionBy("dt").format('avro').save("/user/test/data/data_backlog_backup")
Reasons:
  • Low length (0.5):
  • Has code block (-0.5):
  • Low reputation (0.5):
Posted by: Ahmed