79201662

Date: 2024-11-18 23:13:09
Score: 0.5
Natty:
Report link

I was overthinking it. We can constrain how much Spark is writing at once by simply constraining the resources we give Spark. If I set numPartitions to 500 but only give Spark a single 32-core worker, it will only write 32 partitions at a time, limiting how much we're hammering Oracle. Thus effectively "chunks" the job.

Reasons:
  • Has code block (-0.5):
  • Self-answer (0.5):
  • Single line (0.5):
Posted by: Depressio