79221382

Date: 2024-11-25 01:29:34
Score: 0.5
Natty:
Report link

I've tried to skip the step of writing each small dataframe as a small file to local disk.

It filtered the rows (4 or 5 in Meet) instead of saving all rows in each smaller dataframe, and tried directly wrote them to a final txt file.

The disadvantage was that, I am not able to monitor the progressing as it's running "behind". Taken even longer hours than the old method, I aborted the test.

Another try was to only save the small dataframe when there are filtered rows (4 or 5 in Meet), and save them into small Parquests. Taken a further step to contact the Parquests. This is the most efficient way I can get, even though it still takes long.

Reasons:
  • Blacklisted phrase (1): I am not able to
  • Long answer (-0.5):
  • No code block (0.5):
  • Self-answer (0.5):
  • High reputation (-1):
Posted by: Mark K