79432386

Date: 2025-02-12 08:36:45
Score: 1.5
Natty:
Report link

How foreachPartition works

When you call foreachPartition, Spark:

Serializes the function and sends it to the executors.

Each executor applies the function to its assigned partition(s).

No data is returned to the driver (unlike collect() or show()).

As @Steven has mentioned, you should check logs on Executors side. Moreover, you need to be careful with DB connections that are established from Executors.

This part:

db_client = PostgresDbClient(DbConfigBuilder(config, config['vault']['service_type'], ssl_cert_file=None))

Check whether you have all the needed libraries, open ports, network configuration, etc. - all that you have to use to connect to PostgreSQL in the same way you did it in the main program.

Reasons:
  • Long answer (-0.5):
  • Has code block (-0.5):
  • User mentioned (1): @Steven
  • Starts with a question (0.5): How for
  • Low reputation (1):
Posted by: Alexei Sosenkov