79103071

Date: 2024-10-18 17:23:22
Score: 1
Natty:
Report link

As answered on Dask Discourse Forum.

For Dahsboard information after cluster shutdown, you've got the performance_report context manager.

Another solution would be to have one script for starting the Dask cluster, and another script to submit job to it. You can save Scheduler information into a json file to share it to another process.

For the long running process, a common workflow is to submit a “master” job, which will create Scheduler and client and submit other jobs for Workers through SLURMCluster. You can also take a look at dask-mpi.

Reasons:
  • No code block (0.5):
  • Low reputation (0.5):
Posted by: Guillaume EB