Following @MRocklin response, I tested with Dask version 2024.9.1. Logging in distributed workers can be set up using WorkerPlugin. This approach ensures that logging is configured for each worker at startup. Here’s an example:

from distributed import WorkerPlugin
import logging

class MyWorkerPlugin(WorkerPlugin):
    def setup(self, worker):
        # Configure logging on each worker
        logging.basicConfig(level=logging.INFO)

# Initialize your Dask client...

worker_plugin = MyWorkerPlugin()
client.register_plugin(worker_plugin)

# Make some computation...

Explanation

In this example:

• WorkerPlugin.setup is called at the start of each worker, applying the desired logging configuration.
• By setting up logging in the setup method, you ensure consistent logging across all workers, regardless of where they are instantiated.