To add on what swimmer said:

There is a subtle difference between using on_failure_callback on DAG level and on task level. On task level it appears that the worker is handling the method execution, while on DAG level it seems the scheduler is handling the method execution.

I mention this, because we had a docker setup, where we accidentally only defined the AIRFLOW__WEBSERVER__BASE_URL for the webserver and worker but not for the scheduler.

As a result, when we accessed task_instance.log_url inside the DAG-level failure callback, the hostname defaulted to localhost — since the scheduler didn’t have the proper base URL configured.