79205370

Date: 2024-11-19 23:42:06
Score: 2
Natty:
Report link

It depends on what your Airflow deployment looks like. Is your Airflow deployed on Kubernetes? Then the file system gets set on fire after each task completes. Pods are ephemeral or short lived. So there's nothing to access in the later task.

If you're running everything on a single EC2, then yes it might be feasible. But it's an antipattern in my opinion and according to Airflow's docs. The cross-task communication mechanism in Airflow is Xcoms.

Airflow's example xcoms DAG should be helpful for getting started with the feature.

Reasons:
  • No code block (0.5):
  • Contains question mark (0.5):
  • Low reputation (1):
Posted by: Carlos Caban