79577624

Date: 2025-04-16 16:08:40
Score: 0.5
Natty:
Report link

The best way in production is to use git-sync. Here's a relevant blog post by Airflow contributor and Apache PMC member Jarek Potiuk: https://medium.com/apache-airflow/shared-volumes-in-airflow-the-good-the-bad-and-the-ugly-22e9f681afca.

The crux is - DAGs are code, and code needs versioning to scale. In production, you would create a git repo containing your DAGs, just like one does for code. Meanwhile the git-sync sidecar automatically pulls and syncs your DAGs to airflow.

Another possible way to leverage the power of git is to store the repos in a volume that is used as a shared volume in airflow. This is discouraged because shared volumes bring inefficiencies, i.e., git-sync is expected to scale better.

You could in a way use both by setting persistence as well as git-sync to true (in the helm installation's values.yaml). But this gave me an error. It is an open issue: https://github.com/apache/airflow/issues/27476. If you must use this method, this post discusses what you should take care of: https://www.restack.io/docs/airflow-faq-manage-dags-files-08#clp226jb607uryv0ucjk42a78.

Reasons:
  • Blacklisted phrase (0.5): medium.com
  • Long answer (-0.5):
  • Has code block (-0.5):
  • Low reputation (1):
Posted by: kanishk