✳flâneur — a map of the web's best reading

Modules Management — Airflow Documentation

airflow.apache.org · 2,597 words · saved by 1 readers

Airflow allows you to use your own Python modules in the DAG and in the Airflow configuration. The following article will describe how you can create your own module so that Airflow can load it correctly, as well as diagnose problems when modules are not loaded properly. Often you want to use your own python code in your Airflow deployment, for example common code, libraries, you might want to generate DAGs using shared python code and have several DAG python files. You can do it in one of those ways: add your modules to one of the folders that Airflow automatically adds to PYTHONPATH add extra folders where you keep your code to PYTHONPATH package your code into a Python package and install it together with Airflow. The next chapter has a general description of how Python loads packages and modules, and dives deeper into the specifics of each of the three possibilities above. The list of directories from which Python tries to load the module is given by the variable sys.path. Python r

Modules Management - Airflow 3.3.0 Documentation Content Version: 3.3.0 Search docs ⌘ K Content Overview Quick Start Installation of Airflow® Security Tutorials How-to Guides UI Overview Core Concepts Authoring and Scheduling Administration and Deployment Production Deployment Logging & Monitoring Kubernetes Lineage Listeners Dag Bundles Dag Serialization Modules Management How package/modules loading in Python works Typical structure of packages Built-in PYTHONPATH entries in Airflow Best practices for your code naming Use unique top package name Don’t use relative imports Add __init__.py in

Explore this link on the map →

related reading