Kedro-Airflow
Please Note: As part of our move to the Linux Foundation Kedro-Airflow has been moved to kedro-plugins and this repository will no longer be maintained. Please find an up-to-date copy of Kedro-Airflow here.
Apache Airflow is a tool for orchestrating complex workflows and data processing pipelines. The Kedro-Airflow plugin can be used for:
- Rapid pipeline creation in the prototyping phase. You can write Python functions in Kedro without worrying about schedulers, daemons, services or having to recreate the Airflow DAG file.
- Automatic dependency resolution in Kedro. This allows you to bypass Airflow's need to specify the order of your tasks.
- Distributing Kedro tasks across many workers. You can also enable monitoring and scheduling of the tasks' runtimes.
Installation
kedro-airflow is a Python plugin. To install it:
pip install kedro-airflowUsage
You can use kedro-airflow to deploy a Kedro pipeline as an Airflow DAG by following these steps:
Step 1: Generate the DAG file
At the root directory of the Kedro project, run:
kedro airflow createThis command will generate an Airflow DAG file located in the airflow_dags/ directory in your project.
You can pass a --pipeline flag to generate the DAG file for a specific Kedro pipeline and an --env flag to generate the DAG file for a specific Kedro environment.
Step 2: Copy the DAG file to the Airflow DAGs folder.
For more information about the DAGs folder, please visit Airflow documentation.
Step 3: Package and install the Kedro pipeline in the Airflow executor's environment
After generating and deploying the DAG file, you will then need to package and install the Kedro pipeline into the Airflow executor's environment. Please visit the guide to deploy Kedro as a Python package for more details.
FAQ
What if my DAG file is in a different directory to my project folder?
By default the generated DAG file is configured to live in the same directory as your project as per this template. If your DAG file is located in a different directory to your project, you will need to tweak this manually after running the kedro airflow create command.