Airflow Orchestration

How to integrate DBND with your existing Airflow environment, and run your pipelines from Airflow.

Apache Airflow is an open-source product for programmatically authoring, scheduling, and monitoring data pipelines. It has quickly grown to become a standard for data engineers due to its flexibility, easy of use, and comprehensive UI.

You can integrate DBND with your existing Airflow environment, and run your pipelines from Airflow, leveraging the Airflow scheduler and using the Airflow UI. The appeal of this approach is that it allows you to leverage the benefits of the Airflow scheduler.

You can integrate your existing Airflow environment with DBND, while continuing to use the Airflow UI to monitor your pipelines. Here are two modes of using DBND with your Airflow DAGs:

Operated DAGs

Define a DBND pipeline that’s executed by Airflow’s scheduler as a task. How it works:

  • Create a DBND pipeline
  • Contain it as an operator (Bash, Python, etc.)
  • Use Airflow to run and monitor the pipeline

Decorated DAGs

Functionally wire DBND tasks to create Airflow DAGs. How it works:

  • Use DBND task decorators to seamlessly define tasks using functions
  • Use the DBND pipeline decorator to connect these tasks to form an Airflow DAG
  • Use Airflow to run and monitor the pipeline

Also, you can run your DBND pipelines directly using Airflow Orchestration Engine. Use Task.task_airflow_op_kwargs to pass defaults to the Airflow operator that would run this task


What’s Next
Did this page help you?