Assigning pipelines to projects allow:
- Creating logical grouping of pipelines/runs for a customized user experience
- Separating different use cases
- Filtering pipelines according to project names
- Routing alerts from every project to the different receivers (no UI yet, ask databand support to configure)
Set a tag for your Airflow DAG using the following format:
with DAG( dag_id="my_dag schedule_interval="0 1 * * *", # Every day at 1:00 tags=[ "project: my_project", # my_dag would be assigned to my_project project in Databand UI ], ) as my_dag:
To create and name a new project, you can do the following (in order of priority):
- A project name can be set via property
tracking.project(see SDK Configuration):
- Via CLI in orchestration mode:
dbnd run dbnd_sanity_check --project=ProjectAlfa
Project assignment is controlled by an SDK. All DBND SDKs are designed to support projects’ pipelines.
- In DBND orchestration, a project is a part of the information belonging to a pipeline run.
- Airflow SDK supports project assignment by an environment variable. Airflow tags are also supported.
- Azkaban-native projects are supported, too.
After upgrading to Databand v0.35.1+, all the pipelines (and the runs that belong to those pipelines) are put into a ‘default’ project. This makes projects backward compatible with SDKs that do not define projects.
You cannot rename or delete the ‘default’ project.
Projects can be filtered using a drop-down list in the Databand UI in the Pipelines, Runs, Alerts, and Schedule tabs. Currently, the projects in the drop-down list are sorted in the order of their creation.
Click ‘Clear filter’ to return to the default state and see all the available pipelines again.
One pipeline can be assigned to more than one project on a per-run basis.
For instance, the first run of a pipeline can be assigned to "ProjectA" and the second run of the pipeline can be assigned to "ProjectB", and so on.
These pipelines will be displayed and treated as separate pipelines with the same name, but belonging to different projects.
As a result, all alerts defined for the pipeline/ProjectA can be triggered only when the pipeline is run under "ProjectA".
Updated 2 months ago