Projects

Assigning pipelines to projects allow:

• Creating logical grouping of pipelines/runs for a customized user experience
• Separating different use cases
• Filtering pipelines according to project names
• Routing alerts from every project to the different receivers (no UI yet, ask databand support to configure)

Assigning Pipelines to Projects

Assign Airflow DAG to databand project

Set a tag for your Airflow DAG using the following format:

with DAG(
    dag_id="my_dag
    schedule_interval="0 1 * * *",  # Every day at 1:00
    tags=[
        "project: my_project", # my_dag would be assigned to my_project project in Databand UI
    ],
) as my_dag:

Using SDK

To create and name a new project, you can do the following (in order of priority):

• A project name can be set via property tracking.project (see SDK Configuration):

[tracking]
project=my_project

• Via configuration in tracking mode:

    with dbnd_tracking(
        conf={
            "tracking": {
                "project": "my_project",
            }, 
        }
    ):
      ...

• Via CLI in orchestration mode:

dbnd run dbnd_sanity_check --project=my_project

Project assignment is controlled by an SDK. All DBND SDKs are designed to support projects’ pipelines.

• All the pipelines (and the runs that belong to those pipelines) by default are put into a ‘default’ project. You cannot rename or delete the ‘default’ project.
• In DBND orchestration, a project is a part of the information belonging to a pipeline run.
• Airflow SDK supports project assignment by an environment variable. Airflow tags are also supported.
• Azkaban-native projects are supported, too.

Filtering Projects

Projects can be filtered using a drop-down list in the Databand UI in the Pipelines, Runs, Alerts, and Schedule tabs. Currently, the projects in the drop-down list are sorted in the order of their creation.

Click ‘Clear filter’ to return to the default state and see all the available pipelines again.

Running Pipelines in Different Projects

One pipeline can be assigned to more than one project on a per-run basis.

For instance, the first run of a pipeline can be assigned to "ProjectA" and the second run of the pipeline can be assigned to "ProjectB", and so on.

These pipelines will be displayed and treated as separate pipelines with the same name, but belonging to different projects.

As a result, all alerts defined for the pipeline/ProjectA can be triggered only when the pipeline is run under "ProjectA".