GuidesAPI ReferenceDiscussions
GuidesBlogPlatform

[deprecated] Airflow Tracking Configuration

Using Airflow Connection to Configure Airflow Monitor and SDK

In order to configure the Airflow monitor and SDK, you can create an Airflow connection with an ID of dbnd_config and use its JSON field to provide configuration parameters as shown below. This configuration will become a part of the configuration used by Databand. See SDK Configuration for more ways of configuration. Setting Databand configurations using Airflow connections will overwrite configurations from ENV and project.cfg.

The JSON contains the following fields:

Mandatory Parameters
Replace the values in <...> with the following information:

Optional Parameters

  • <dag_ids> - list of specific DAG IDs that you want to track
  • <number_of_airflow_log_head_bytes> - number of bytes to collect from DAG log heads
  • <number_of_airflow_log_tail_bytes> - number of bytes to collect from DAG log tails

In order to add a new config you need to:

  1. Go to connections page in Airflow UI.
  2. Add a new Connection:
    1. conn_id: dbnd_config
    2. conn_type: HTTP
    3. Extra: databand configurations formatted in json
    For example:
{
  "core": {
      "databand_url": "http://localhost:8080"
    }
}

Edit dbnd_config

The JSON object contains the minimum parameters required to establish communication between Airflow and Databand.

In general, the JSON keys map to the parameters found in databand-core.cfg. See our Configuration documentation on how you can extend this object with additional parameters.

However, you should NOT modify the values of the following fields manually:

  • dag_ids
  • number_of_airflow_log_head_bytes
  • number_of_airflow_log_tail_bytes
  • track_source_code

These fields get their values automatically from the values you provide in the syncer dialog.
Even when you edit an existing syncer, the Databand monitor would sync them to dbnd_config in Airflow automatically.