GuidesAPI ReferenceDiscussions

Introduction to Configuration

This is an overview of the supported configuration layers in DBND.

DBND configuration system provides fine-grained control over your executions.
The DBND configuration is built in layers. You can augment and change the configuration for every task by adding new configuration layers of your own.

Metadata specified in a higher configuration layer is overridden with the configuration specified in a lower layer:

Configuration layers in DBNDConfiguration layers in DBND

Configuration layerDescriptionReference
The --override command in CLIThis command is the most "authoritative" way to change configuration - it is absolute, and cannot be overridden, hence it should be used only in suitable situations. In most cases, other layers are suitable.Overriding Configuration in CLI
Constructor configuration layerThe constructor configuration layer draws its data from the parameters provided when instantiating the task.Overriding constructor default values
Task configuration (--set)Task configuration is used for configuring tasks that may or may not have default values.
It sets a specific value for a parameter, but does not override the task constructor.
Configuration files and Environment VariablesConfiguration files can be used for configuring specific tasks, as well as project-wide settings.Configuration files overview
Default values in the task definition specified in codeDefault values are the last fallback for configuration changes. If no other value is configured, the default value is used.


from dbnd import task

def calculate_alpha(alpha=0.5, beta=0.1):
    return alpha

alpha_task = calculate_alpha(alpha=4)

Using the constructor configuration layer

Using this layer is as simple as overriding the default values that each task defines when invoking the task constructor. In the above example, we override the positional parameter alpha whose default value is 0.5. We set its value to 4.

Using the task configuration layer

With the CLI command:

dbnd run calculate_alpha --task-version now --set beta=0.2 --set alpha=3

Our beta value is going to be 0.2. --set supersedes the task's default value.
However, alpha value is going to be 4, as it has 4 coming from the constructor.

Using configuration files for specific tasks

Let us look at the same code example again. Let's define a configuration section for the task calculate_alpha in project.cfg:

alpha = 2
beta = 2

The value of the beta during execution is going to be 2. The configuration is given in the project.cfg file supersedes the task's default value, and hence overrides beta.
However, alpha is going to stay 4 as the constructor has a higher priority.