Orchestration Examples

There is a project with multiple examples of DBND usage. You can find it at https://github.com/databand-ai/dbnd/blob/master/examples.

For orchestration use cases, see DBND orchestration examples on GitHub

Predict Wine Quality

This example shows a more complex version of the Predict Wine Quality tutorial to demonstrate:

• conditional task invocation
• a pipeline that calls another pipeline
• custom logic to fetch partitioned production data.

We start with a basic predict_wine_quality pipeline.

Run a pipeline and set data parameter from the command line

dbnd run predict_wine_quality --set data=dbnd_examples/data/wine_quality_minimized.csv
dbnd run predict_wine_quality_parameter_search  --set predict_wine_quality.data=dbnd_examples/data/wine_quality_minimized.csv

Every parameter at any task in a pipeline can be set from the command line.

dbnd run predict_wine_quality --set validate_model.validation_dataset s3://databand-example/data/wine_quality.csv

Run pipeline with complex fetch data logic:

dbnd run predict_wine_quality  --env prod --set alpha=0.2 --set l1_ratio=0.4  --set fetch.data_period=30d

Our predict_wine_quality pipeline expects a pandas DataFrame as data parameter. Here we show how we build data DataFrame from complex production data. Observe that fetch_data pipeline is called from predict_wine_quality pipeline.

fetch_data() and calculate_alpha() are DBND tasks that are executed in specific conditions (lines 61 and 65)

We write another pipeline to perform a grid search and to invoke predict_wine_quality pipeline numerous times

Syntax Tutorials and Examples

To find our more about the possible syntax for DBND tasks and pipelines, please take a look at this GitHub folder.