DBND provides out-of-the-box environments that you need to configure before you can start running your pipelines.
Out-of-the-box, DBND supports the following environment types:
- Persistency - Local file system, AWS S3, Google Storage, Azure Blob Store, and HDFS
- Spark - Local Spark, Amazon EMR, Google DataProc, Databricks, Qubole, and Livy
- Docker Engine - Local Docker, AWS Batch, Kubernetes.
You can also create custom environments and engines.
• The environments parameter in the
core section specifies a list of environments enabled and available for the project.
Possible values include
local, gcp, aws, azure, and
local - set by default.
[core] environments = ['local', 'gcp']
The following table describes the environment types supported in DBND:
In the default
Google Cloud Platform (GCP),
The Spark engine is preset for Google DataProc.
Amazon Web Services (AWS),
The Spark engine is preset for Amazon EMR.
Microsoft Azure (Azure),
The Spark engine is preset for Databricks.
You can define a custom environment from scratch or inherit settings from an existing environment.
To use out-of-the-box environments, set up one or more of the environments described in the referenced topics.
Updated 11 days ago