Azure Environment
How to configure an environment for Azure.
Before You Begin
You must have dbnd-azure
plugin installed.
To Set up an Environment for Microsoft Azure
- Open the project.cfg file and add
Azure
to the list of environments. - Review the
[azure]
section and configure it in accordance with your login settings. - Optionally, provide a root bucket/folder for your data:
[azure]
root = https://<your_account>.blob.core.windows.net/dbnd
spark_engine = databricks
- Configure access to your Blob storage:
dbnd airflow connections --delete --conn_id=azure_blob_storage_default
dbnd airflow connections --add --conn_id=azure_blob_storage_default --conn_login=<acount name> --conn_type=wasb --conn_password=<acount key>
[azure]
Configuration Section Parameter Reference
[azure]
Configuration Section Parameter Referenceenv_label
- Set the environment type to be used. E.g. dev, int, prod.production
- This indicates that the environment is production.conn_id
- Set the cloud connection settings.root
- Determine the main data output location.local_engine
- Set which engine will be used for local executionremote_engine
- Set the remote engine for the execution of driver/taskssubmit_driver
- Enable submitting driver toremote_engine
.submit_tasks
- Enable submitting tasks to remote engine one by one.spark_config
- Determine the Spark Configuration settingsspark_engine
- Set the cluster engine to be used. E.g. local, emr (aws), dataproc (gcp), etc.hdfs
- Set the Hdfs cluster configuration settingsbeam_config
- Set the Apache Beam configuration settingsbeam_engine
- Set the Apache Beam cluster engine. E.g. local or dataflow.docker_engine
- Set the Docker job engine, e.g. docker or aws_batch
Updated 3 months ago