Kubernetes Engine Configuration

How to configure DBND to use your Kubernetes Cluster.

Configuring Kubernetes engine

To direct DBND to interact with your cluster, you need to update the databand-system.cfg configuration file with the Kubernetes Cluster connection details.

Step 1. In the environments configuration, add a new environment for the Kubernetes cluster. For example, kubernetes_cluster_env - you can use an existing one, but in this example, let's assume its name is 'kubernetes_cluster_env':

environments = ['local', 'local_minikube', 'kubernetes_cluster_env']

Step 2. In the configuration file, add the environments' specification for your new or existing environment (kubernetes_cluster_env):

_from = local
remote_engine = your_kubernetes_engine
Parameter `_from` means "from where to draw previous definitions". In this example, the `[local]` section is used (see [Extending Configurations](doc:custom-environment)).
The `remote_engine` setting defines what engine is going to run your remote tasks (submitted tasks).

Step 3. In the configuration file, add the engine configuration.
The following example describes the engine configuration:
_type = kubernetes

container_repository = databand_examples
container_tag =

docker_build = True
docker_build_push = True

cluster_context = databand_context
namespace = databand
service_account_name = databand

debug = False
secrets = [
   { "type":"env", "target": "AIRFLOW__CORE__SQL_ALCHEMY_CONN", "secret" : "databand-secret", "key": "airflow_sql_alchemy_conn"},
   { "type":"env", "target": "AIRFLOW__CORE__FERNET_KEY", "secret" : "databand-secret", "key": "airflow_fernet_key"},
   { "type":"env", "target": "DBND__CORE__DATABAND_URL", "secret" : "databand-secret", "key": "databand_url"}

pod_error_cfg_source_dict = {
                            "255": {"retry_count": 3, "retry_delay": "3m"},
                            "err_image_pull": {"retry_count": 0, "retry_delay": "3m"},
                            "err_pod_deleted": {"retry_count": 3, "retry_delay": "3s"},
-   `_type` - Implies that this is a Kubernetes engine (see [Extending Configurations](doc:custom-environment)). 
-   `container_repository` - Where is the Docker image repository to pull the pod images from? If you are running user code, this is where you need to supply your repository and tag settings. 
-   `container_tag` - If defined, Docker will not be built and the specified tag will be used.
-   `docker_build` - Should the Kubernetes executor build the Docker image on the fly? Useful if you want a different image every time.
-   `docker_build_push` - Should the built Docker image be pushed to the repository? Useful for specific cases.
- `cluster_context` - The Kubernetes context; you can check which context you are on by using `kubectl config get-contexts`.
-   `namespace` - The namespace in which Databand is installed inside the cluster (`databand` in this case).
- `service_account_name` - You need permissions to create pods for tasks, namely -  you need to have a `service_account` with the correct permissions.
-   `debug` - When true, displays all pod requests sent to Kubernetes and more useful debugging data.
-   `secrets` - This is a setting that will be sent along with every pod DBND creates to allow it to access your secret AWS credentials (If AWS is required).
-   `pod_error_cfg_source_dict` (optional) - Allows flexibility of sending retry on pods that have failed with specific exit codes. Optional. 

**Other useful configurations**
-   `image_pull_secrets` - The secret with the connection information for the `container_repository`.
-   `in_cluster` - Defines what Kubernetes configuration is used for the kube client.  Use `false` to enforce using local credentials, use `true` to enforce the `in_cluster` mode.  Default: `None` (Databand will automatically decide what mode to use).
- `labels` - Set a list of pods' labels (see [Labels](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/))
- `node_selectors` and `affinity` - Assign `nodeSelector` or `affinity` to the pods (see [Assigning Pods to Nodes](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/))
- `annotations` - Assign annotations to the pod (see [Annotations](https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/)) 
- `tolerations` - Assign tolerations to the pod (see [Taints and Tolerations](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/))
-  requests and limits - Setting the requests and limits for the pod can be achieved by setting those
For more information see [Manage Container Resources](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/) and make sure you are aware of [Quality of Service for Pods](https://kubernetes.io/docs/tasks/configure-pod-container/quality-service-pod/)
    - `request_memory`
    - `request_cpu`
    - `limit_memory`
    - `limit_cpu`

How to Config a Tasks Submission from a Local Machine

If you are going to submit tasks from your local machine, make sure you have access to the Databand server. In the [core] section, update your local DBND settings with the relevant connection strings:

databand_url = http://databand-web:8080 

Providing Access to AWS

If you want to provide an access to AWS Services explicitly, you can do it by using secrets:

secrets = [   { "type":"env", "target": "AWS_ACCESS_KEY_ID", "secret" : "aws-secrets" , "key" :"aws_access_key_id"},
          { "type":"env", "target": "AWS_SECRET_ACCESS_KEY", "secret" : "aws-secrets" , "key" :"aws_secret_access_key"}]        

Custom configuration per task

You can adjust configuration settings of specific task

@task(task_config=dict(kubernetes=dict( limits={"nvidia.com/gpu": 1})))
def prepare_data_gpu(data):

Using this configuration, you'll add an extra limit to the pod definition of this specific task.

Did this page help you?