Process Monitoring

What information can be monitored with the help of Databand.

Databand helps data engineers guarantee reliable SLAs. You can use the solution to monitor runs, alert on failures, and do root cause analysis to find where errors are coming from. You can collect application logs and metadata from your pipelines including:

  • Runtime information
  • Task dependencies and lineage
  • Resource consumption
  • Code versioning information

Orchestration Level Information

If you are using an orchestrator such as Apache Airflow, Databand can sync metadata from the Airflow database and provide you deeper insights and utilities for monitoring your DAG health, for example alerting on anomalous runtimes.

Task Metadata

If executing jobs in remote or distributed systems like Spark, SQL databases, or docker containers, it may be a challenge to gather the right information from your execution environment and understand how it aligns with your DAGs. This can lead to information silos that slow down debugging/RCA, or even create inconsistencies between systems. Databand can track metadata and logs from task executors, so you can access log and error information in one place.

Spark job metrics in Databand UISpark job metrics in Databand UI

Spark job metrics in Databand UI

Function Metadata

Databand can provide granular tracking of workflows to the function level, helping you drill into pipelines and instantly discover where errors arise.


Did this page help you?