This version was released on November 11, 2021.
Observe metric trends over time: missing charts, run state charts, state metrics charts, schema change metric trends, etc.
- AS-1200944564541400: The final state of sub-pipeline tasks that have failed child tasks is now called "Failed" instead of "Upstream Failed".
- AS-1201110693029241: Operations merged from the SDK and BigQuery logged with
log_dataset_opare now recognized and displayed accordingly (with BigQuery ID) in the Dataset Operations tab, including with stateful issues.
- AS-1201314083672982: Resolved notifications are no longer the default configuration for Slack receivers.
- AS-1201141157594482: Alert dialog loads promptly and displays the correct state.
- AS-1201257264431636: Added support for tracking specific DAGs.
- AS-1201226401826259: Syncer memory guard limit is now configurable.
- AS-1201266020003308: Pipelines with zero associated runs are hidden in the Databand UI.
- AS-1201260807813998: Resolved notifications are now correctly delivered to the update/validate receivers.
- AS-1201036807083630: In the Alert Definitions' editor, individual tasks can be addressed when DAG has multiple tasks with the same name.
COPY INTOqueries are now supported for Snowflake.
- AS-1201154986480578: Operations section on the Datasets tab notifies of unusual activity in an operation (schema change or failed operation).
- AS-1201080247459590: Operations are now tracked with BigQuery even when no SDK is installed.
- AS-1201080247459582: Dataset Overview section combines the data from SDK and from BigQuery.
- AS-1201160985639176: Solved a bug that emerged when deleting a DAG.
This version was released on October 28, 2021.
Java SDK is now able to catch and display operation errors.
Added time picker to the Overview tab.
- AS-1201233822389401: Airflow monitor doesn't load the entire AirflowMonitor SmartDagBag, but loads each DAG as requested.
- AS-1201270762446144: Added AirflowMonitor memory guard to allow the Monitor DAG to automatically stop if it consumes more memory than allowed (8GB by default, configurable).
- AS-1201161190744548: Metric view does not display system metrics from tasks.
- AS-1200887319437205: Receiver configuration is now accessible from the Alerts tab.
- AS-1200978826558970: BigQuery Monitor errors are followed by a 'red dot" notification on the left bottom corner of the UI.
- AS-1201174491849652: Chart "Current Table Size" renamed to "Total Rows Over Time" in the Dataset Overview.
- AS-1201174491849648: Added a "copy" icon next to the path and a "Copy to clipboard" tooltip on hover in the Datasets tab.
- AS-1200941787313784: Added navigation from the Dataset operation list to the Task Run details.
- AS-1200998434897582: Relevant parts of long metric names are not mistakenly hidden in the UI anymore.
- AS-1201137363846218: Unclickable "bad tasks" in Affected Datasets are not clickable anymore.
- AS-1201068945769376: Archived runs stopped being displayed in the record count history of Affected Datasets.
- AS-1201225016615890: Dataset operation list table is scrollable again.
- AS-1201225016615878: Fixed histogram calculation.
This version was released on October 13, 2021.
Metric view now includes dataset read and write metrics.
You can enable Status Alerts on pipelines and sub-pipelines.
- AS-1200766175220433: Added time picker to the Overview section of the Datasets tab.
- AS-1200768017051953: Introduced empty states for Affected Datasets.
- AS-1200834591237808: Added Record columns to the list of operations in Affected Datasets.
- AS-1200964446833790: Both future and past Runs are shown in the operation's volume history in Affected Datasets.
- AS-1200565098066843: Both future and past Runs are shown in the records’ history overview chart in Affected Datasets.
- AS-1201110693029252: Failed operations no longer falsely marked as "Successful" in Affected Datasets.
- AS-1201133337683376: When alerts are fired on missing data, the issue is listed in the Affected Datasets section as it should be.
- AS-1201133337683372: Data operations are properly displayed in Affected Datasets.
- AS-1201110693029259: Charts in Dataset Overview include no "Failed" operations or transactions now.
- AS-1201110693029254: Fixed the issue with record count that falsely displayed 0 records.
Note: Starting from version 0.51, a pipeline that contains a failed task will be marked as "failed". Previously, the pipeline status would be marked as "upstream failed".
This version was released on October 04, 2021.
Added SDK Client for Metadata (basic access API endpoint for collecting all errors in a Run).
- AS-1201013784336315: Improved behavior of Reset Selection in Affected Datasets.
- AS-1200766175220432: Now you can create alerts directly from the redesigned Metrics tab.
- AS-1200964944935744: Each step on the X-axis in Dataset Overview now equals 24 hours from the previous point.
- AS-1200964944935725: "Reported from" and "Dataset Type" columns in the Datasets tab follow the same logic as elsewhere in the Databand UI.
This version was released on September 20, 2021.
Datasets can be archived now.
Added empty state clickable component.
Default operation is pre-selected on the Alert definition screen.
- AS-1200948790296167: Dropped all
- AS-1200766175220434: All alerted metrics that are being reported are now visible.
- AS-1200892336582865: Reviewed documentation links in the code.
This version was released on August 31, 2021.
Run metrics in the list of metrics are now selected by default and are available from the filters in the Databand UI.
A new look for the Dataset page.
Metrics page added - see all the metrics, alerts, and trends for numeric values in one place (coming in a later minor release).
- AS-1200876485130367: Dropped py27 support for
- AS-1200854192669862: Dropped support for Tensorflow <2.4 and py27 in
- AS-1200495012906365: Defined and implemented Network Policy to isolate workloads in namespaces.
- AS-1200845451577692: Delete alert definitions in bulk using the “Delete” button.
- AS-1200791698233717: Trigger value of Missing Operation alert now reflects the total number of missing operations in the run.
- AS-1200667315624593: Tooltip in schema change now opens on click.
- AS-1200587432732541: Text in the UI now aligns to the left after a line break.
- AS-1200785050007987: Fixed a bug where the “last modified” field returned an empty value.
- AS-1200847984199269: Fixed a “type change” reporting bug in schema change alerts.
This version was released on August 19, 2021.
BigQuery connector - Databand can now sync with your BigQuery tables.
Databand Dashboard can now be shared and will include filters, metrics, and aggregations.
- AS-1200700177752978: Airflow Monitor V1 is deprecated.
- AS-1200618993918094: Improved visibility of missing operations in Affected Datasets.
- AS-1200721895327001: Updated the UI on the Integrations page.
- AS-1200624166545000: Full table path in the Affected Datasets tab is now a tooltip.
- AS-1200654355859365: Source and project names are displayed correctly.
- AS-1200670021315930: Missing operations are now properly represented in the History Trend chart in the Affected Datasets tab.
- AS-1200655482328132: Fixed the dropdown table column in the 'Add new alert' page.
This version was released on August 03, 2021.
Pipeline and run lists optimized for tracking.
Now you can choose whether to have alerts fire on historical runs by editing alert definitions.
Range alerts added. Now it’s possible to add an alert if the metric's value is in a predefined range or out of range.
New schema Alert flow changes in Affected Datasets: schema changes in a data operation are now shown in the Affected Datasets screen. Now you'll know when a column is added, removed, or when its type changes. Some minor cosmetic changes in the UI were also implemented.
- AS-1200610543227298: Simplified usage of
in_clusterin cases where
- AS-1200629604891227: Run view performance improvements.
This version was released on July 21, 2021.
Databand Alerting now supports email as a notification channel.
Per Project and Source Alert routing - you can define Alerts on any job as you like.
New Run duration metrics: Duration, Total Duration, Compute Time.
New alert type: get notifications when a schema changes on the data you logged with
- AS-1200535433472848: Dashboard widgets now show the project or source name if there are multiple sources. This addresses the case of a pipeline with the same name coming from 2 sources.
- AS-1200587432732540: Look at the affected datasets of past Runs by clicking on Runs.
- AS-1200557294252243: New tooltip added to the Dataset overview chart - you now can display the Run status and add/view label.
- AS-1200602972144983: Dataset overview chart changed from stacked to a regular graph.
- AS-1200526324086040: New alert type: when using
log_dataset_op, you can get notifications when a pipeline didn’t read or write as expected.
- AS-1200430856087576: Alerts can now be set directly from the Alerts page.
- AS-1200557294252244: Missing operations are now prominently displayed in the history chart and in the Trend operation bars graph.
- AS-1200557294252253: Improved coloring for better readability of dataset graphs.
- AS-1200587833068258: The description in the alert pop-up is no longer missing.
- AS-1200587301890745: Datasets page sorting and filtering are fixed - dataset operations are sorted by most recent first.
- AS-1200557970734951: Current Run is marked as usual.
- AS-1200579848765971: Datasets page filters project correctly.
This version was released on July 07, 2021.
Choose and view any suitable metric in a new chart in the Favorite Metric widget.
View Run's data operations (writes, schema, etc.) in the Run with alerts and see what happened in the last 6 Runs.
Helm chart variables now support Global image configs.
- AS-1200526450615534: Improved performance of Databand Web UI configuration.
- AS-1200371843828304: Errors from Airflow are now displayed in the Run list.
- AS-1200463339266254: An extensive round of improvements of the metrics reporting and display processes.
- AS-1200258862047040: Access to custom metrics is now available. Reporting metrics and discovering what metrics are reported within Databand got much easier.
- AS-1199332360687562: [Breaking change!] Deprecated
__executetasks and separation for root-Run and sub-Runs per Airflow task. All the information which was previously reported on
<task_name>__execute task(like metrics or log) is now reported on
- AS-1200483852764557: The
log_dataset_opbehavior is now the same for
log_dataset_opcan have the
This version was released on June 22, 2021.
Outside-of-the-box tracking support for PySpark with Databricks.
Added Live Logs for Kubernetes workers in the Airflow UI.
Missing Data now triggers an alert.
Now you can calculate metric aggregations like SUM/AVG and others and display them in the Favorite charts widget. You can also compare trends over the filtered time period and display trends.
Dataset Screen is deprecated and replaced by Data Targets and Datasets screen in the UI.
Added support for custom headers for Alert Manager webhooks.
- AS-1200403795815183: Improved Favorite Metric widget usability.
- AS-1200463339266254: Improved Metric tab interface.
- AS-1200452512156534: ‘Triggered by column’ is removed from the Run list.
- AS-1200408168252754: History tab is removed from the Run details.
- AS-1200430599620006: Timeline Tab in the Run details is deprecated.
- AS-1200181993522176: Proper support for DagRun submit args and XCom; values are reported with correct types.
- AS-1200275295497774: Added support for the Airflow retries and restarts in Alerts.
- AS-1200372371285197: It’s now possible to expose Run env and user in the
- AS-1200402990123287: Fixed a bug that prevented proper tracking from Databricks.
- AS-1200358546721956: Metrics that cannot be presented in a graph now cannot be added.
- AS-1200275951379648: Fixed incorrect handling of attempts.
- AS-1200440061411916: Fixed tracking tasks from
- AS-1200035644894886: Fixed browser ‘Back’ button behavior in the Run details page.
And other bug fixes.
This version was released on June 10, 2021.
Airflow Monitor v2 now supports Airflow 2.0.
The Favorite Metric widget in the Dashboard now automatically displays anomalies for metrics.
Now supporting Google Composer for Airflow tracking.
Main menu now expands to include titles.
Added a chart display toggle for switching between Run charts in the Favorite Metric widget.
Users can select the main page view. The default mode is Dashboard; options: Dashboard or Pipelines.
The Syncer page was moved to the Settings page.
Now you can identify errors in different Runs - both the error message and the error type now lead to the Run page; the task that triggered the error is selected so you can view the error.
- AS-1200403425760171: Added new Syncer issue counters in the Settings section.
- AS-1200420269065898: Only the table is now scrollable in the Metrics widget.
- AS-1200400792119627: Changes in the chart type design.
- AS-1200400792119629: Changes in the Dashboard chart tooltips - now, the nearest point tooltip will be displayed.
- AS-1200251794221201: Improved the readability of the metric charts tooltip.
- AS-1200378661150501: Added Error Column to All Runs and Pipeline Runs lists.
- AS-1200369692119674: Improved usability of the Alert Definition section.
- AS-1200374990956464: Tasks in Airflow sub-processes are now automatically assigned unique names.
- AS-1200102937873725: Added a unified context manager for handling log dataset operations.
- AS-1200279378863216: Aligned and improved alert counters in the Pipeline and Run lists.
- AS-1200401407223048: Fixed custom date picker behavior and placed it next to the Pipeline filter.
- AS-1200126969897499: Task ID is now correctly displayed in the UI.
And other bug fixes.
This version was released on May 26, 2021.
Added Deequ + PySpark integration.
Added multiple metrics support in the Favorite Metric widget.
Home Screen selection option is now available in the Settings Page.
It’s now possible to enable debug mode in PyDev from the CLI.
Slack notifications can now be received from the Databand UI.
- AS-1199965369989833: Tracking is now compatible with Airflow 2.0.
- AS-1200213466288282: Airflow tracking is running on Google Composer.
- AS-1200105822969991: Added support for liveness probe check in Airflow Monitor v2.0.
- AS-1200327639593933: Latest Pandas version (1.2.x) is now supported.
- AS-1200223284217045: Improved Alert screen.
- AS-1200345876827058: Alert rules can be enabled based on the Run env.
- AS-1200269646547649: “Last Active” widget moved to the bottom of the screen.
- AS-1200269646547644: Pipeline screen's default view is set to “List”.
- AS-1200310831229608: PyWheel build tasks are not displayed in the UI now.
- AS-1200315872631815: Changed the cursor in Favorite Charts.
- AS-1200305391366264: Fixed missing filters and Run names in the Alert API.
- AS-1200324498682162: Identified and fixed the issue with Sentry integration.
- AS-1200275951379648: Fixed incorrect handling of attempts.
- AS-1200290917819476: Changed Error Widget headline to “Top Errors”.
- AS-1200359124635729: Fixed error on wrong outputs’ schema in Orchestration.
This version was released on May 12, 2021.
Databand v.0.40.1 was released on May 13, 2021.
Stats widget changed to display the top failure rate Pipelines.
Server-side sorting for Pipeline stats widget.
New Anomaly Detection alert is triggered if Custom Metric grows too fast.
Airflow Monitor v.2.
It’s now possible to see an alert’s effect on the affected data.
- AS-1200044288700180: Drill down to Runs & Pipelines from the Dashboard.
- AS-1200259770331112: Click iteration and reporting stats added to Dashboard.
- AS-1200058439151159: Airflow monitor heartbeat indication.
- AS-1200243606518228: Errors Discovery Flow added for non-Python Operators.
- AS-1200276060368190: Negative values are now displayed in Dashboard charts.
- AS-1200285224722668: Fixed Docker version incompatibility issue (extended the number of supported versions).
- AS-1200275951379659: Fixed Kubernetes execution bug when tasks were wrongly marked as canceled on the end of Pipelines.
This version was released on April 28, 2021.
Auto-log paths became available for Spark JVM I/O (Alpha version).
It’s now possible to trigger a numeric alert when an anomaly is detected.
Added alerts integration with Opsgenie.
- AS-1199534439165430: Tracking system error handling is out - enjoy safe tracking without the performance penalty.
- AS-1200183406021460: See how a newly defined alert would have affected your past runs.
- AS-1200183406021459: Define sensitivity and the number of runs for anomaly detection.
- AS-1200194910187394: Slack alert headline now leads to the Alert page and displays the number of firing alerts.
- AS-1200141500926246: Fixed the URL in Slack message alerts.
- AS-1200194910187389: Slack alert’s link is now leading to the first alert.
This version was released on April 15, 2021.
Added Favorite Metrics widget.
Added reference date and fix time range widget.
Added support for Date picker in the UI.
Added Python Spark Metrics listener (similar to JVM).
New alerting mechanism for on-prem users (available on demand).
Zero-cost integration of SQL Airflow DAGs with Snowflake (in DBND SDK).
- AS-1199510429341979: Configured POC alerting.
- AS-1199592642123487: Run name/run description is now editable from the UI.
- AS-1199917986044490: It’s now possible to access the latest webserver logs via the webserver.
- AS-1200092086191209: Added a descriptive error message that appears when installing Airflow Monitor not in the same folder as Airflow.
This version was released on March 31, 2021.
Search bar is back in the details view.
Users who have Airflow with Docker Compose environment can now use the Databand cloud solution.
Favorite charts now appear in a separate tab in the Runs page, making it possible to see more metric information in one place at the same time.
A flag for switching on\off the transmission (sending) of the source code was added.
- AS-1199921442266004: AlertDef history now displays seconds in the UI.
- AS-1199995565427900: Version and Environment badges are now displayed on the Login page.
- AS-1200093251774759: Graphic info card in the Dashboard is now black.
- AS-1200127804370172: Quick updates tab added to the integration page.
QueryApi.force_paginationis now enabled by default.
- AS-1200126969897487: Remote logs are truncated.
- AS-1200058448826966: Support for "zero computational cost" data insights and preview added.
- AS-1200129802903561: Databand logo back on the Login page.
- AS-1200111167427102: Time format bug is fixed.
- AS-1199965226804188: The search bar is back in Details view.
- AS-1200091724058557: Runs and Task by Start Time are a stacked graph.
This version was released on March 18, 2021.
Introducing the Cloud Environment of Databand Web server.
New Design and filters for Run List including sticky filters.
Project filters now support Airflow tags.
Azkaban projects are now supported.
You can now track Log paths from Spark.
Extended tracking of Operational Health in the Dashboard: see Runs and Task Runs over time.
- AS-1199965687445132: Added columns for the task table in the Dashboard.
- AS-1199917986044517: Added Spark LogPath.
This version was released in early March 2021.
Added orchestration error reporting.
Project filters added in the Databand UI (phase 1).
Multiple Azkaban instances are now supported.
- AS-1199562651279659: Azkaban Client SDK-related improvement - support of multiple Azkaban instances.
- AS-1199873906435234: Airflow System Grafana Dashboard bugs fixed.
- AS-1199965687445132: Task and task attempt info is now displayed in the dashboard.
- AS-1199894674400827: Fixed the alert list checkbox color.
- AS-1199700822487533: Changed the default sync name from
http://to an empty string.
- AS-1199593767695403: Removed the wrong config values from databand-core (Airflow).
- AS-1199685842345221: Added ability to merge configuration from task class level and a selected section.
- AS-1199198703043527: Grafana dashboard can now present system metrics for Airflow monitoring.
Updated 8 months ago