BigQuery Integration

How to connect Databand to a BigQuery project.

You can connect Databand to a BigQuery project using a JSON key file of a service account that has the following roles (you can find more info about the roles in the official Google BigQuery documentation:

  1. BigQuery Data Viewer – To get table metadata, schema, and preview
  2. BigQuery Resource Viewer – To get transactions’ metadata
  3. BigQuery Job User – To run queries for getting metadata

Connecting to BigQuery

In the Databand web UI:

  1. Go to Integrations tab
  2. Select BigQuery and click ‘Connect’
  1. Follow the on-screen instructions to create a Service Account
  1. Create and upload the Security Key
  1. Follow the instructions in the section below to start monitoring the data in your BigQuery project

Using BigQuery with Databand

After connecting to BigQuery, Databand will the metadata below from the last 12 hours, and will perform metadata collection every 6 hour on the previous time window.

  1. BigQuery tables – in Databand, they are called ‘datasets’.
    The following data is saved and exposed in the tables/datasets:

metadata over time, which includes

  • table size
  • row_count
  • schema
  • last update time
  • creation date of creation
  1. BigQuery jobs – in Databand, they are called transactions (see more here)

Note: Currently, only the BigQuery QueryJob type can be tracked using Databand. CopyJob, LoadJob, ExtractJob are not supported yet.

The following data is saved and exposed on transactions:

metadata, which includes:

  • query
  • create date
  • start date
  • end date
    operations, where Databand analyzes the query plan to determine the different operations (read/ write) performed on different datasets and saves the record count of each operation.

Note: Only the operations data is currently exposed. In the future, a connection between an operation that happened in the DBND SDK and an operation that came from the datasource monitor is planned.

Did this page help you?