Manage datasources

Set up accounts and connections

The procedures on this page require a Workspace Admin role.

To monitor your data quality, you connect to each datasource you want to monitor. To make that possible, you'll need a Lightup user account for each of them.

Some datasources use existing database connection APIs (such as ODBC/JDBC), and others have proprietary APIs. The details of connecting to a datasource vary by platform.

A datasource typically requires an account for Lightup in the target data asset (such as a database). Lightup supports numerous data platforms. Find steps on the corresponding subpage to prepare the lightup_user account before you connect to the datasource.

Connect a datasource

1. In the left pane, open a workspace menu and select Datasources. Then, in the main form select +.

2. Enter a Datasource Name, then select a Connector Type.

3. Provide connection information for the connector you chose. For details, see the corresponding subpage.

4. After entering the required settings and any optional settings that apply, below the Connector Setting section select Test Connection.

5. After a successful connection test, select Save.

6. Your new datasource appears in the list of available datasources. By default, these are listed in alphabetical order, so you might have to scroll or change the sort order to see your new datasource.

Connector subpages

• Athena

• BigQuery

• Databricks

• Hive

• Microsoft SQL

• Oracle

• Postgres

• Redshift

• Snowflake

Set up query governance

Not all query governance features are available for every data source connector. You might not see the feature, or it might appear but not be active. We're enabling query governance for more connectors. Look to the connector subpages for current information about query governance.

Open the Query Governance settings

In the left nav, on your workspace menu, select Query Governance. The main form displays Query Governance settings on the left, and a view of query performance history on the right.

Set query governance for a datasource

1. Open the Query Governance settings.

2. Under Datasources, choose a datasource to govern.

3. Set your governance settings:

• Query timeout - use the toggle to turn this governor on and off. If Query timeout is on, any Lightup query (the basis of metrics) that hasn't returned a value within the interval specified by Terminate after (number with time unit) stops running (fails)

• Query date range limit- use the toggle to turn this governor on and off. If Query date range limit is on, any Lightup query (the basis of metrics) that attempts to summarize data over a period that's longer that what's specified by Limit range to (number with time unit) is terminated (canceled).

Review query history

The Query History view shows totals for: queries Completed, Failed, Cancelled, and Processing. This information helps you decide when to turn on/off query governors on active datasources. For example, if a lot of Lightup queries are failing, the Query timeout might be set too low.​

Query history for a data asset