User Guide
  • Welcome to Lightup!
  • Quick start
  • Use the Lightup app
  • Prepare a workspace for data quality monitoring
    • Add users to Lightup
    • Manage app users
  • Manage datasources
    • Connect to Athena
    • Connect to BigQuery
    • Connect to Databricks
    • Connect to Hive
    • Connect to Incorta
    • Connect to Microsoft SQL
    • Connect to Oracle
    • Connect to Postgres
    • Connect to Redshift
    • Connect to Snowflake
  • Prepare tables for data quality metrics
  • Configure data quality metrics
    • Aggregation metrics
    • Compare aggregate metrics
    • Conformity check metrics
    • Data delay metrics
    • Data volume metrics
    • Distribution metrics
    • Null percent metrics
    • Row by row metrics
    • SQL metrics
  • Monitor metrics for data quality variations
    • Edit a monitor
  • Manage data quality incidents
  • Send alerts to responders
  • Review data quality summaries
    • Request data exports
  • Add an integration
    • Alation
    • Email List
    • Flock
    • Jira
    • Mattermost
    • Microsoft Teams
    • PagerDuty
    • Slack
  • Databricks Partner Connect - Start Here
  • Deploy Lightup
    • Deployment models
    • Lightup Hybrid
    • Lightup Enterprise
    • Lightup Cloud
  • Security and data privacy
  • FAQ
Powered by GitBook
On this page
  • Lightup account setup: Databricks
  • Connector settings
  1. Manage datasources

Connect to Databricks

Steps to prepare and connect to Databricks

PreviousConnect to BigQueryNextConnect to Hive

Last updated 2 years ago

Lightup account setup: Databricks

Make sure that your Databricks workspace meets the following requirements, and gather the following information:

Step 1: Get a personal access token. Lightup connects to a Databricks All Purpose Compute Cluster using a personal access token. If you don't already have one, you can generate one in Databricks on the Access Tokens tab of the User Settings page.

Step 2: Get connection details for the compute cluster

  2. Choose a cluster to connect to.

  3. Navigate to Advanced Options.

  4. Click on the JDBC/ODBC tab.

  5. Copy the Server Hostname and HTTP Path field values.

Step 3: Get a Databricks personal access token

  1. Open your Databricks workspace.

  3. Click User Settings.

  4. Go to the Access Tokens tab.

  5. Click the Generate New Token button.

  6. Optionally enter a description (comment) and lifetime.

  7. Click the Generate button.

  8. Copy the generated token and store in a secure location.

5. Get your Server Hostname and HTTP Path

Do one of the following:

  • If you're using a SQL endpoint:

    2. Choose an endpoint to connect to.

    3. Navigate to the Connection Details tab.

    4. Copy the Server Hostname and the HTTP Path.

  • If you're using a compute cluster:

    2. Choose a cluster to connect to.

    3. Navigate to Advanced Options.

    4. Click on the JDBC/ODBC tab.

    5. Copy the Server Hostname and the HTTP Path.

Connector settings

  • Workspace URL - The Server Hostname of the compute cluster

  • HTTP Path - The HTTP Path of the compute cluster.

  • Token - The personal access token

Click Compute in the sidebar.

A Databricks or a Databricks SQL .

A Databricks .

If you’re using a SQL endpoint, the . Specifically, you will need the Server Hostname and HTTP Path field values.

If you’re using a cluster, the . Specifically, you will need the Server Hostname and HTTP Path field values.

Click Settings in the lower left corner of your Databricks workspace.

Click SQL Endpoints in the sidebar.

Click Compute in the sidebar.

cluster
endpoint
personal access token
SQL endpoint connection details
cluster connection details
compute icon
User Settings Icon
Last step to generate a personal access token for Databricks, with optional "Comment" and "Lifetime" settings
compute icon
Endpoints Icon
The "Generate New Token" command highlighted on the "Access Tokens" tab of the Databricks "User Settings" page