---
description: Prerequisites and process for initiating ongoing data exports
---

# Request data exports

{% hint style="warning" %}
The process described on this page requires a Lightup deployment that uses S3 or Azure Blob storage, as detailed below.&#x20;
{% endhint %}

Upon request, Lightup will begin regular exports of your data quality metrics to your cloud-based storage (currently, S3 or Azure Blob). You can then use the data exports for custom analysis, custom dashboard generation, and archiving purposes. If you use S3 for cloud storage, you can create a Redshift database and import the data there. &#x20;

### What data we export

When exporting is turned on, each day we export a a summary of all metric data points collected that day along with indicators for whether or not an incident was detected for each data point. The export goes in a new CSV file in your cloud storage (S3 bucket or Azure Blob Container), using the following columns with one row per incident logged. You can then use these exports as needed.&#x20;

{% hint style="info" %}
Items listed below have the format **columnName** - value source: _value data type._
{% endhint %}

* **eventTs** - datapoint timestamp: _seconds from Unix Epoch_
* **slice** - datapoint slice: _JSON object_
* **value** - datapoint value (what it represents depends on the type of metric): _floating point number_
* **metricUuid** - datapoint metric uuid in Lightup db: _string_
* **metricName** - datapoint metric name: _string_
* **metricType** - datapoint metric type (whatever we have in config.aggregation.type): _string_
* **metricAggregationWindow** - datapoint metric aggregation window (`hourly`, `daily`, `etc`.): _string_
* **metricTags** - datapoint metric tags: _Array\[string]_
* **sourceUuid** - datapoint metric source uuid in Lightup db: _string_
* **sourceName** - datapoint metric source name: _string_
* **schemaName** - datapoint metric schema name: _string_
* **tableUuid** - datapoint metric table uuid in Lightup db: _string_
* **tableName** - datapoint metric table name in Lightup db: _string_
* **columnUuid** - datapoint metric column uuid in Lightup db: _string_
* **columnName** - datapoint metric column name: _string_
* **incidentCount** - number of active incidents associated with the datapoint metric at eventTs time: _integer_
* **workspaceId** - workspace id of the metric: _uuid_
* **monitorUuid** - uuid of the monitor if datapoint was monitored: _string_
* **monitorName** - name of the monitor if datapoint was monitored: _string_
* **monitorTags** - monitor tags if datapoint was monitored: _string_
* **monitoredValue** - processed value that is compared with monitor bounds: _floating point number_
* **monitorLowerBound** - lower bound of the monitor if datapoint was monitored: _floating point number_
* **monitorUpperBound** - upper bound of the monitor if datapoint was monitored: _floating point number_
* **dataExtractionComments** - Description of any errors that were encountered during data extraction: _string_

### Start getting data exports

[Email support@lightup.ai](mailto:support@lightup.ai) and provide the following information, based on your cloud storage account. There are currently two options: S3 and Azure Blob.

* [Start exports to S3](#s3-exports-parameters) &#x20;
* [Start exports to Azure Blob](#undefined)

#### S3 exports - parameters

Include these details in your message:

* S3 **Bucket name** where you want the export files (Required)
* **AWS access key ID** (Optional)&#x20;
* **AWS secret access key** (Optional)
* **Region name** (Optional)

#### AWS Blob exports - parameters

Include these details in your message:

* Azure Blob **Container name** where you want the export files (Required)
* **Account name** (Required)
* **Account key** (Required)&#x20;

### Import S3 CSV files into a Redshift database

If you use S3 cloud storage for Lightup you can create a Redshift database and import the data from your CSV files, unlocking numerous analytical options.&#x20;

{% hint style="warning" %}
The schema of the exports is subject to planned change. If you create a database per the following steps, include this factor in your maintenance plans.
{% endhint %}

1. Create a Redshift database. For syntax, see [Amazon Redshift - CREATE DATABASE](https://docs.aws.amazon.com/redshift/latest/dg/r_CREATE_DATABASE.html).
2. Open Redshift query editor in AWS console.
3. Make sure Redshift IAM role is allowed to read from the source S3 bucket. For help using IAM roles, see [AWS Identity and Access Management - Using IAM Roles](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use.html).
4.  Create a table using following SQL:

    ```sql
    CREATE TABLE datapointsexport (
    eventTs timestamp,
    slice super,
    value float,
    metricUuid varchar,
    metricName varchar,
    metricType varchar,
    metricAggregationWindow varchar,
    metricTags super,
    sourceUuid varchar,
    sourceName varchar,
    schemaName varchar,
    tableUuid varchar,
    tableName varchar,
    columnUuid varchar,
    columnName varchar,
    incidentCount int,
    workspaceId varchar,
    monitorUuid varchar,
    monitorName varchar,
    monitorTags varchar,
    monitoredValue float,
    monitorLowerBound float,
    monitorUpperBound float,
    incidentData varchar,
    userDescription varchar,
    dataExtractionComments varchar
    );
    ```


5.  To import a CSV file, run the following SQL, but replace "datapointsexport" with the name of the CSV file (excluding the _.csv_ extension), "lightup-datapoints-dump" with your S3 bucket name, and the value of _iam\_role_:\


    ```sql
        COPY datapointsexport
        FROM 's3://lightup-datapoints-dump'
        iam_role 'arn:aws:iam::231612517276:role/RedshiftCopy'
        csv
        IGNOREHEADER 1
        DELIMITER ','
        EMPTYASNULL
        TIMEFORMAT 'epochsecs';
        
    ```

{% hint style="info" %}
For information about automating imports from S3 files into Redshift, see [A Zero-Administration Amazon Redshift Database Loader](https://aws.amazon.com/blogs/big-data/a-zero-administration-amazon-redshift-database-loader/). &#x20;
{% endhint %}
