Starburst AI agent-based data product enrichment#

Starburst supports data product enrichment using the Starburst AI agent. This feature automatically generates descriptions for data products, datasets, and columns. It uses metadata and the relationships between data elements, and, optionally, sampled data and column statistics from the datasets.

Requirements#

To use the data products enrichment feature, you need:

A valid AI workflows license.
A valid Agentic layer license.
Access to at least one configured language AI model.
Write permissions for data products.
Read permissions for the data product’s datasets when using data sampling.

Configuration#

To configure the enrichment feature, add the following property to your coordinator configuration file:

starburst.agent.enabled=true

Profiling configuration properties#

The following properties control the queries that enrichment runs against your datasets when you allow data sampling. Add relevant properties to the coordinator configuration file.

Enrichment profiling configuration properties#
Property name	Description	Default
`enrichment.sample-row-limit`	Maximum number of rows sampled per dataset. Valid values range from `1` to `1000`.	`100`
`enrichment.profiling-query-timeout`	The maximum duration allowed for each sampling or statistics query. When a query exceeds this limit, it is canceled and enrichment continues without that query’s results.	`30s`
`enrichment.profile-cache-ttl`	The duration sampled data and statistics are cached per user after they are collected. Set to `0` to disable caching.	`10m`
`enrichment.aggregation-sample-percentage`	Percentage of each dataset scanned by the statistics-aggregation query. Valid values range from `0.01` to `100`. A value below `100` applies `TABLESAMPLE BERNOULLI` to lower query cost. The measured counts reflect only the sampled rows rather than the whole dataset.	`100`
`enrichment.profiling-columns-per-query`	Number of columns projected per sampling or aggregation query. Wider datasets are profiled with several queries, because a single projection over hundreds of columns exceeds `ai.agent.max-result-set-size`. Valid values range from `1` to `200`.	`50`
`enrichment.semantics-columns-per-request`	Number of columns described per AI model request. Wider datasets are split across several requests so that the prompt and the expected response both fit the AI model’s context.	`50`

Enrich a data product#

After confirming access and configuration, follow these steps to generate metadata for a data product:

In the Data products tab of the Starburst Enterprise web UI, select an existing data product.
Click the Enrich with AI button to open the Context & permissions section of the Enrich data product dialog.
Optionally, describe the data product in the Business context field, for example its business purpose, the meaning of domain-specific codes, or the conventions its datasets follow. This text is passed to the model with each generation request in this run. It is limited to 2000 characters, and it is not saved to the data product.
Optionally, enable Allow data sampling so the agent reads a bounded data sample, measures column statistics, and sends them to the selected AI model. See Data sampling and profiling. This setting defaults to off and is not saved. You must enable it on each run.
Click Next to open the Datasets section.
From the drop-down menu, choose an AI model.
Click AI generate all datasets to enrich each dataset within the data product:
- Column descriptions are generated from column names and types, any existing descriptions, the SQL that defines the dataset, context such as dataset and data product names, and the business context you provided. When sampling is allowed, they are also generated from sampled values and column statistics. Measured column facts are appended to each description as plain key: value lines. See Generated column metadata.
- Dataset descriptions are generated based on column descriptions, existing dataset descriptions (if available), and context.
To generate metadata for a specific dataset, select it, then choose AI generate this dataset from the AI generate all drop-down menu and click AI generate this dataset.

Use the Columns and Show data tabs to review the generated column descriptions and to preview the dataset’s data.
Click Next.
In the Data product details section, click Generate values for all fields.

This generates a description for the data product, based on its name, current description (if any), and its datasets and their descriptions. The description also includes a block of plain-text lines stating what the agent could determine with confidence about the data product as a whole:
- relationships: foreign-key to primary-key joins between datasets, each with a ready-to-use SQL LEFT JOIN clause. These are inferred from the SQL that defines the datasets, from column names, and, when sampling is allowed, from distinct-value counts.
- mandatory_table_usage_rules: topics that must be answered from a specific dataset.
- no_mandatory_table_usage_rules_match: the default behavior and refusal message for questions that are out of scope.
The Summary and Tags are based on the generated description. Only tags that already exist in your account are suggested.
Click Save.

Generated descriptions are suggestions. Review and edit them before saving; anything the agent could not determine with confidence is left out rather than guessed.

Data sampling and profiling#

Note

This feature is available as a public previewin Starburst Enterprise. Contact your Starburst account team with questions or feedback.

When a user enables Allow data sampling, the agent runs the following queries per dataset before generating descriptions, using the requesting user’s own privileges. Enrichment never exposes data the user cannot already query. Row filters and column masks apply.

A sample of up to enrichment.sample-row-limit rows. Long values are truncated.
SHOW STATS for the dataset, which reports the connector’s statistics estimates.
An aggregation query that measures per-column row counts, distinct values, null counts, minimum and maximum values, maximum string length, and mean, standard deviation, and median for numeric columns. Because SHOW STATS returns no values for logical views, these measured statistics are the usual source for data products built from views.
A query against information_schema.columns to detect columns declared NOT NULL.

Caution

Sampled values and statistics are included in the prompts sent to the selected AI model, and measured facts are written into the generated column descriptions. Both can contain actual data values, so consider the exposure before you enable sampling on datasets that hold sensitive or regulated data.

Profiling is best effort. Queries that fail or time out are skipped, and enrichment continues with whatever was collected, or from metadata alone if nothing was collected. Results are cached per user for enrichment.profile-cache-ttl, so repeated generation within that duration does not re-query the data.

When data sampling is not allowed, no sampling, statistics, or information_schema queries are run, and descriptions are generated from metadata only.

Generated column metadata#

Column descriptions consist of the AI model’s prose description followed by the measured fields, written as plain key: value lines in the description field. Fields that cannot be determined are omitted, and no measured fields are appended when the dataset is not profiled:

Column metadata fields#
Field	Description
`data_type`	The column’s data type.
`cardinality`	`low` when the column has fewer than 15 distinct values, otherwise `high`.
`distinct_count`	Number of distinct values.
`null_count`	Number of null values.
`nullable`	`true` only when nulls are observed, `false` only when the column is declared `NOT NULL`. It is omitted when neither is established.
`max_length`	Length of the longest value, for string columns.
`value_stats`	Mean, median, and standard deviation, for numeric columns.
`sample_values`	Up to 10 sampled values, for high-cardinality columns.
`exhaustive_values`	The column’s complete set of values, for low-cardinality columns whose sampled values cover every distinct value.
`value_range`	Minimum and maximum values, for high-cardinality columns.

Enrich a specific field#

To enrich a specific dataset field using the Starburst AI agent:

Navigate to the Data products tab and select a data product.
Click Enrich with AI, then click Next to open the Datasets section.
Select the dataset, then generate metadata for the field you want:
- For the dataset description, click the AI generate button in the Description field.
- For every column at once, click AI generate all columns above the columns table.
- For a single column, open the column’s actions menu in the columns table and select Generate description. You can also select Edit description to write the description yourself.
In the Data product details section, the Summary, Description, and Tags fields each have their own AI generate button.

Reset dataset fields#

You can reset metadata fields during the initial AI generation, either for all datasets or for specific ones. Resetting restores the values last saved to the data product; it does not undo saved changes.

To reset a single field within a dataset, navigate to the field and click the corresponding autorenew reset button. For a single column, open the column’s actions menu in the columns table and select Refresh description.

To reset all fields in a specific dataset:

Navigate to the Data products tab and select a data product.
Click Enrich with AI, then click Next to open the Datasets section.
Select the dataset you want to reset.
From the AI generate all datasets drop-down menu, select Reset this dataset.
Click autorenew Reset this dataset.
Click Next.
In the Data product details section, select Reset values for all fields from the Generate values for all fields drop-down menu.
Click autorenew Reset values for all fields.
Click Save.

To reset all fields across all datasets:

Navigate to the Data products tab and select a data product.
Click Enrich with AI, then click Next to open the Datasets section.
From the AI generate all datasets drop-down menu, select Reset all.
Click autorenew Reset all to reset all fields across all datasets in the selected data product.
Click Next.
In the Data product details section, select Reset values for all fields from the Generate values for all fields drop-down menu.
Click autorenew Reset values for all fields.
Click Save.