Catalogs on control plane#

A catalog on Starburst control plane is the same kind of catalog that you know from SEP: a name, a connector, and the properties of the connector. The difference is where it lives. In managed mode, every catalog lives in control plane, and your clusters get their catalogs from it. This topic describes what is different. For the properties of each connector, see the SEP documentation.

Where catalogs live#

  • In managed mode, every catalog lives in control plane. A managed cluster has no catalogs of its own, and you cannot set catalog.management on it.

  • You do not assign catalogs to clusters or to cluster groups. Every cluster sees every catalog. Routing rules decide where a query runs, and access control decides who can use a catalog.

  • Each catalog is governed or delegated. The setting indicates what type of access control authorizes the catalog. It does not change where the catalog lives.

To use catalogs on control plane, set control-plane.enabled to true on control plane, and run each cluster in managed mode.

How clusters get catalogs#

When a client sends a query, the coordinator of the cluster compares its catalogs with the catalogs in control plane. It fetches the definitions that are missing or changed, and it creates the connectors from them. This happens at the start of each transaction, and a single statement is one transaction.

A cluster must reach control plane. If it cannot, its queries fail. If the coordinator cannot load a consistent set of catalogs after several attempts, the query fails with the error CATALOG_UNAVAILABLE.

A change to a catalog applies to the next transaction. A transaction that is already running keeps the catalogs that it started with.

Create and change catalogs#

Create catalogs with SQL in the control plane query editor. The statements CREATE CATALOG, ALTER CATALOG, DROP CATALOG, SHOW CATALOGS, and SHOW CREATE CATALOG work as they do in SEP, and they act on the catalogs of control plane. For example:

CREATE CATALOG sales USING postgresql
WITH (
  "connection-url" = 'jdbc:postgresql://db.example.com:5432/sales',
  "connection-user" = 'reader',
  "connection-password" = '${ENV:SALES_DB_PASSWORD}'
);

The Catalogs pane, under Data in the main navigation, lists your catalogs. The pane appears only when control-plane.enabled is true. It shows the catalog name, type, and the authorization of each catalog. From the menu of a catalog, the owner role can switch the authorization. A role that owns the catalog and holds the new owner role, or that has the MANAGE_SECURITY privilege, can change the owner. See Governed and delegated catalogs. The page also has a Files tab. See Catalog files.

Names and validation#

A catalog name must be lowercase and not empty. You cannot use the names system and starburst. Control plane checks the name before it resolves secrets and before it checks the connector.

Before control plane saves a catalog, it builds a temporary connector on a healthy cluster, to validate the properties. To create a catalog, you need at least one healthy managed cluster that is connected to control plane. If the validation fails, control plane returns the message “Catalog validation failed”, and it does not save the catalog. The message has no text from the connector, because connector errors can contain secrets.

Privileges#

The following table lists what you need for each task. For how ownership and privileges work in Starburst access control, see Ownership and Privileges. For the privileges for all control plane features, see Privileges for Control Plane features.

Task

Requirement

Create a catalog

The CREATE_CATALOG privilege. Your current role owns the catalog.

Change, rename, or drop a catalog

You own the catalog.

Switch the authorization of a catalog

You own the catalog.

Change the owner of a catalog

Your current role owns the catalog and holds the new owner role, or you have the MANAGE_SECURITY privilege.

Show the definition of a catalog with SHOW CREATE CATALOG

You own the catalog.

Manage catalog files

The CREATE_CATALOG privilege. To update or purge a file that catalogs use, you also need to own each of those catalogs.

When you create a catalog, control plane grants its owner the privileges CREATE_SCHEMA and CREATE_TABLE, and SELECT, INSERT, UPDATE, and DELETE, on the catalog. If you transfer the ownership to another role, the former owner keeps its existing grants.

SHOW CREATE CATALOG hides credentials, even from the owner. A secret reference shows as you wrote it, and a sensitive value that you wrote as a literal shows as [REDACTED]. To create the catalog again from the output, use new secret references.

Secrets#

Use a secret reference in a catalog property, and not the value. Control plane resolves the references with its secrets providers when you create or change the catalog, when a cluster loads it, and when it refreshes the secrets. Control plane does not store the resolved values. A reference to an environment variable or to a file must resolve on control plane, not on the cluster. For the setup of a provider, see Configure a secrets provider.

Control plane refreshes the secrets of its catalogs every five minutes by default. When a secret changes, the version of the catalog advances, and the clusters load the new version, without an edit from you. To change the interval, set catalogs.catalog-refresh-interval.

Properties#

Control plane has the following properties for catalogs:

Property

Description

Default

catalogs.catalog-refresh-interval

How often control plane refreshes the secrets of its catalogs. The minimum is 1m.

5m

catalogs.cleanup.interval

How often the cleanup of catalogs runs. The minimum is 1m.

1h

catalogs.cleanup.min-version-age

Superseded catalog versions that are older than this are eligible for deletion. The range is 1m to 7d.

2d

catalogs.files.max-size

The maximum size of a file that you upload for use in catalog properties. The maximum value is 512kB.

256kB

catalogs.files.revision-retention

How long replaced or deleted catalog file contents stay available to clusters that run older catalog versions. It must be longer than starburst.catalog.max.staleness on the clusters. The minimum is 1m.

72h

A cluster has three properties for how it loads catalogs from control plane. The previous names starburst.portal-catalog.* also work.

Property

Description

Default

starburst.catalog.max.staleness

The time after which a stale catalog can be unloaded. The minimum is 1m.

24h

starburst.catalog.max.loader.threads

The number of threads that load catalogs.

4

starburst.catalog.stale-catalog-check-interval

How often the cluster checks for stale catalogs. The minimum is 1m.

1m

Troubleshooting#

A query fails with the error CATALOG_UNAVAILABLE#

The coordinator could not load a consistent set of catalogs from control plane after several attempts. Check that the cluster can reach control plane, and then run the query again.

A catalog is missing on a cluster#

Check that the cluster runs in managed mode and can reach control plane. The coordinator loads the catalogs from control plane at the start of each transaction. See Connect a cluster to control plane.

A catalog is not created, and the message is “Catalog validation failed”#

Control plane could not build the connector with your properties. The message hides the error of the connector on purpose. Check the connection properties, and check that your secret references resolve on control plane. Make sure that a healthy cluster can reach the data source.

A catalog name is rejected#

Use a lowercase name that is not empty, and not system or starburst.

SHOW CREATE CATALOG fails with an access error#

Only the owner of the catalog can show its definition. See Privileges.

A governed catalog is rejected because of connector access control#

A governed catalog cannot use the access control of its connector. See Governed catalogs and connector access control.

A coordinator skips a catalog#

A coordinator skips a catalog with an invalid name, and it logs the ID and the version of the catalog. The other catalogs keep working. Correct or delete the catalog in control plane.