Catalogs on control plane#
A catalog on Starburst control plane is the same kind of catalog that you know from SEP: a name, a connector, and the properties of the connector. The difference is where it lives. In managed mode, every catalog lives in control plane, and your clusters get their catalogs from it. This topic describes what is different. For the properties of each connector, see the SEP documentation.
Where catalogs live#
In managed mode, every catalog lives in control plane. A managed cluster has no catalogs of its own, and you cannot set
catalog.managementon it.You do not assign catalogs to clusters or to cluster groups. Every cluster sees every catalog. Routing rules decide where a query runs, and access control decides who can use a catalog.
Each catalog is governed or delegated. The setting indicates what type of access control authorizes the catalog. It does not change where the catalog lives.
To use catalogs on control plane, set control-plane.enabled to true on
control plane, and run each cluster in managed mode.
How clusters get catalogs#
When a client sends a query, the coordinator of the cluster compares its catalogs with the catalogs in control plane. It fetches the definitions that are missing or changed, and it creates the connectors from them. This happens at the start of each transaction, and a single statement is one transaction.
A cluster must reach control plane. If it cannot, its queries fail. If the
coordinator cannot load a consistent set of catalogs after several attempts, the
query fails with the error CATALOG_UNAVAILABLE.
A change to a catalog applies to the next transaction. A transaction that is already running keeps the catalogs that it started with.
Create and change catalogs#
Create catalogs with SQL in the control plane query editor. The statements
CREATE CATALOG, ALTER CATALOG, DROP CATALOG, SHOW CATALOGS, and SHOW CREATE CATALOG work as they do in SEP, and they act on the catalogs of
control plane. For example:
CREATE CATALOG sales USING postgresql
WITH (
"connection-url" = 'jdbc:postgresql://db.example.com:5432/sales',
"connection-user" = 'reader',
"connection-password" = '${ENV:SALES_DB_PASSWORD}'
);
The Catalogs pane, under Data in the main navigation, lists your
catalogs. The pane appears only when control-plane.enabled is true. It shows
the catalog name, type, and the authorization of each catalog. From the menu of
a catalog, the owner role can
switch the authorization. A role that owns the catalog and holds the new owner
role, or that has the MANAGE_SECURITY privilege, can change the owner. See
Governed and delegated catalogs. The page also has a Files tab. See
Catalog files.
Names and validation#
A catalog name must be lowercase and not empty. You cannot use the names
system and starburst. Control plane checks the name before it resolves
secrets and before it checks the connector.
Before control plane saves a catalog, it builds a temporary connector on a healthy cluster, to validate the properties. To create a catalog, you need at least one healthy managed cluster that is connected to control plane. If the validation fails, control plane returns the message “Catalog validation failed”, and it does not save the catalog. The message has no text from the connector, because connector errors can contain secrets.
Privileges#
The following table lists what you need for each task. For how ownership and privileges work in Starburst access control, see Ownership and Privileges. For the privileges for all control plane features, see Privileges for Control Plane features.
Task |
Requirement |
|---|---|
Create a catalog |
The |
Change, rename, or drop a catalog |
You own the catalog. |
Switch the authorization of a catalog |
You own the catalog. |
Change the owner of a catalog |
Your current role owns the catalog and holds the new owner role, or you have
the |
Show the definition of a catalog with |
You own the catalog. |
Manage catalog files |
The |
When you create a catalog, control plane grants its owner the privileges
CREATE_SCHEMA and CREATE_TABLE, and SELECT, INSERT, UPDATE, and
DELETE, on the catalog. If you transfer the ownership to another role, the
former owner keeps its existing grants.
SHOW CREATE CATALOG hides credentials, even from the owner. A secret reference
shows as you wrote it, and a sensitive value that you wrote as a literal shows
as [REDACTED]. To create the catalog again from the output, use new secret
references.
Secrets#
Use a secret reference in a catalog property, and not the value. Control plane resolves the references with its secrets providers when you create or change the catalog, when a cluster loads it, and when it refreshes the secrets. Control plane does not store the resolved values. A reference to an environment variable or to a file must resolve on control plane, not on the cluster. For the setup of a provider, see Configure a secrets provider.
Control plane refreshes the secrets of its catalogs every five minutes by
default. When a secret changes, the version of the catalog advances, and the
clusters load the new version, without an edit from you. To change the interval,
set catalogs.catalog-refresh-interval.
Properties#
Control plane has the following properties for catalogs:
Property |
Description |
Default |
|---|---|---|
|
How often control plane refreshes the secrets of its catalogs. The minimum
is |
|
|
How often the cleanup of catalogs runs. The minimum is |
|
|
Superseded catalog versions that are older than this are eligible for
deletion. The range is |
|
|
The maximum size of a file that you upload for use in catalog properties.
The maximum value is |
|
|
How long replaced or deleted catalog file contents stay available to
clusters that run older catalog versions. It must be longer than
|
|
A cluster has three properties for how it loads catalogs from control plane. The
previous names starburst.portal-catalog.* also work.
Property |
Description |
Default |
|---|---|---|
|
The time after which a stale catalog can be unloaded. The minimum is |
|
|
The number of threads that load catalogs. |
|
|
How often the cluster checks for stale catalogs. The minimum is |
|
Troubleshooting#
A catalog is missing on a cluster#
Check that the cluster runs in managed mode and can reach control plane. The coordinator loads the catalogs from control plane at the start of each transaction. See Connect a cluster to control plane.
A catalog is not created, and the message is “Catalog validation failed”#
Control plane could not build the connector with your properties. The message hides the error of the connector on purpose. Check the connection properties, and check that your secret references resolve on control plane. Make sure that a healthy cluster can reach the data source.
A catalog name is rejected#
Use a lowercase name that is not empty, and not system or starburst.
SHOW CREATE CATALOG fails with an access error#
Only the owner of the catalog can show its definition. See Privileges.
A governed catalog is rejected because of connector access control#
A governed catalog cannot use the access control of its connector. See Governed catalogs and connector access control.
A coordinator skips a catalog#
A coordinator skips a catalog with an invalid name, and it logs the ID and the version of the catalog. The other catalogs keep working. Correct or delete the catalog in control plane.