Catalog files#

Some connectors read files, such as a truststore, a keytab, an S3 security mapping, or a name-mapping file. On Starburst control plane, you upload these files to control plane, and you refer to them in the properties of a catalog. Control plane gives each node the exact file that the catalog needs.

Upload a file#

You need the CREATE_CATALOG privilege.

  1. In the sidebar, under Data, click Catalogs, and then click the Files tab. The tab does not appear if you do not have the privilege.

  2. Upload the file, and give it a name.

A file can have up to 256 kB. To change the limit, set catalogs.files.max-size, which has a maximum of 512 kB. Control plane stores the file in the backend database, encrypted with a key that it derives from gateway.clusters.encryption.secret.

Use a file in a catalog#

Refer to the file with ${catalog-file:<name>} in a property of the catalog. For example:

CREATE CATALOG lake USING hive
WITH (
  "hive.metastore.uri" = 'thrift://metastore.example.com:9083',
  "s3.security-mapping.config-file" = '${catalog-file:s3-mapping.json}'
);

You can use the reference as a part of a value, and several catalogs can share one file. Control plane keeps the reference as you wrote it. SHOW CREATE CATALOG shows the reference, and not the content.

Each coordinator and each worker fetches the file that its catalog version needs, checks it, and keeps a local copy that only its own account can read. If the fetch or the check fails, the catalog does not load. The node does not fall back to another version of the file.

Replace a file#

When you replace a file, control plane changes the version of every catalog that uses it, and the clusters load the new version in their next transaction. You need permission to change every catalog that uses the file. If you replace a file with the same content, nothing changes.

Control plane keeps the replaced content for catalogs.files.revision-retention, 72 hours by default, so that clusters that still run an older catalog version can load it. The retention must be longer than starburst.catalog.max.staleness on the clusters.

Delete a file#

You cannot delete a file that a catalog uses. Control plane refuses the request with HTTP status 409. Remove the reference from the catalog first.

Use the API#

The public API service catalogFile lists, creates, replaces, and deletes files, and it removes old versions with :purgeRevisions. Uploads use multipart/form-data. A replace must send the current syncToken, or control plane refuses it with HTTP status 409.