Amazon S3 catalogs #

You can use an Amazon S3 catalog to configure access to an Amazon S3 object storage hosted on Amazon Web Services.

The metadata about the objects and related type mapping needs to be stored in a metastore. You can use Amazon Glue, a Hive Metastore Service, or the built-in metastore.

Configure a catalog #

To create an Amazon S3 catalog, select Catalogs in the main navigation and click Configure a catalog. Click on the Amazon S3 button in the Select a data source screen.

Define catalog name and description #

The Name of the catalog is visible in Query editor and other clients. It is used to identify the catalog when writing SQL or showing the catalog and nested schemas and tables in client applications.

The name is displayed in Query editor, and when running a SHOW CATALOGS. It is used to fully qualify the name of any table in SQL queries following the catalogname.schemaname.tablename syntax. For example, you can run the following query in the demo cluster without setting any context of catalog or schema.

SELECT * FROM tpch.sf1.nation;

The Description is a short, optional paragraph that provides more details about the catalog than the name alone. It appears in the user interface and can help other users to determine what data can be accessed with the catalog.

Enter catalog name and description

Read only #

Toggle the Read only switch as desired to enable only read operations on the data source, or allow data changes such as inserts, updates, and deletes.

Make this catalog read only

Authentication to S3 #

Select between Cross account IAM role or AWS access key to grant access to the object storage.

With Cross account IAM role, provide an alias for the role in Starburst Galaxy and the AWS IAM ARN.

With AWS access key, provide the AWS access key for S3 and the AWS secret key for S3.

Read External security in AWS to learn about configuring these details in the AWS console.

Metastore configuration #

AWS Glue #

You can use AWS Glue to manage the metadata about your object storage. Configure access to AWS Glue with the following parameters:

  • AWS Glue region
  • Access security with AWS access key for Glue and AWS secret key for Glue or Cross account IAM role

Read External security in AWS to learn about configuring these details in AWS.

Hive Metastore Service #

You can use a Hive Metastore Service (HMS) to manage the metadata about your object storage. The HMS must be located in the same cloud provider and region as the object storage itself.

A connection to the HMS can be established directly, if the Starburst Galaxy IP range/CIDR is allowed to connect.

If the HMS is only accessible inside the virtual private cloud (VPC) of the cloud provider, you can use an SSH tunnel with a bastion host in the VPC.

In both cases, configure access with the following parameters:

  • Hive Metastore host, the fully qualified domain name of the HMS server.
  • Hive Metastore port, the port used by the HMS, typically 9083.
  • Allow creating external tables, switch to indicate if new tables can be created in the objects storage and HMS from Starburst Galaxy with CREATE TABLE or CREATE TABLE AS.
  • Allow writing to external tables, switch to indicate if data management write operations are permitted.

Starburst Galaxy metastore #

A metastore is necessary to query object storage systems. Starburst Galaxy includes a metastore for your convenience. You do not need to use and manage a separate Hive Metastore Service deployment or equivalent system.

Use the Starburst Galaxy metastore in Metastore configuration by selecting I don’t have a metastore.

Provide the name of the object store bucket name and directory name. The location is then used to store the metastore data with the object storage.

Note that deletion of the catalog also results in removal of the Starburst Galaxy metastore data.

Save the catalog #

Click the Save catalog button, and proceed to add the catalog to a cluster. Then proceed to use SQL queries to analyze and work with the data source with SQL.