You can use a Google Cloud Storage (GCS) catalog to configure access to a Cloud Storage object storage hosted on Google Cloud.
The data can be stored using Iceberg, Delta Lake, Hive, or Hudi table formats.
The metadata about the objects and related type mapping needs to be stored in a metastore. You can use a Hive Metastore, or the built-in metastore.
Follow these steps to begin creating a catalog for Google Cloud Storage:
On the Select a data source pane, click the Google Cloud Storage icon.
The Catalog name is visible in the query editor and other clients. It is used to identify the catalog when writing SQL or showing the catalog and its nested schemas and tables in client applications.
The name is displayed in the query editor, and in the output of a SHOW
CATALOGS command. It is used to fully
qualify the name of any table in SQL queries following the
catalogname.schemaname.tablename syntax. For example, you can run the
following query in the sample cluster without first setting the catalog or
SELECT * FROM tpch.sf1.nation;.
The Description is a short, optional paragraph that provides further details about the catalog. It appears in the Starburst Galaxy user interface and can help other users determine what data can be accessed with the catalog.
Allow public access to the data in your object storage.
Provide the GCS JSON key to grant access to the object storage.
Create a JSON-formatted API key for your Google Cloud account.
Before you can query data in an object storage account, it is necessary to have a metastore service associated with that object storage.
For more information about object storage and the requirement for a metastore, see Using object storage systems.
You can use a Hive Metastore Service (HMS) to manage the metadata for your object storage. The HMS must be located in the same cloud provider and region as the object storage itself.
A connection to the HMS can be established directly, if the Starburst Galaxy IP range/CIDR is allowed to connect.
If the HMS is only accessible inside the virtual private cloud (VPC) of the cloud provider, you can use an SSH tunnel with a bastion host in the VPC.
In both cases, configure access with the following parameters:
Starburst Galaxy provides its own metastore service for your convenience. You do not need to configure and manage a separate Hive Metastore Service deployment or equivalent system.
In Metastore configuration, select Starburst Galaxy to set up and use the built-in metastore provided by Galaxy.
For Amazon S3 and Google Cloud Storage, create a bucket in your object storage account, and create a directory in that bucket. Provide that bucket name and directory name. This location is then used to store the metastore data associated with this S3 or GCS account.
For Azure ADLS, create a container in your storage account, and create a directory in that container. Provide this storage container name and directory name. This sets up the location used to store the metadata associated with this storage account.
The meanings of the two Allow controls are the same for a Starburst Galaxy metastore as for a separate Hive Metastore Service, described previously.
Note that deletion of the catalog also results in removal of the associated Starburst Galaxy metastore data.
Starburst Galaxy provides a simple way to specify the default table format for an object storage catalog. This applies to newly created tables, and does not convert any existing tables. The following table formats are supported:
If you are unsure which format to use, we recommend leaving the default
Iceberg format selected. More information is available in the discussion of
the format options on the
Once you have configured the connection details, click Test connection to confirm data access is working. If the test is successful, you can save the catalog.
If the test fails, look over your entries in the configuration fields, correct any errors, and try again. If the test continues to fail, Galaxy provides diagnostic information that you can use to fix the data source configuration in the cloud provider system.
Click Connect catalog, and proceed to set permissions where you can grant access to certain roles.
This optional step allows you to configure read-only access or full read and write access to the catalog.
Use the following steps to assign read-only access to all roles:
You can specify read-only access and read-write access separately for different sets of roles. That is, one set of roles can get full read and write access to all schemas, tables, and views in the catalog, while another set of roles gets read-only access.
Use the following steps to assign read/write access to some or all roles:
You can add your catalog to a cluster later by editing a cluster. Click Skip to proceed to the catalogs page.
Use the following steps to add your catalog to an existing cluster or create a new cluster in the same cloud region:
Click Add to cluster to view your new catalog’s configuration.
The Pending changes to clusters dialog appears when you try to add a catalog to a running cluster.
On the Clusters page, click the Update icon beside the running cluster, to add the catalog.
Now that your object storage catalog has been added to a cluster, you can run queries against Iceberg, Delta, Hive, and Hudi table formats using Great Lakes connectivity.
Is the information on this page helpful?
Is the information on this page helpful?