Azure Data Lake Storage catalogs #
You can use an Azure Data Lake Storage (ADLS) catalog to configure access to a Azure Data Lake Storage object storage hosted on Microsoft Azure.
The metadata about the objects and related type mapping needs to be stored in a metastore. You can use a Hive Metastore, or the built-in metastore.
Configure a catalog #
To create an ADLS catalog, select Catalogs in the main navigation and click Configure a catalog. Click on the Azure Data Lake Storage button in the Select a data source screen.
Define catalog name and description #
The Name of the catalog is visible in Query editor and other clients. It is used to identify the catalog when writing SQL or showing the catalog and nested schemas and tables in client applications.
The name is displayed in Query editor, and when running a SHOW
CATALOGS. It is used to fully qualify the
name of any table in SQL queries following the
catalogname.schemaname.tablename syntax. For example, you can run the
following query in the demo cluster without setting any context of catalog or
SELECT * FROM tpch.sf1.nation;
The Description is a short, optional paragraph that provides more details about the catalog than the name alone. It appears in the user interface and can help other users to determine what data can be accessed with the catalog.
Read only #
Switch the Read only checkbox as desired to enable read operations on the data source only, or allow data changes such as inserts, updates and deletes.
Authentication to ADLS #
Provide the GCS JSON key to grant access to the object storage.
The connection to the object storage requires the ABFS storage account name and access key.
Metastore configuration #
Hive Metastore Service #
You can use a Hive Metastore Service (HMS) to manage the metadata about your object storage. The HMS must be located in the same cloud provider and region as the object storage itself.
Configure access with the following parameters:
- Hive Metastore host, the fully qualified domain name of the HMS server.
- Hive Metastore port, the port used by the HMS, typically 9083.
Starburst Galaxy metastore #
A metastore is necessary to query object storage systems. Starburst Galaxy includes a metastore for your convenience. You do not need to use and manage a separate Hive Metastore Service deployment or equivalent system.
Use the Starburst Galaxy metastore in Metastore configuration by selecting I don’t have a metastore.
Provide the name of the object store bucket name and directory name. The location is then used to store the metastore data with the object storage.
Note that deletion of the catalog also results in removal of the Starburst Galaxy metastore data.
Save the catalog #
Click the Save catalog button, and proceed to add the catalog to a cluster to be able to query the data source.
Is the information on this page helpful?