Starburst Documentation
Introduction
  • Overview
  • Concepts
  • Architecture
  • Choose your Starburst product
  • Data sources and catalogs
  • Data lakes
  • Apache Polaris
  • Using object storage systems
  • Metastore options
  • Security
  • Querying data with Starburst
  • Using SQL
  • Migrating analytics to Starburst
  • Query performance
  • Performance features

    • introduction > Metastore options

    Metastore options #

    Metastores provide management for metadata, which is data about tables and table contents. Metastores map files in distributed object storage cloud locations such as AWS S3 and HDFS into tables, and provide mapping for schemas, partitions, data types, and more. The original metastore was HMS, the Hive Metastore Service, which combines a relational database for metadata storage with active metadata management. The concept of a metastore has expanded significantly into services such as the Unity Catalog, which offers management of multiple table types in multiple cloud or local storage locations.

    A metastore service must be specified for Starburst Enterprise platform (SEP) and Starburst Galaxy for numerous use cases, including the following:

    The following metastore options are available:

    Starburst Enterprise #

    • Starburst Metastore is a self-hosted, Glue-compatible metastore, deployable in Kubernetes environments using Helm. Starburst Metastore lets you run self-managed SEP clusters with Glue-compatible metadata management without requiring access to AWS infrastructure.

    • HMS SEP includes a Helm chart to manage your own Hive Metastore for the cluster in Kubernetes. You can also run and connect your self-managed Hive Metastore Service deployment, which is based on the Thrift API.

    • AWS Glue For catalogs using the Hive, Delta Lake, or Iceberg connector with data stored on AWS, you can use AWS Glue.

    • Unity Catalog For catalogs using the Hive, Delta Lake, or Iceberg connectors, you can use the Unity Catalog. For details, see one of the following table-specific pages:

    • Iceberg-specific metastores Apache Iceberg defines a REST-based Catalog API for managing table metadata and performing catalog operations. Implementations and features supported by SEP include the following:

      • Apache Polaris Polaris is an open-source catalog for Apache Iceberg that implements the Iceberg REST API.

      • Amazon S3 Tables The AWS S3 Tables feature defines a table bucket that contains managed Iceberg tables. An S3 Tables installation can be configured to connect to an AWS Glue metastore. SEP supports S3 Tables through the Iceberg REST Catalog API.

      • OAuth 2.0 token pass-through is supported on AWS by the Iceberg REST catalog implementations listed above, and also by the Unity catalog.

      • JDBC catalog SEP supports the Iceberg JDBC catalog when using the Iceberg connector. Note that metadata tables must already exist when using the Iceberg JDBC catalog.

    Starburst Galaxy #

    Metastores supported by Starburst Galaxy include the following options.

    For Amazon S3, Azure Data Lake Storage, and Google Cloud Storage catalogs, you can select either:

    • Starburst Galaxy metastore Galaxy provides a built-in metastore that requires no additional installation. (Note that this is not the same as the Starburst Metastore supported by SEP.)

    • HMS You can run your own self-managed Hive Metastore Service deployment and connect to its URI.

    AWS Glue support

    • AWS Glue For S3 catalogs, you can also specify your configured AWS Glue instance.

    Some Galaxy catalogs connect directly to specific metastores or APIs instead of to object storage catalogs. These metastores are presumed to be configured to support one or more existing object storage tables.

    • Unity Use Galaxy’s Unity catalog to connect to managed or external tables on AWS, Azure, or Google Cloud.

    • Polaris Use Galaxy’s Apache Polaris catalog to connect to Iceberg tables through the Iceberg REST API on AWS S3 or Google Cloud.

    • S3 Tables Use Galaxy’s Amazon S3 Tables catalog to connect to Iceberg tables in an AWS table bucket through the Iceberg REST API on AWS S3. S3 Tables are presumed to be configured to use AWS Glue as to manage metadata.

    • Lakekeeper Use Galaxy’s Lakekeeper catalog to connect to Iceberg tables on Azure Data Lake Storage through Lakekeeper.