Configuring the Hive Metastore Service with CFT#

The SEP CloudFormation Template (CFT) supports several different methods for configuring a Hive Metastore Service (HMS), Hive metastore, and optionally a backing database in the cluster. The following SEP connectors and features require a HMS:

Note

SEP supports the Hive Metastore Service (HMS) version 3.13. HMS version 4.X is not supported.

You can set the MetastoreType to the following values.

Supported Hive metastore types#

MetastoreType

Description

None

(Default) No Hive metastore is configured.

Standalone (ephemeral)

Use a standalone Hive metastore instance in EC2

AWS Glue Data Catalog

Use the AWS Glue data catalog as an HMS

External MySQL RDBMS

Use an external MySQL RBDMS as a Hive metastore

External PostgreSQL RDBMS

Use an external PostgresSQL RBDMS as a Hive metastore

External Hive Metastore Service

Connect to an existing, external Hive Metastore Service.

No metastore#

The default configuration sets to MetastoreType to None and does not configure a Hive metastore.

Standalone (ephemeral) metastore#

By setting MetastoreType to Standalone (ephemeral), a separate EC2 instance is created by the CFT. It contains both the Hive metastore and its underlying RDBMS.

Note that information stored in such a metastore only lives as long as the SEP cluster. Because of that such configuration should be avoided on production system, while it is the best option to test Trino and the Hive connector.

AWS Glue data catalog#

By setting MetastoreType to AWS Glue Data Catalog, the Hive catalog uses the AWS Glue Data Catalog as its metastore service.

External MySQL RDBMS#

By setting MetastoreType to External MySQL RDBMS, a separate EC2 instance is created by the CFT. It runs a Hive Metastore Service that leverages an external MySQL RDBMS as its underlying storage.

This new instance requires network access to the external MySQL system. You must configure your networking and security groups accordingly. We recommend using AWS RDS, but you can use your own MySQL instance.

This configuration requires the following properties to be set:

  • ExternalMetastoreHost the host address of the MySQL service.

  • ExternalMetastorePort the port number of MySQL service. If 0 is set then 3306 (default MySQL port) is used.

  • ExternalRdbmsMetastoreUserName the MySQL user name

  • ExternalRdbmsMetastorePassword the MySQL user password

  • ExternalRdbmsMetastoreDatabaseName the MySQL database name that is used for storing Hive Metastore data.

RDBMS does not require any schema initialization other than database creation. It is well suited for MySQL provisioned with AWS RDS service.

External PostgreSQL RDBMS#

By setting the MetastoreType to External PostgreSQL RDBMS, a separate EC2 instance is created by CFT which runs a Hive Metastore Service. It leverages an external PostgreSQL RDBMS as its underlying storage.

This new instance needs network access to the external PostgreSQL system. You must configure your networking and security groups accordingly. We recommend using AWS RDS, but you can use your own PostgreSQL instance.

This configuration requires the following properties to be set:

  • ExternalMetastoreHost the host address of the PostgreSQL service.

  • ExternalMetastorePort the port number of PostgreSQL service. If 0 is set then 5432 (default PostgreSQL port) is used.

  • ExternalRdbmsMetastoreUserName the PostgreSQL user name

  • ExternalRdbmsMetastorePassword the PostgreSQL user password

  • ExternalRdbmsMetastoreDatabaseName the PostgreSQL database name that is used for storing Hive Metastore data.

RDBMS does not require any schema initialization other than database creation. It is well suited for PostgreSQL provisioned with AWS RDS service.

External Hive metastore service#

By setting MetastoreType to External Hive Metastore Service, the Hive connector uses an existing Hive Metastore Service.

This configuration requires the below properties to be set:

  • ExternalMetastoreHost the host address of the Hive Metastore Service.

  • ExternalMetastorePort the port number of the Hive Metastore Service. If 0 is set then 9083 (default Hive Metastore Service port) is used.