Installation#

As of version 476-e, Starburst Gateway and Starburst data catalog are deployed together using a unified Helm chart called starburst-portal. This new deployment artifact simplifies installation and ensures compatibility with the latest features and fixes. The previous individual Helm chart for starburst-gateway is no longer supported.

This page provides instructions for installing and configuring Starburst Gateway.

Install Starburst Gateway with Helm.

Configuration#

The following sections describe the configuration requirements for Starburst Gateway.

Each component of Starburst Gateway has a corresponding node in the configuration YAML file.

Backend database#

Starburst Gateway requires a MySQL, PostgreSQL, or Oracle database.

The Gateway automatically initializes your database when you start it.

Starburst clusters#

Starburst Galaxy and the current and previous Starburst Enterprise LTS versions are compatible with Starburst Gateway.

Note

Starburst Gateway is not compatible with Trino or Amazon Athena.

SEP Configuration#

Starburst Gateway acts as a transparent proxy for one or more clusters.

Configure all clusters behind Starburst Gateway with the following settings.

Enable forward processing#

If you route all client and server communication through Starburst Gateway, enable process forwarded HTTP headers:

http-server.process-forwarded=true

Without this setting, initial requests may route correctly from your client to Starburst Gateway and then to Starburst Enterprise. However, subsequent requests for quests for query results use the cluster’s local URL instead of the Gateway URL, bypassing the Gateway entirely. If the cluster’s local URL is private to the network, these subsequent calls fail.

This setting is also required for SEP to authenticate when TLS terminates at the Gateway. Normally, SEP refuses to authenticate plain HTTP requests, but with http-server.process-forwarded=true, SEP authenticates over HTTP when the request includes X-Forwarded-Proto: HTTPS.

Configure header forwarding#

To prevent Starburst Gateway from sending X-Forwarded-* headers, set the following configuration property:

routing:
  addXForwardedHeaders: false

When proxying clusters that are behind firewalls or otherwise not directly routable from your client, set this property to true.

Environment variables#

Set environment variables in the command line or in your configuration file.

To manually set an environment variable in the command line:

export DB_PASSWORD=my-super-secret-pwd

To use an environment variable in your configuration file, reference it using the syntax ${ENV:VARIABLE}.

Routing rules#

Find more information in the routing rules documentation.

Logging #

To configure the logging level for classes, specify the path to the log.properties file by using log.levels-file in serverConfig.

For additional configurations, use the log.* properties from the logging properties documentation and specify the properties in serverConfig.

Additional statement maths#

The SEP client protocol initiates queries with a POST to v1/statement. Starburst Gateway incorporates this into its routing logic by extracting and recording the query ID from responses to these requests.

If you use a build of SEP that supports additional endpoints, configure Starburst Gateway to treat them like /v1/statement by adding them under additionalStatementPaths. The standard /v1/statement path is always included:

additionalStatementPaths:
  - '/ui/api/insights/ide/statement'

Load balancer#

To deploy multiple instances of Starburst Gateway behind a generic load balancer, you must the serverConfig to enable process forwarded HTTP headers:

serverConfig:
  http-server.process-forwarded: true

Proxy response size#

Starburst Gateway reads responses from SEP in bytes (up to 32MB by default). To configure a larger size, set the following configuration property:

proxyResponseConfiguration:
  responseSize: 50MB

Deploy Gateway using Starburst Portal#

Use the following steps to deploy Starburst Gateway on Kubernetes with Helm.

Access the Helm Chart#

The Starburst Portal Helm chart is available in the Starburst Helm chart project. It contains both Starburst Gateway and Starburst data catalog.

Use the following commands to access the chart:

helm registry login harbor.starburstdata.net/starburstdata
# Enter your credentials
helm pull oci://harbor.starburstdata.net/starburstdata/charts/starburst-portal --version <version>

Create a secret#

Use the following command to create a secret for Harbor registry authentication:

kubectl create secret docker-registry harbor-auth \
  --docker-server=harbor.starburstdata.net \
  --docker-username=<your-username> \
  --docker-password=<your-password> \
  --docker-email=<your-email>

Deploy a PostgreSQL database#

Use the following command to create a PostgreSQL pod:

kubectl apply -f val-posgres.yaml

Install Starburst Gateway#

Use the following command to install Starburst Gateway via the Portal chart:

helm upgrade --install starburst-portal starburst-portal-<version>.tgz -f gateway-config.yaml

Enable Starburst Gateway#

By default, Starburst Gateway is disabled in the Portal deployment.

To enable it:

gateway.enabled=true
gateway.config.file=config.yaml

The values.yaml file defines how Starburst Portal is configured during Helm deployment. The following example shows a basic configuration:

etcFiles:
  config: |
    node.environment=starburst_portal
    http-server.http.port=8080
    http-server.http.enabled=true
    http-server.https.port=8443
    http-server.https.enabled=false
    credentials-provider.type=file
    credentials-provider.credentials-file-path=/opt/starburst/etc/catalog-credentials.json
    persistence.database.url=jdbc:postgresql://postgresql:5432/<db/schema name>
    persistence.database.user=<db username>
    persistence.database.password=<db password>

Access the Gateway UI#

User the following command to access the Starburst Gateway UI:

kubectl port-forward service/starburst-gateway 8080:8080

Access the UI at http://localhost:8080.

Configure routing rules#

To implement static routing rules, create a configMap from your routing rules YAML definition:

kubectl create cm routing-rules --from-file your-routing-rules.yaml

Then mount it to your container:

volumes:
    - name: routing-rules
      configMap:
          name: routing-rules
          items:
              name: your-routing-rules.yaml
              path: your-routing-rules.yaml

volumeMounts:
    - name: routing-rules
      mountPath: "/etc/routing-rules/your-routing-rules.yaml"
      subPath: your-routing-rules.yaml

Ensure the mountPath matches the rulesConfigPath you specify in your configuration. The subPath is optional. Without the subPath, the file mounts at mountPath/<configMap key>.

Standard Helm options such as replicaCount, image, imagePullSecrets, service, ingress and resources are supported in helm/values.yaml.

Health checks#

Starburst Gateway periodically performs health checks and maintains an in-memory health status for each backend. When a backend fails a health check, the Gateway marks it as UNHEALTHY and stops routing requests to it.

The health status differs from the active/inactive state of a backend. The active/inactive state indicates whether a backend is on or off, while health status is determined programmatically by the health check process. Health checks only run on active backends.

Starburst recommends using either INFO_API or METRICS for your health check. Other options may be deprecated in the future.

For more details, see the health status section.

To select the type of health check, set the following configuration property:

clusterStatsConfiguration:
  monitorType: ""

Choose from the following health check types.

INFO_API (default)#

By default, Starburst Gateway uses the v1/info REST endpoint. A successful requires a 200 response with starting: false. Configure connection timeout parameters through the monitor node:

monitor:
  connectTimeoutSeconds: 5
  requestTimeoutSeconds: 10
  idleTimeoutSeconds: 1
  retries: 1

All timeout parameters are optional.

METRICS#

This method pulls statistics from the OpenMetrics endpoint and retrieves the number of running and queued queries for use with QueryCountBasedRouter.

You must enable either METRICS or JDBC if you use QueryCountBasedRouter.

By default, METRICS uses trino_execution_name_QueryManager_RunningQueries and trino_execution_name_QueryManager_QueuedQueries to track the number of running and queued queries. See the following example:

monitor:
    runningQueriesMetricName: io_starburst_galaxy_name_GalaxyMetrics_RunningQueries
    queuedQueriesMetricName: io_starburst_galaxy_name_GalaxyMetrics_QueuedQueries

By default the monitor pulls metrics using the /metrics endpoint. To configure an alternative endpoint, use the metricsEndpoint property.

monitor:
    metricsEndpoint: /v1/metrics

This monitor supports custom health definitions by comparing metrics using two maps: metricMinimumValues and metricMaximumValues. The keys metric names, and the values are minimum or maximum values (inclusive) considered healthy. By default, the only metric populated is trino_metadata_name_DiscoveryNodeManager_ActiveNodeCount:

monitor:
    metricMinimumValues:
        trino_metadata_name_DiscoveryNodeManager_ActiveNodeCount: 1

For the cluster to be considered healthy, one worker node must be active. To increase the minimum worker count to 10 and disqualify clusters experiencing frequent major garbage collections, set the following configuration properties:

monitor:
    metricMinimumValues:
        trino_metadata_name_DiscoveryNodeManager_ActiveNodeCount: 10
    metricMaximumValues:
        io_airlift_stats_name_GcMonitor_MajorGc_FiveMinutes_count: 2

JDBC#

This method uses a JDBC connection to query system.runtime tables for cluster information. It is required for the query count-based routing strategy.

This method is recommended over UI_API since it does not restrict the Starburst Enterprise web UI authentication method of backend clusters.

Configure a username and password by adding backendState to your configuration:

backendState:
  username: "user"
  password: "password"

The credentials must be valid across all backends.

Starburst Gateway uses explicitPrepare=false by default. This property uses a single query for prepared statements instead of a PREPARE/EXECUTE pair. If you are using the JDBC health check option with older versions of Trino, set the following configuration property:

monitor:
   explicitPrepare: true

To set the query timeout, set the following configuration property:

monitor:
    queryTimeout: 10

Other timeout parameters do not apply to the JDBC connection.

JMX#

The method collects cluster information, required for QueryCountBasedRouterProvider using the v1/jmx/mbean endpoint on clusters.

To enable JMX monitoring, complete the following steps:

  1. Activate JMX monitoring on all Trino clusters:

jmx.rmiregistry.port=<port>
jmx.rmiserver.port=<port>

  1. Allow JMX endpoint access by adding rules to your file-based access control configuration:

{
  "catalogs": [
    {
      "user": "user",
      "catalog": "system",
      "allow": "read-only"
    }
  ],
  "system_information": [
    {
      "user": "user",
      "allow": ["read"]
    }
  ]
}

  1. Configure a username and password in the the backendState section:

backendState:
  username: "user"
  password: "password"

The credentials must be consistent across all backend clusters and have read rights on system_information. The JMX monitor uses these credentials to authenticate against the JMX endpoint of each cluster and collect metrics like running queries, queued queries, and worker nodes information.

UI_API#

This method pulls cluster information from the ui/api/stats REST endpoint. Starburst supports this method for legacy reasons, and it may be deprecated in the future. It only works with backend clusters using web-ui.authentication.type=FORM. Set a username and password using backendState as with the JDBC option.

NOOP#

This option disables health checks.