Google Kubernetes Engine deployments

The Google Kubernetes Engine (GKE), is certified to work with Starburst Enterprise platform (SEP).

Introduction

This user guide provides information on using Helm charts to deploy SEP in Google Cloud - Kubernetes Applications. For more information about deploying on Kubernetes, see our Kubernetes guide with customization guide, examples, and tips and the Kubernetes reference documentation.

We recommend that you use Google Cloud’s Marketplace UI with Click to Deploy on GKE to deploy. The command line Deploy via command line installation is not supported.

Getting started

Setting up and deploying SEP to a Google Kubernetes Engine cluster using Google Cloud is pretty straightforward.

You can get started by simply following the UI instructions in the marketplace.

Cluster requirements

SEP can be deployed to an existing GKE cluster, or to a newly created one. If using an existing GKE cluster, make sure it is configured with the appropriate number and type of instances needed to deploy an SEP cluster. Review the SEP requirements to ensure that you have sufficient resources. With insufficient resources available in the cluster, the deployed pods may fail to be scheduled.

You can also choose to deploy a new cluster with minimal resources and default configuration from the marketplace offering configuration page. After the default SEP chart deploys successfully in this cluster, you must review the cluster configuration before updating it and making any adjustments to ensure that the cluster is sufficiently large.

Networking

When using GKE with a Google Cloud External Load Balancer, expose.ingress.path: you must have a wildcarded path ending with /\* to ensure that routing is not limited to a single URL. The \* character is the only supported wildcard character. It must follow a forward slash / and must be the last character in the pattern. See the Google URL Maps documentation for more information about path limitations and pattern matching in GKE.

Deploying

The default values of the SEP configuration in the UI work for most of the cases.

Warning

If you are enabling global access control using Ranger, make sure you note down the values of the admin and service user passwords you provide in the UI for later use.

Once done, you can click on Deploy to install SEP in the cluster. You can track the progress of the deployment in the applications page.

Once the application is successfully deployed, SEP is deployed using the Helm charts. If you enable either or both of Ranger and Hive Metastore Service these are also deployed. The deployment uses the values you provide in the UI and some default values when deployed the first time. If you want to upgrade or change the deployment configuration, you can either use the Application info from the application page or the following SEP upgrade instructions.

Upgrading SEP

You can upgrade SEP using the cloud shell.

You need to download the Helm charts and extract them to modify the deployment, as shown in the following:

export TAG=2.0.1
export APP_INSTANCE_NAME=<name of the application instance>
wget https://console.cloud.google.com/storage/browser/starburst-enterprise-presto/helmCharts/presto-gcp/starburst-enterprise-presto-charts-$TAG.tgz;
tar -zxvf starburst-enterprise-presto-charts-$TAG.tgz;

Now update the values.yaml in the umbrella chart folder (starburst-enterprise-presto-charts-$TAG) before you apply the manifest file. Alternatively, you can use --set parameter in helm template command to update the chart configuration. The values set in --set takes precedence over the ones set in values.yaml.

Once you have the manifest created, apply (kubectl apply) the file to update the deployment.

More details about the SEP configuration are found in the reference documentation.

Make sure you have usage metrics enabled in your SEP configuration. Our Google Cloud offering relies on the metrics and if disabled, the deployment is terminated after 36 hours.

The minimum set of parameters required for the upgrade are shown below:

helm template starburst-enterprise-presto-charts-$TAG/ \
  --name-template="$APP_INSTANCE_NAME" \
  --set deployerHelm.image="gcr.io/cloud-marketplace/starburst-public/starburst-presto/deployer:$TAG" \
  --set reportingSecret="$APP_INSTANCE_NAME-reporting-secret" \
  --set image.repository="gcr.io/cloud-marketplace/starburst-public/starburst-presto" \
  --set image.tag="$TAG" \
  --set initImage.repository="gcr.io/cloud-marketplace/starburst-public/starburst-presto/presto-init" \
  --set initImage.tag="$TAG" \
  --set metricsReporter.image="gcr.io/cloud-marketplace/starburst-public/starburst-presto/metrics_reporter:$TAG" \
  --set imageUbbagent="gcr.io/cloud-marketplace-tools/metering/ubbagent:latest" \
  --set [PUT OTHER CHART PARAMETERS YOU WANT TO UPDATE] > presto_manifest.yaml

The following example shows how to update the number of SEP workers:

helm template starburst-enterprise-presto-charts-$TAG/ \
  --name-template="$APP_INSTANCE_NAME" \
  --set deployerHelm.image="gcr.io/cloud-marketplace/starburst-public/starburst-presto/deployer:$TAG" \
  --set reportingSecret="$APP_INSTANCE_NAME-reporting-secret" \
  --set image.repository="gcr.io/cloud-marketplace/starburst-public/starburst-presto" \
  --set image.tag="$TAG" \
  --set initImage.repository="gcr.io/cloud-marketplace/starburst-public/starburst-presto/presto-init" \
  --set initImage.tag="$TAG" \
  --set metricsReporter.image="gcr.io/cloud-marketplace/starburst-public/starburst-presto/metrics_reporter:$TAG" \
  --set imageUbbagent="gcr.io/cloud-marketplace-tools/metering/ubbagent:latest" \
  --set worker.count > presto_manifest.yaml

Configure kubectl to point at your cluster and then apply the configuration:

kubectl apply -f presto_manifest.yaml

Enabling global access control with Apache Ranger

You can deploy or upgrade Ranger for global access control by creating a manifest file with Ranger parameters set.

You can either update the values.yaml in the umbrella chart folder starburst-enterprise-presto-charts-$TAG for the configuration (starburst-ranger) prior to generating the manifest file or or use --set parameter while creating the manifest file. The --set parameters must be prefixed by starburst-ranger.

The complete reference of the chart configuration can be found in the Ranger chart documentation.

Example with Ranger enabled:

helm template starburst-enterprise-presto-charts-$TAG/ \
  --name-template="$APP_INSTANCE_NAME" \
  --set deployerHelm.image="gcr.io/cloud-marketplace/starburst-public/starburst-presto/deployer:$TAG" \
  --set reportingSecret="$APP_INSTANCE_NAME-reporting-secret" \
  --set image.repository="gcr.io/cloud-marketplace/starburst-public/starburst-presto" \
  --set image.tag="$TAG" \
  --set initImage.repository="gcr.io/cloud-marketplace/starburst-public/starburst-presto/presto-init" \
  --set initImage.tag="$TAG" \
  --set metricsReporter.image="gcr.io/cloud-marketplace/starburst-public/starburst-presto/metrics_reporter:$TAG" \
  --set imageUbbagent="gcr.io/cloud-marketplace-tools/metering/ubbagent:latest" \
  --set starburst-ranger.enabled=true \
  --set [PUT OTHER CHART PARAMETERS YOU WANT TO UPDATE] > presto_manifest.yaml

Enabling the Hive Metastore Service

Similar to Ranger, to deploy or upgrade the Hive Metastore Service (HMS) provided by Starburst, generate a manifest file with Hive chart parameters set. You can either use values.yaml (for starburst-hive) or use the --set parameters. For Hive charts, the --set parameters must be prefixed by starburst-hive.

The complete reference of the chart configuration for the metastore can be found in the HMS documentation.

Example with HMS enabled:

helm template starburst-enterprise-presto-charts-$TAG/ \
  --name-template="$APP_INSTANCE_NAME" \
  --set deployerHelm.image="gcr.io/cloud-marketplace/starburst-public/starburst-presto/deployer:$TAG" \
  --set reportingSecret="$APP_INSTANCE_NAME-reporting-secret" \
  --set image.repository="gcr.io/cloud-marketplace/starburst-public/starburst-presto" \
  --set image.tag="$TAG" \
  --set initImage.repository="gcr.io/cloud-marketplace/starburst-public/starburst-presto/presto-init" \
  --set initImage.tag="$TAG" \
  --set metricsReporter.image="gcr.io/cloud-marketplace/starburst-public/starburst-presto/metrics_reporter:$TAG" \
  --set imageUbbagent="gcr.io/cloud-marketplace-tools/metering/ubbagent:latest" \
  --set starburst-hive.image.repository="gcr.io/cloud-marketplace/starburst-public/starburst-presto/hive" \
  --set starburst-hive.image.tag="$TAG" \
  --set starburst-hive.enabled=true > presto_manifest.yaml

Configure kubectl to point at your cluster and then apply the configuration:

kubectl apply -f presto_manifest.yaml

Deleting a deployment

To delete a GKE deployment, run the following command:

kubectl delete -f presto_manifest.yaml