Google Kubernetes Engine deployments #

The Google Kubernetes Engine (GKE), is certified to work with Starburst Enterprise platform (SEP).

Introduction #

This user guide provides information on using Starburst Helm charts to deploy SEP, and eventually a cluster in Google Cloud - Kubernetes Applications. More information is available in our Kubernetes reference documentation, which includes a customization guide, examples and tips.

Before you get started installing SEP, we suggest that you read our customization guide to learn how to customize SEP clusters.

We also have a helpful installation checklist for an overview of the general Helm-based installation and upgrade process.

We recommend that you use Google Cloud’s marketplace UI with Click to Deploy on GKE to deploy. The command line Deploy via command line installation is not supported.

Getting started #

Setting up and deploying SEP to a Google Kubernetes Engine cluster using Google Cloud is pretty straightforward.

You can get started by simply following the UI instructions in the marketplace.

Cluster requirements #

SEP can be deployed to an existing GKE cluster, or to a newly created one. If using an existing GKE cluster, make sure it is configured with the appropriate number and type of instances needed to deploy an SEP cluster. Review the SEP requirements to ensure that you have sufficient resources. With insufficient resources available in the cluster, the deployed pods may fail to be scheduled.

You can also choose to deploy a new cluster with minimal resources and default configuration from the marketplace offering configuration page. After the default SEP chart deploys successfully in this cluster, you must review the cluster configuration before updating it and making any adjustments to ensure that the cluster is sufficiently large.

Deploying #

The default values of the SEP configuration in the UI work for most of the cases.

Once done, you can click on Deploy to install SEP in the cluster. You can track the progress of the deployment in the applications page.

Once the application is successfully deployed, SEP is deployed using the Helm charts. If you enable either or both of Ranger and Hive Metastore Service these are also deployed. The deployment uses the values you provide in the UI and some default values when deployed the first time. If you want to upgrade or change the deployment configuration, you can either use the Application info from the application page or the following SEP upgrade instructions.

Upgrading SEP #

You can upgrade SEP using the cloud shell.

You need to download the Helm charts and extract them to modify the deployment, as shown in the following:

export TAG=2.0.1
export APP_INSTANCE_NAME=<name of the application instance>
wget https://console.cloud.google.com/storage/browser/starburst-enterprise-presto/helmCharts/presto-gcp/starburst-enterprise-presto-charts-$TAG.tgz;
tar -zxvf starburst-enterprise-presto-charts-$TAG.tgz;

Now update the values.yaml in the umbrella chart folder (starburst-enterprise-presto-charts-$TAG) before you apply the manifest file. Alternatively, you can use --set parameter in helm template command to update the chart configuration. The values set in --set takes precedence over the ones set in values.yaml.

Once you have the manifest created, apply (kubectl apply) the file to update the deployment.

More details about the SEP configuration are found in the reference documentation.

Make sure you have usage metrics enabled in your SEP configuration. Our Google Cloud offering relies on the metrics and if disabled, the deployment is terminated after 36 hours.

The minimum set of parameters required for the upgrade are shown below:

helm template starburst-enterprise-presto-charts-$TAG/ \
  --name-template="$APP_INSTANCE_NAME" \
  --set deployerHelm.image="gcr.io/cloud-marketplace/starburst-public/starburst-presto/deployer:$TAG" \
  --set reportingSecret="$APP_INSTANCE_NAME-reporting-secret" \
  --set image.repository="gcr.io/cloud-marketplace/starburst-public/starburst-presto" \
  --set image.tag="$TAG" \
  --set initImage.repository="gcr.io/cloud-marketplace/starburst-public/starburst-presto/presto-init" \
  --set initImage.tag="$TAG" \
  --set metricsReporter.image="gcr.io/cloud-marketplace/starburst-public/starburst-presto/metrics_reporter:$TAG" \
  --set imageUbbagent="gcr.io/cloud-marketplace-tools/metering/ubbagent:latest" \
  --set [PUT OTHER CHART PARAMETERS YOU WANT TO UPDATE] > presto_manifest.yaml

The following example shows how to update the number of SEP workers:

helm template starburst-enterprise-presto-charts-$TAG/ \
  --name-template="$APP_INSTANCE_NAME" \
  --set deployerHelm.image="gcr.io/cloud-marketplace/starburst-public/starburst-presto/deployer:$TAG" \
  --set reportingSecret="$APP_INSTANCE_NAME-reporting-secret" \
  --set image.repository="gcr.io/cloud-marketplace/starburst-public/starburst-presto" \
  --set image.tag="$TAG" \
  --set initImage.repository="gcr.io/cloud-marketplace/starburst-public/starburst-presto/presto-init" \
  --set initImage.tag="$TAG" \
  --set metricsReporter.image="gcr.io/cloud-marketplace/starburst-public/starburst-presto/metrics_reporter:$TAG" \
  --set imageUbbagent="gcr.io/cloud-marketplace-tools/metering/ubbagent:latest" \
  --set worker.count > presto_manifest.yaml

Configure kubectl to point at your cluster and then apply the configuration:

kubectl apply -f presto_manifest.yaml

Enabling global access control with Apache Ranger #

You can deploy or upgrade Ranger for global access control by creating a manifest file with Ranger parameters set.

You can either update the values.yaml in the umbrella chart folder starburst-enterprise-presto-charts-$TAG for the configuration (starburst-ranger) prior to generating the manifest file or or use --set parameter while creating the manifest file. The --set parameters must be prefixed by starburst-ranger.

The complete reference of the chart configuration can be found in the Ranger chart documentation.

Example with Ranger enabled:

helm template starburst-enterprise-presto-charts-$TAG/ \
  --name-template="$APP_INSTANCE_NAME" \
  --set deployerHelm.image="gcr.io/cloud-marketplace/starburst-public/starburst-presto/deployer:$TAG" \
  --set reportingSecret="$APP_INSTANCE_NAME-reporting-secret" \
  --set image.repository="gcr.io/cloud-marketplace/starburst-public/starburst-presto" \
  --set image.tag="$TAG" \
  --set initImage.repository="gcr.io/cloud-marketplace/starburst-public/starburst-presto/presto-init" \
  --set initImage.tag="$TAG" \
  --set metricsReporter.image="gcr.io/cloud-marketplace/starburst-public/starburst-presto/metrics_reporter:$TAG" \
  --set imageUbbagent="gcr.io/cloud-marketplace-tools/metering/ubbagent:latest" \
  --set starburst-ranger.enabled=true \
  --set [PUT OTHER CHART PARAMETERS YOU WANT TO UPDATE] > presto_manifest.yaml

Enabling the Hive Metastore Service #

Similar to Ranger, to deploy or upgrade the Hive Metastore Service (HMS) provided by Starburst, generate a manifest file with Hive chart parameters set. You can either use values.yaml (for starburst-hive) or use the --set parameters. For Hive charts, the --set parameters must be prefixed by starburst-hive.

The complete reference of the chart configuration for the metastore can be found in the HMS documentation.

Example with HMS enabled:

helm template starburst-enterprise-presto-charts-$TAG/ \
  --name-template="$APP_INSTANCE_NAME" \
  --set deployerHelm.image="gcr.io/cloud-marketplace/starburst-public/starburst-presto/deployer:$TAG" \
  --set reportingSecret="$APP_INSTANCE_NAME-reporting-secret" \
  --set image.repository="gcr.io/cloud-marketplace/starburst-public/starburst-presto" \
  --set image.tag="$TAG" \
  --set initImage.repository="gcr.io/cloud-marketplace/starburst-public/starburst-presto/presto-init" \
  --set initImage.tag="$TAG" \
  --set metricsReporter.image="gcr.io/cloud-marketplace/starburst-public/starburst-presto/metrics_reporter:$TAG" \
  --set imageUbbagent="gcr.io/cloud-marketplace-tools/metering/ubbagent:latest" \
  --set starburst-hive.image.repository="gcr.io/cloud-marketplace/starburst-public/starburst-presto/hive" \
  --set starburst-hive.image.tag="$TAG" \
  --set starburst-hive.enabled=true > presto_manifest.yaml

Configure kubectl to point at your cluster and then apply the configuration:

kubectl apply -f presto_manifest.yaml

Deleting a deployment #

To delete a GKE deployment, run the following command:

kubectl delete -f presto_manifest.yaml