Google Kubernetes Engine#

Google Cloud Platform (GCP), and specifically the Google Kubernetes Engine, is certified to work with Starburst Enterprise platform (SEP).

GKE-specific tips can be found in this document.

Starburst Enterprise platform (SEP) is available on the Google Cloud Platform Marketplace. It offers a simple and convenient way to get started with SEP running on Kubernetes.

The marketplace offering uses the Kubernetes Helm charts and is listed on the marketplace as starburst-presto.

Further applicable information can be found in the documentation for SEP on the Google Kubernetes Engine.

Introduction#

This user guide provides information on using the charts to deploy SEP and a and eventually a cluster on the GCP Marketplace - Kubernetes Applications.

We recommend you to use the GCP Marketplace UI with Click to Deploy on GKE to deploy. The command line Deploy via command line installation is not supported.

Getting started#

Setting up and deploying SEP to a Google Kubernetes Engine (GKE) cluster using Google Cloud Marketplace is pretty straightforward.

You can get started by simply following the UI instructions in the marketplace.

Cluster requirements#

You can either use an existing GKE cluster or create a new one. If using an existing cluster make sure you have the GKE cluster configured with the appropriate number and type of instances you may need to deploy a SEP cluster. With insufficient resources available in the cluster, the deployed pods may fail to be scheduled. Ensure you are aware of the requirements for any additional prerequisites for your GKE cluster.

You can also choose to deploy a new cluster with minimal resources and default configuration from the Marketplace offering configuration page. After the default SEP chart deploys successfully in this cluster, you have to review the cluster configuration before updating it to adjust configuration to the actual desired needs. You need to ensure that the cluster is sufficiently large.

Deployment#

The default values of the configuration in the UI work for most of the cases.

Warning

If you are enabling Global Security using Ranger, as described in Global access control with Apache Ranger make sure you note down the values of the admin and service user passwords you provide in the UI for later use.

Once done, you can click on Deploy to install SEP in the cluster. You can track the progress of the deployment in the applications page.

Once the application is successfully deployed, SEP is deployed using the Helm charts. If you enable Ranger and/or Hive Metastore Service these are also deployed. The deployment uses the values you provide in the UI and some default values when deployed the first time. If you want to upgrade or change the deployment configuration you can either use the Application info from the application page or the SEP deployment documentation.

Upgrade SEP#

You can upgrade SEP using the cloud shell or local shell with helm and kubectl configured to your GKE cluster.

You need to download the Helm charts and extract them to modify the deployment.

export TAG=2.0.1
export APP_INSTANCE_NAME=<name of the application instance>
wget https://console.cloud.google.com/storage/browser/starburst-enterprise-presto/helmCharts/presto-gcp/starburst-enterprise-presto-charts-$TAG.tgz;
tar -zxvf starburst-enterprise-presto-charts-$TAG.tgz;

Now update the values.yaml in the umbrella chart folder (starburst-enterprise-presto-charts-$TAG) before you apply the manifest file. Alternatively, you can use --set parameter in helm template command to update the chart configuration. The values set in --set takes precedence over the ones set in values.yaml.

Once you have the manifest created, apply (kubectl apply) the file to update the deployment.

More details about the SEP configuration are reference documentation.

Make sure you have usageMetrics enabled in your SEP configuration. Our GCP offering relies on the metrics and if disabled, the deployment is terminated after 36 hours.

The minimum set of parameters required for the upgrade are shown below:

helm template starburst-enterprise-presto-charts-$TAG/ \
  --name-template="$APP_INSTANCE_NAME" \
  --set deployerHelm.image="gcr.io/cloud-marketplace/starburst-public/starburst-presto/deployer:$TAG" \
  --set reportingSecret="$APP_INSTANCE_NAME-reporting-secret" \
  --set image.repository="gcr.io/cloud-marketplace/starburst-public/starburst-presto" \
  --set image.tag="$TAG" \
  --set initImage.repository="gcr.io/cloud-marketplace/starburst-public/starburst-presto/presto-init" \
  --set initImage.tag="$TAG" \
  --set metricsReporter.image="gcr.io/cloud-marketplace/starburst-public/starburst-presto/metrics_reporter:$TAG" \
  --set imageUbbagent="gcr.io/cloud-marketplace-tools/metering/ubbagent:latest" \
  --set [PUT OTHER CHART PARAMETERS YOU WANT TO UPDATE] > presto_manifest.yaml

The following example shows how to update the number of Presto workers:

helm template starburst-enterprise-presto-charts-$TAG/ \
  --name-template="$APP_INSTANCE_NAME" \
  --set deployerHelm.image="gcr.io/cloud-marketplace/starburst-public/starburst-presto/deployer:$TAG" \
  --set reportingSecret="$APP_INSTANCE_NAME-reporting-secret" \
  --set image.repository="gcr.io/cloud-marketplace/starburst-public/starburst-presto" \
  --set image.tag="$TAG" \
  --set initImage.repository="gcr.io/cloud-marketplace/starburst-public/starburst-presto/presto-init" \
  --set initImage.tag="$TAG" \
  --set metricsReporter.image="gcr.io/cloud-marketplace/starburst-public/starburst-presto/metrics_reporter:$TAG" \
  --set imageUbbagent="gcr.io/cloud-marketplace-tools/metering/ubbagent:latest" \
  --set worker.count > presto_manifest.yaml

Configure kubectl to point at your cluster and then apply the configuration:

kubectl apply -f presto_manifest.yaml

Enable global access control with Apache Ranger#

You can deploy or upgrade Ranger Global access control with Apache Ranger for global security by creating manifest file with Ranger parameters set.

You can either update the values.yaml in the umbrella chart folder starburst-enterprise-presto-charts-$TAG for the configuration (starburst-ranger) prior to generating the manifest file or or use --set parameter while creating the manifest file. The --set parameters must be prefixed by starburst-ranger.

The complete reference of the chart configuration can be found in the Ranger chart documentation.

Example with Ranger enabled:

helm template starburst-enterprise-presto-charts-$TAG/ \
  --name-template="$APP_INSTANCE_NAME" \
  --set deployerHelm.image="gcr.io/cloud-marketplace/starburst-public/starburst-presto/deployer:$TAG" \
  --set reportingSecret="$APP_INSTANCE_NAME-reporting-secret" \
  --set image.repository="gcr.io/cloud-marketplace/starburst-public/starburst-presto" \
  --set image.tag="$TAG" \
  --set initImage.repository="gcr.io/cloud-marketplace/starburst-public/starburst-presto/presto-init" \
  --set initImage.tag="$TAG" \
  --set metricsReporter.image="gcr.io/cloud-marketplace/starburst-public/starburst-presto/metrics_reporter:$TAG" \
  --set imageUbbagent="gcr.io/cloud-marketplace-tools/metering/ubbagent:latest" \
  --set starburst-ranger.enabled=true \
  --set [PUT OTHER CHART PARAMETERS YOU WANT TO UPDATE] > presto_manifest.yaml

Enable the Hive Metastore Service#

Similar to Ranger, to deploy or upgrade the Hive Metastore Service (HMS) provided by Starburst, generate a manifest file with Hive chart parameters set. You can either use values.yaml (for starburst-hive) or use the --set parameters. For Hive charts, the --set parameters must be prefixed by starburst-hive.

The complete reference of the chart configuration for the metastore can be found in the HMS documentation.

Example with HMS enabled:

helm template starburst-enterprise-presto-charts-$TAG/ \
    --name-template="$APP_INSTANCE_NAME" \
    --set deployerHelm.image="gcr.io/cloud-marketplace/starburst-public/starburst-presto/deployer:$TAG" \
    --set reportingSecret="$APP_INSTANCE_NAME-reporting-secret" \
    --set image.repository="gcr.io/cloud-marketplace/starburst-public/starburst-presto" \
    --set image.tag="$TAG" \
    --set initImage.repository="gcr.io/cloud-marketplace/starburst-public/starburst-presto/presto-init" \
    --set initImage.tag="$TAG" \
    --set metricsReporter.image="gcr.io/cloud-marketplace/starburst-public/starburst-presto/metrics_reporter:$TAG" \
    --set imageUbbagent="gcr.io/cloud-marketplace-tools/metering/ubbagent:latest" \
    --set starburst-hive.image.repository="gcr.io/cloud-marketplace/starburst-public/starburst-presto/hive" \
    --set starburst-hive.image.tag="$TAG" \
    --set starburst-hive.enabled=true > presto_manifest.yaml

Configure kubectl to point at your cluster and then apply the configuration:

kubectl apply -f presto_manifest.yaml

Delete the deployment#

To delete the deployment:

kubectl delete -f presto_manifest.yaml