Installing Starburst Enterprise in a Kubernetes cluster#
Now that you satisfied all the requirements you are ready to proceed with installing Starburst Enterprise platform (SEP).
Installing charts with Helm on your cluster follows the same steps:
Establish access to the Helm chart repository
Configure credentials to access the Docker registry in a registry access YAML file for use in all of your Starburst k8s deployments.
Create a YAML file specific to each chart and cluster, for example
sep-prod-setup.yamlfor your production cluster, as in this example.
Ensure your Helm/kubectl configuration points at the correct cluster with
Run Helm to install the chart
Access the cluster and check for success
Each Helm chart includes a values.yaml file that sets a reasonable set of default values. This default setup does not include catalog definitions and others that are necessary for your cluster.
You have to change your values file to add or update any configuration and run a Helm upgrade to apply the changes to the cluster.
Iterate on the configuration in the values YAML file with minimal setup until
you have a working system. Depending on your cluster you have to adjust memory
requirements for the worker and coordinator and other settings. Inspect the
kubectl or Octant to determine details.
After you achieved a running cluster, ensure to store the values YAML file as a reference and then add more details as required for the specific cluster need.
The SEP installation is managed with the
starburst-enterprise Helm chart.
Installation and upgrades are done with the
helm upgrade command with the
following options at a minimum:
a minimal YAML configuration file with the memory and CPU resource configurations for
worker:that reflect your cluster’s resources, for example
The following example command assumes the registry access file, the production cluster configuration file and the catalog configuration file are located in the current directory:
$ helm upgrade my-sep starburstdata/starburst-enterprise \ --install \ --version 360.15.0 \ --values ./registry-access.yaml --values ./sep-prod-setup.yaml --values ./sep-prod-catalogs.yaml
The version value is available from the Helm repository.
The default values result in one coordinator and two worker nodes in the cluster, with very specific memory and CPU resources that likely will not match your cluster’s specifications. We strongly suggest that you initially install SEP with the minimal changes needed to reflect your cluster’s memory and CPU resources, then make small, focused customizations to suit your organization’s needs.
The following sections describe the initial installation and how to begin customizing SEP.
Create this one file before you begin#
No matter what other configurations you need for your deployments, create a
registry-access.yaml file for reuse across all clusters and charts. This
helps to ensure a smooth install and deployment experience.
registry-access.yaml file, add the following to configure access for
Starburst’s Harbor registry:
registryCredentials: enabled: true registry: harbor.starburstdata.net/starburstdata username: <yourusername> password: <yourpassword>
The contents of this file override the default, empty values and ensure that the Helm charts can download the required Docker containers.
If you have multiple clusters, this same file is used for all of them. You can also use the same file for the optional Ranger and Hive Metastore Service charts. Other configurations should be managed in separate files.
As an alternative to using a username and password directly, you can use a Kubernetes secret:
Creating a secret containing the access token for your registry
Configuring your pod to use the secret
Initial installation checklist#
The following checklist describes the initial installation process:
Gather repository credentials for the Helm chart repository and the Docker registry provided by Starburst Support.
registry-access.yamlfile to override the default, empty values.
Create your correctly-sized Kubernetes cluster .
Ensure your Helm/
kubectlconfiguration points at the correct cluster with
Add your license file. We strongly suggest using a shared secret to add the license file. The following command assumes you are running it from the directory where your
$ kubectl create secret generic starburstdata --from-file starburstdata.license
Create a minimal YAML configuration file with the memory and CPU resource configurations for
worker:that reflect your cluster’s available resources, for example
Do not skip this step. The default values for memory and CPU resources likely vary significantly from your cluster’s available resources. If you attempt to run SEP with the defaults, SEP may not start.
Run Helm to install the default chart, as well as any override YAML files using the
--valuesargument, as in the following example:
$ helm upgrade sep-prod-cluster starburstdata/starburst-enterprise \ --install \ --version 360.15.0 \ --values ./registry-access.yaml \ --values ./sep-prod-setup.yaml
Determine the IP address or the DNS hostname of the coordinator by running the
kubectl get podscommand.
Use the IP address or hostname to verify the coordinator is running by accessing the Web UI. You can use the same information to connect with the CLI or the JDBC driver.
Continue configuring customizations for your cluster. The Configuring Starburst Enterprise in Kubernetes page provides details for configuration of catalogs to access data sources, worker and coordinator settings and much more.
Learn about operating your cluster with tips about testing, connecting, updating and more.
Updating to a new release#
If you have created focused, well-managed override files following our best practices guide, the upgrade process is a straightforward Helm-based process. As with any enterprise-scale application, we do recommend that you test upgrades from one release to another using a test cluster. This allows you to catch any configuration changes and update Helm charts before deploying into production:
Review the Helm charts release notes for any relevant changes that affect your override files. For instance, there may be new configuration options to add, or deprecated properties to remove.
Review the SEP release notes for new capabilities and breaking changes.
Run Helm with the updated Helm chart version and with the updated YAML configuration files, as in the following example:
$ helm upgrade my-sep-staging-cluster starburstdata/starburst-enterprise \ --install \ --version 360.15.0 \ --values ./registry-access.yaml \ --values ./sep-stage-setup.yaml
Hive Metastore Service installation#
The Hive Metastore Service installation is managed with the
Helm chart. Installation follows the same approach as the SEP chart with a
minimal values YAML file with the registry credentials and custom values file for your HMS configuration,
hms-prod.yaml, in the current directory:
$ helm upgrade my-sep starburstdata/starburst-hive \ --install \ --version 360.15.0 \ --values ./registry-access.yaml \ --values ./hms-prod.yaml
The Configuring the Hive Metastore Service in Kubernetes page provides details for the configuration of the Hive Metastore Service.
The Ranger installation is managed with the
starburst-ranger Helm chart.
Installation follows the same approach as the SEP chart with a minimal
values YAML file with the registry credentials and
custom values file for your Ranger configuration, for example
ranger-prod.yaml, in the current directory:
$ helm upgrade my-sep starburstdata/starburst-ranger \ --install \ --version 360.15.0 \ --values ./registry-access.yaml \ --values ./ranger-prod.yaml
The Configuring Starburst Enterprise with Ranger in Kubernetes provides details for the configuration of Apache Ranger.
Starburst Cache Service installation#
The Starburst Cache Service installation is managed with the
starburst-cache-service Helm chart. The installation follows the same
approach as the SEP chart with a minimal values YAML file with the
registry credentials and custom values file for your
cache service configuration. The following example uses the
cache-service-prod.yaml in the current directory:
$ helm upgrade my-caching-service starburstdata/starburst-cache-service \ --install \ --version 360.15.0 \ --values ./registry-access.yaml \ --values ./cache-service-prod.yaml
Configure the cache service in Kubernetes provides details for the configuration.
Recommended customization files#
We strongly suggest that you follow best practices for customization by creating a series of focused configuration files. The file set described below accomplishes this. If you have more than one cluster, such as a test cluster and a production cluster, name the files accordingly.
Do not copy over entire sections of the default
values.yaml file. Only
include nodes that you are changing or adding. Review how to create
these files before you begin.
We have provided examples of these files with content.
Docker registry access credentials file, typically to access the Docker
registry on the Starburst Harbor instance. Include the
Catalog configuration for all catalogs configured for SEP on the
Main configuration file for the
Create and manage additional configuration files, if you are operating multiple
clusters, while reusing the credentials file. For example, if you run a
stage cluster use the following additional files:
If you are optionally implementing one or both of the Hive Metastore Service and Apache Ranger, you to create an configuration file for each of these per cluster as well:
Private registries and repositories instead of Starburst Harbor#
Typically, you must use your username and password for accessing the Helm chart repositories and the Docker registry on the Starburst Harbor instance. You can instead use private Docker and Helm chart repositories.