Configuring Starburst Enterprise in Kubernetes#

The starburst-enterprise Helm chart configures the Starburst Enterprise platform (SEP) coordinator and worker nodes and the internal cluster communication with the values.yaml file detailed in the following sections.

We strongly suggest that you follow best practices when customizing your cluster, creating small, targeted files to override any defaults and add any configuration properties. There is an example file set in our install guide that illustrates the recommended way to manage your customizations.

Note

Many section in this page have links to examples of the relevant YAML in our YAML examples page. While you are configuring your cluster, it is is helpful to refer to SEP’s directory structure to ensure you are configuring any file locations correctly.

Throughout this page there are links to more descriptive documentation for a given configuration area. In many cases, these refer to configuration files by name. Wherever possible and applicable, it is noted in this that these files need to be configured in the etcFiles or additionalFiles: nodes. If this is not noted, the configuration applies directly to the YAML node discussed in the section.

Applying configuration changes#

To update SEP with any configuration changes, run the helm update command with the updated YAML files and the --install switch, as in this example:

$ helm upgrade my-sep-prod-cluster starburstdata/starburst-enterprise \
    --install \
    --version 359.0.0 \
    --values ./registry-access.yaml
    --values ./sep-prod-setup.yaml

You can use the same command as if you were updating to a new release. Helm will compare all --values files and the version, and safely ignore any that are unchanged.

Top level nodes#

The top level nodes are as follows, in order of appearance in the default values.yaml file embedded in the Helm chart. Click on a section header to access the relevant documentation:

Top level values.yaml nodes#

Node name

Description

Docker images section

image:

Contains the details for the SEP Docker image.

initImage:

Contains the details for the SEP bootstrap Docker image.

Docker registry access section

registryCredentials:

Defines authentication details for Docker registry access. Cannot be used if using imagePullSecrets:.

imagePullSecrets:

Alternative authentication for selected Docker registry using secrets. Cannot be used if using registryCredentials:.

Internal communications section

sharedSecret:

Sets the shared secret value for internal communications.

environment:

The environment name for the SEP cluster.

internal:

Specifies port numbers used for internal cluster communications.

Ports and networking

expose: node

Defines the mechanisms and their options that expose the SEP cluster to an outside network.

Coordinator and workers

The coordinator and worker nodes have many properties in common. They are detailed in the Coordinator section, and referenced in the Worker section, rather than repeating them. Likewise, examples are provided for the coordinator: top level node, and can be applied to the worker top level node by simply replacing coordinator with worker.

coordinator: node

Extensive section that configures the SEP coordinator pod.

worker: node

Extensive section that configures the SEP worker pods.

Startup options section

initFile: ""

Specifies a shell script to run after coordinator and work pod launch, and before SEP startup on the pods.

extraArguments: []

List of extra arguments to be passed to the initFile script.

extraSecret:

Lists secret to be used for initFile:.

Advanced security options

externalSecrets: node

Defines mounted external secrets from a Kubernetes secrets manager.

userDatabase: node

Configures the user database to use for file-based user authentication. This is disabled by default for versions 356-e and later.

securityContext: {} node

Sets the Kubernetes security context that defines privilege and access control settings for all SEP pods.

Pod lifecycle management section

readinessProbe:

Specifies the Kubernetes readiness probe that monitor containers for readiness to accept traffic.

livenessProbe:

Specifies the Kubernetes liveness probe that monitor containers for the need to restart.

SEP cache, disk and memory management

query: node

Specifies query processing and management configuration properties.

spilling: node

Configures spilling of query processing data to disk.

cache: node for hive configuration

Configures Hive storage caching.

Catalogs to configure connected data sources

catalogs: node

Configures the catalog properties files.

Miscellaneous

additionalVolumes: node

Defines mounted volumes for various uses in the cluster.

usageMetrics: node

Configure usage metrics gathering.

prometheus: node

Configures cluster metrics gathering with Prometheus.

commonLabels: {}

Defines common labels to identify objects in a KRM to use with the kustomize utility and other tools.

Docker images#

The default configuration automatically contains the details for the relevant Docker image on the Starburst Harbor instance. Typically you should not configure any overrides. An exception is if you host the Docker image on a different Docker registry.

Top level docker image nodes#

Node name

Description

image:

The image section controls the Docker image to use including the version. Typically, the Helm chart version reflects the SEP version. Specifically, the SEP version 356-e.x is reflected in the chart version of 356.x.0, or patches for the chart are released as 356.x.1, 356.x.2, and similar.

initImage:

The initImage section defines the container image and version to use for bootstrapping configuration files for SEP from the Helm chart and values files. This functionality is internal to the chart, and you should not change or override this configuration.

Defaults and examples:

Docker registry access#

These nodes enable access to the Docker registry. They should be added to the registry-access.yaml file to access the registry as shown in the best practices guide.

Top level docker image and registry credentials nodes#

Node name

Description

registryCredentials:

The registryCredentials section defines authentication details for Docker registry access. Typically, you need to use your username and password for the Starburst Harbor instance. NOTE: Cannot be be used if using imagePullSecrets:.

imagePullSecrets:

Alternative authentication for selected Docker registry using secrets. NOTE: Cannot be used if using registryCredentials:.

Warning

You can only define authentication with registryCredentials or imagePullSecrets. Defining both is invalid, and causes an error.

Defaults and examples:

Internal communication#

SEP provides two top level nodes for internal cluster communication between the coordinator and the workers. Because internal communication configurations and credentials are unique, these are not configured by default.

Top level internal communications nodes#

Node name

Description

environment

The environment name for the SEP cluster, used to set the node.environment node configuration property. Examples are values such as production, staging, batch_cluster, site_analytics, or any other short, meaningful string.

sharedSecret

Sets the shared secret value for secure communication between coordinator and workers in the cluster to a long, random string that you provide. If not set, the node.environment value from the etcFiles.node.properties configuration on the coordinator is used.

Defaults and examples:

Exposing the cluster#

You must expose the cluster to allow users to connect to the SEP coordinator with tools such as the CLI, applications using JDBC/ODBC drivers and any other client application. This service-type configuration is defined by the expose top level node. You can choose from four different mechanisms by setting the type value to the common configurations in k8s.

Depending on your choice, you only have to configure the identically-named sections.

Types for expose: top level node#

Type

Description

clusterIp

Default value. Only exposes the coordinator internally within the k8s cluster using an IP address internal to the cluster. Use this in the early stages of configuration.

nodePort

Configures the internal port number of the coordinator for requests from outside the cluster on the nodePort port number. External service requests are made to <nodeIP>:<nodePort>. The clusterIP service is automatically created to supply all internal IP addresses.

loadBalancer

Used with platforms that provide a load balancer. This option automatically creates nodePort and clusterIP services , to which the external load balancer routes requests.

ingress

This option provides a production-level, securable configuration. It allows a load balancer to route to multiple apps in the cluster, and may provide load balancing, SSL termination, and name-based virtual hosting. For example, the SEP coordinator and Ranger server can be in the same cluster, and can be exposed via ingress: configuration.

Defaults and examples:

Coordinator#

The top level coordinator: node contains property nodes that configure the pod of the cluster that runs the SEP coordinator. The default values are suitable to get started with reasonable defaults on a production-sized k8s cluster.

Note

These property nodes also appear under the top level worker: node unless otherwise noted.

coordinator: nodes#

Node name

Description

etcFiles

Specifies the content of the jvm.config file and the *.properties files associated with the coordinator. These files are described in greater detail in a separate table below. NOTE: Not all files described in the detail table are applicable to workers. They are noted as such.

resources

The CPU and memory resources to use for the coordinator pod. Request and limit values should be identical. These settings can be adjusted to match your workload and available node sizes in the cluster.

nodeMemoryHeadroom

The size of the container memory headroom. The value needs to be less than resource allocation limit for memory defined in resources.

heapSizePercentage

Percentage of container memory reduced with headroom assigned to Java heap. Must be less than 100.

heapHeadroomPercentage

Headroom of Java heap memory not tracked by SEP during query execution. Must be less than 100.

additionalProperties

Any properties described in the reference documentation that are not specific to any other YAML node are set here. Example usages are to set query time-out and memory values and to enable Starburst Insights. Configuration properties can be found throughout the reference documentation, including the Properties reference page.

envFrom: []

Allows for the propagation of environment variables from different sources complying with K8S schema can be used to deliver values to SEP configuration properties file by creating a Kubernetes secret holding variable values.

nodeSelector: {}, affinity: {} and tolerations: []

Configuration to determine the node and pod to use.

deploymentAnnotations: {}

Configuration to annotate the coordinator deployment.

podAnnotations: {}

Configuration to annotate the coordinator pod.

priorityClassName

Priority class for coordinator pod for setting k8s pod priority and preemption.

Defaults and examples:

etcFiles on the coordinator#

Note

These property nodes also appear under the top level worker: node unless otherwise noted.

etcFiles on the coordinator#

Node name

Description

coordinator.etcFiles.jvm.config

Defines the content of the JVM config for the coordinator.

coordinator.etcFiles.properties

Defines configuration files located in the etc folder. Each nested section defines the filename. Defaults are provided for the main configuration files etc/jvm.config, etc/config.properties, etc/node.properties and etc/log.properties. Additional properties files can be added by adding a nested section and the desired content of the configuration file, for instance for the event logger.

coordinator.etcFiles.properties.config.properties

Defines the content of the default configuration file for the coordinator. You can also use additionalProperties for adding other configuration values.

coordinator.properties.node.properties

Defines the contents of the node.properties file for the coordinator, including paths to installation and log directories. The unique identifier and environment are pulled in from the environment, as documented in the internal communication section.

coordinator.etcFiles.properties.log.properties

Defines the contents of the log configuration files for the coordinator.

coordinator.etcFiles.properties.password-authentication.properties

Configures password authentication for SEP on the coordinator. NOTE: Does not apply to worker.etcFiles.properties.

coordinator.etcFiles.properties.access-control.properties

Enables and configures access control for SEP on the coordinator. NOTE: Does not apply to worker.etcFiles.properties.

coordinator.etcFiles.other

Other files that need to be placed in the etc directory (example).

Node assignment#

All charts allow you to configure criteria to define which node and pod in the cluster is suitable to use for running the relevant container. In SEP, this may be useful if you have a cluster spanning availability zones, or if you are using Ranger or an HMS in your cluster with smaller node sizes. The following configurations are available, and by default are not defined:

nodeSelector: {}
affinity: {}
tolerations: []

Example configurations are available in the k8s documentation. Specific usage and values are highly dependent on your k8s cluster configuration.

Further resources:

Coordinator defaults and examples#

Workers#

The worker section configures the pods of the cluster that run the SEP workers.

The top level worker: node contains property nodes that configure the pod of the cluster that runs the SEP workers. The default values are suitable to get started with reasonable defaults on a production-sized k8s cluster.

Many of the properties for this node also appear in the coordinator: top level node. They are fully documented in the Coordinator section:

  • worker.etcFiles.*

  • worker.resources

  • worker.nodeMemoryHeadroom

  • worker.heapSizePercentage

  • worker.heapHeadroomPercentage

  • worker.additionalProperties

  • worker.envFrom

  • worker.nodeSelector

  • worker.affinity

  • worker.tolerations

  • worker.deploymentAnnotations

  • worker.podAnnotations

  • worker.priorityClassName

The following properties apply only to worker:

worker:-only nodes#

Node name

Description

etcFiles

Specifies

worker.replicas

The number of worker pods for a static cluster.

worker.autoscaling

Configuration for the minimum and maximum number of workers. Ensure the additional requirements for scaling are fulfilled on your k8s cluster. Set enabled to true to activate autoscaling. The targetCPUUtilizationPercentage sets the threshold value that triggers scaling up by adding more workers until maxReplicas is reached.

Scaling down proceeds until minReplicas is reached and is controlled by by deploymentTerminationGracePeriodSeconds and starburstWorkerShutdownGracePeriodSeconds.

WARNING: The autoscaling feature does not yet work with OpenShift clusters (as of the latest release 4.6), due to a known limitation with OpenShift clusters. HPA does not work with pods having init containers in OpenShift.

Read more information in our scaling section.

worker.deploymentTerminationGracePeriodSeconds

Specifies the termination grace period for workers. Workers are not terminated until queries running on the pod are finished and the grace period passes.

worker.starburstWorkerShutdownGracePeriodSeconds

Sets shutdown.grace-period to configure the grace period for worker process shutdown.

The following configuration properties for workers are identical to the coordinator properties, documented in preceding section.

Startup shell script for coordinator and workers nodes#

initFile: is a top level node that allows you to create a startup shell script (example) to customize how SEP is started on the coordinator and workers, and pass additional arguments to it. These are undefined by default.

Startup script nodes#

Node name

Description

initFile

A shell script to run before SEP is launched. The content of the file has to be an inline string in theYAMLfile. The script is started as /bin/bash <<init_file>>. When called, it is passed the single parameter value coordinator or worker depending on the type of pod. The script needs to invoke the launcher script /usr/lib/starburst/bin/run-starburst for a successful start of SEP. NOTE: In the initFile, for chart versions released before middle of Oct 2020 use the launcher script exec /usr/lib/starburst/bin/launcher run. For newer releases use exec /usr/lib/starburst/bin/run-starburst to enable graceful shutdown of workers .

extraArguments

List of extra arguments to be passed to the initFile script.

User security considerations#

SEP has extensive security options that allow you specify how to authenticate users. Because user security configurations and credentials are unique, these are not configured by default.

Note

We strongly suggest that you watch our Securing Starburst Enterprise training video before customizing your SEP security configuration.

Security context#

You can configure a security context to define privilege and access control settings for the SEP pods.

securityContext:
  <<security settings for the pod>>

External secret reference#

There are several locations where properties require pointing to files delivered outside of SEP, such as CA certificates. In such cases, you can use a special notation that allows you to point to a k8s secret.

For example, you can configure password authentication using LDAP (example). This requires k8s to create the etc/password-authenticator.properties configuration file, which in turn points to the ca.crt certificate file.

Defining external secrets#

You can automatically mount external secrets, for example from the AWS Secrets Manager, using the secretRef or secretEnv notation.

externalSecrets nodes#

Node name

Description

externalSecrets.type

Type of the external secret provider. Currently, only goDaddy is supported. NOTE: The external secret provider needs to be configured with proper access to the specified backendType in order to successfully pull in the external secrets.

externalSecrets.secretPrefix

Prefix of all secrets that need to be mapped to external secret.

externalSecrets.goDaddy.backendType

The GoDaddy backend type, such as secretsManager or systemManager.

The Helm chart scans for all secretRef or secretEnv references in the referenced YAML files which start with the configured secretPrefix string. For each secret found, it generates an``ExternalSecret`` K8s manifest (example).

Note

The selected external secrets provider needs to be deployed and configured separately. The secret names in the external storage must match names of K8s secrets you reference. When using secretEnv, the external storage secret must contain only a single value. For each external secret a single K8s secret is created, including one key with external secret value.

File-based authentication#

The unix command htpasswd can generate a user database, which can be used to configure file-based user authentication. It creates the file under /usr/lib/starburst/etc/auth/{{.Values.userDatabase.name}}. This allows you to statically deliver user credentials to the file etc/password-authenticator.properties (example).

Alternatively, you can use a secret to add an externally created user database file to SEP. Set the file.password-file property to the user database file, and ensure to disable the built-in user database (example).

Performance considerations#

SEP has performance management configuration options that help you to optimize aspects of SEP’s performance.

Note

We strongly suggest that you watch our cluster tuning training video before customizing your SEP security configuration.

Query memory usage control#

The top level query: node allows you to set the maxConcurrentQueries configuration property. All other query processing properties must be set using the additionalProperties: node on the coordinator.

You can view the default setting and an example of how to increase the maximum allowable number of concurrent queries.

Spilling#

The top level query: node (example) allows you to configure SEP’s spilling properties.

Spilling uses internal node storage, which is mounted within the container.

Warning

Spilling is disabled :ref:` by default <k8s-defaults-performance-spilling>`, and we strongly suggest to leave it disabled. Enabling spill should be used as a method of last resort to allow for rare, memory-intensive queries to succeed on a smaller cluster at the expense of query performance and overall cluster performance.

Hive connector storage caching#

The cache: top level node (example) allows you to configure Hive connector storage caching. It is disabled by default.

Hive connector caching property nodes#

Node name

Description

cache.enabled

Enable or disable caching for all catalogs using the Hive connector. If you want to only enable it for a specific catalog, you have to configure it with the catalog configuration and additionalVolumes.

cache.diskUsagePercentage

Set the value for the hive.cache.disk-usage-percentage property.

cache.ttl

Set the value for the hive.cache.ttl property.

cache.volume

Configure the volume in which to store the cached files.

Catalogs#

The catalogs: top level node allows you to configure a catalog for each of your data sources. The catalogs defined in this node are used to create catalog properties files, which contain key-value pairs for catalog properties. Information for specific properties supported in each catalog can be found with the documentation for the connectors. At the very minimum, a catalog definition must consist of the name of the name of the catalog, and the connector.name property.

For best practices, use the YAML multi-line syntax shown in the examples to configure the content in multiple lines indented below the catalog name.

Defaults and examples:

Additional volumes#

Additional k8s volumes can be necessary for persisting files, for Hive object storage caching, and for a number of other use cases. These can be defined in the additionalVolumes section (example). None are defined by default.

additionalVolume property nodes#

Node name

Description

path

Specifies the path to the mounted volume. If you specify path only, a directory named in path is created. When mounting ConfigMap or Secret, files are created in this directory for each key.

volume

A directory, with or without data, which is accessible to the containers in a pod.

hostpath

A volume with a hostPath defined mounts a file or directory from the host node’s filesystem into your pod.

hostpath.path

REQUIRED when using hostPath-mounted volumes.

subPath

When specified, a specific key named subPath from ConfigMap or Secret is mounted as a file with name provided by path. This is an alternative to using hostpath; you cannot define both for a volume.

Adding files#

Various use cases around security and event listeners need additional config files as properties or XML files. You can add any file to a pod using config maps.

Types of files you may need to add:

  • LDAP authentication file

  • Hive site xml file

Alternatively you can also use additionalVolumes to mount the files and copy the files to appropriate location using path and subPath parameter (example).

Usage metrics#

The top level usageMetrics node configures usage metric. Metrics gathering is enabled by default, which results in an additional sidecar container with the coordinator called usage-client. The container gathers and exposes the metrics over the lifetime of the coordinator.

Metrics are not persisted by default. Restarts of the coordinator pod result in a reset of the metrics unless a persistent volume is defined for them. To accomplish this, the path to the persisted log file is configured under the top level additionalProperties node with the usage-metrics.log.path property. The persistent volume must then be configured as additional volume.

There are a number of usage metrics configuration properties, all of which can be defined as key-value pairs under the additionalProperties top level node.

We strongly suggest reviewing usage metrics examples.

usageMetrics: property nodes#

Node name

Description

enabled

Specifies the path to the mounted volume. If you specify path only, a directory named in path is created. When mounting ConfigMap or Secret, files are created in this directory for each key.

usageClient.initialDelay

The initial delay to before tracking usage, allowing the cluster to start up before metrics gathering starts. Defaults to 1m if not defined.

usageClient.interval

Length of the interval between usage metric log entry creation. Ideally, this should be set to a small value, such as the default of 1 min.

You can view usage metrics data using kubectl:

$ kubectl logs -n <namespace> <pod_name> usage-client
{"startTime":1598353893065,"time":"2020-08-25T11:13:00.753Z","cumulativeCpuTime":"0.00s",...}
{"startTime":1598353893065,"time":"2020-08-25T11:14:00.707Z","cumulativeCpuTime":"9.61s",...}

Prometheus#

This top level node configures the cluster to create Prometheus metrics. It is not to be confused with the connector of the same name. It is enabled by default.

We strongly suggest reviewing our example Prometheus rules.

SEP built-in directories#

The location of specific directories on the SEP container is important, if you configure additional files or otherwise want to tweak the container.

SEP is configured to use the following directories in the container:

  • /usr/lib/starburst: Top level folder for the SEP binaries.

  • /usr/lib/starburst/plugin: All plugins for SEP including connectors

  • /usr/lib/starburst/etc: etc folder for SEP configuration files such as config.properties and others.

  • /usr/lib/starburst/bin: Location of the run-starburst script, which is invoked with the container start.

  • /data/starburst/var/run: Contains launcher.pid file used by the launcher script.

  • /data/starburst/var/log: Contains secondary, rotated log files. Main log is redirected to stdout as recommended on containers.