6.5. Presto Kubernetes resource

Presto Kubernetes resources constitute particular Presto clusters. Each such a resource represents a single Presto cluster. A Presto Kubernetes resource YAML allows specifying various properties related to the Presto server, Presto connectors and Kubernetes. Properties are defined in the spec section of Presto Kubernetes resource YAML. The following snippet shows the structure of a Presto Kubernetes Resource YAML including all available properties (with defaults). When defining your particular Presto cluster resource you need to specify only the properties with non-default values.

apiVersion: starburstdata.com/v1
kind: Presto
metadata:
  name: presto-cluster-name
spec:
  nameOverride: ""
  clusterDomain: cluster.local
  environment: ""
  additionalJvmConfigProperties: ""
  additionalCatalogs: {}
  additionalEtcPrestoTextFiles: {}
  additionalEtcPrestoBinaryFiles: {}
  licenseSecretName: ""
  imageNamePrefix: ""
  additionalBootstrapScriptVolume: {}
  additionalBootstrapScriptVolumes: {}

  prometheus:
    enabled: false
    additionalRules: {}

  service:
    type: ClusterIP
    name: ""
    additionalSpecProperties: {}
    nodePort: 31234

  image:
    name: starburstdata/presto
    pullPolicy: IfNotPresent
    tag: 312-e.7.k8s-0.14

  memory:
    nodeMemoryHeadroom: 2Gi
    xmxToTotalMemoryRatio: 0.9
    heapHeadroomPerNodeRatio: 0.3
    queryMaxMemory: 1Pi
    queryMaxTotalMemoryPerNodePoolFraction: 0.333

  coordinator:
    cpuLimit: ""
    cpuRequest: 16
    memoryAllocation: 60Gi
    nodeSelector: {}
    affinity: {}
    additionalProperties: ""

  worker:
    count: 2
    autoscaling:
      enabled: false
      minReplicas: 1
      maxReplicas: 100
      targetCPUUtilizationPercentage: 80
    deploymentTerminationGracePeriodSeconds: 7200 # 2 hours
    cpuLimit: ""
    cpuRequest: 16
    memoryAllocation: 100Gi
    nodeSelector: {}
    affinity: {}
    additionalProperties: ""

  readinessProbe:
    initialDelaySeconds: 5
    periodSeconds: 5
    timeoutSeconds: 15

  livenessProbe:
    initialDelaySeconds: 300
    periodSeconds: 300
    failureThreshold: 1
    timeoutSeconds: 15

  hive:
    metastoreUri: ""
    awsSecretName: ""
    googleServiceAccountKeySecretName: ""
    azureAbfsSecretName: ""
    azureAdlSecretName: ""
    additionalProperties: ""
    internalMetastore:
      mySql:
        jdbcUrl: ""
        username: ""
        password: ""
      postgreSql:
        jdbcUrl: ""
        username: ""
        password: ""
      internalPostgreSql:
        enabled: false
        image:
          name: postgres
          pullPolicy: IfNotPresent
          tag: 9.6.10
        storage:
          className: ""
          size: 1Gi
          claimSelector: {}
        memory: 2Gi
        cpu: 2
        nodeSelector: {}
        affinity: {}
      s3Endpoint: ""
      image:
        name: starburstdata/hive-metastore
        pullPolicy: IfNotPresent
        tag: k8s-0.5
      memory: 6Gi
      cpu: 2
      nodeSelector: {}
      affinity: {}

General properties

Property name

Example

Description

nameOverride

-–––————————————-

clusterDomain

nameOverride: presto-cluster

Presto Operator by default assigns a unique name to various Kubernetes resources (e.g. Services). The name consists of the cluster name and a unique suffix. Use this property to assign a static name instead.

clusterDomain: cluster.local

Domain of the K8s cluster.

environment

environment: production

The name of the Presto cluster environment.

licenseSecretName

licenseSecretName: license_secret

Name of a Kubernetes Secret that contains a Presto license file. The license file within the secret should be named signed.license.

imageNamePrefix

imageNamePrefix: gcr.io/project-name/

Specifies prefix of Docker image names used by the Presto cluster. This property enables using a private Docker registry.

image

image:
  name: starburstdata/presto-customized
  pullPolicy: IfNotPresent
  tag: 312-e.7.k8s-0.14

Image section allows you to specify a custom Docker image to be used by Presto cluster.

readinessProbe

readinessProbe:
  initialDelaySeconds: 5
  periodSeconds: 5
  timeoutSeconds: 15

Properties of Presto Coordinator and Worker readiness probes. For more information on readiness probes, refer to Kubernetes probes.

livenessProbe

livenessProbe:
  initialDelaySeconds: 300
  periodSeconds: 300
  failureThreshold: 1
  timeoutSeconds: 15

Properties of Presto Coordinator and Worker liveness probes. For more information on liveness probes, refer to Kubernetes probes.

additionalBootstrapScriptVolume

additionalBootstrapScriptVolume:
  configMap:
    name: my-bootstrap-script

Property of Presto Coordinator and Worker pod. Allows adding a custom bootstrap script to customize the Presto pods. Executed prior to staring Presto on the pod. Refer to Custom Bootstrap Script

additionalBootstrapScriptVolumes

additionalBootstrapScriptVolumes:
  - configMap:
      name: my-bootstrap-script-1
  - configMap:
      name: my-bootstrap-script-2

Property of Presto Coordinator and Worker pod. Allows adding a custom bootstrap scripts to customize the Presto pods. Executed prior to staring Presto on the pod. Refer to Custom Bootstrap Script

Service properties

Access to Presto Coordinator is possible via the Kubernetes Service. By default the service is only accessible within the Kubernetes cluster at http://presto-coordinator-CLUSTER_NAME_UUID.NAMESPACE.svc.cluster.local:8080, where NAMESPACE is the Kubernetes namespace where the given Presto cluster is deployed and CLUSTER_NAME_UUID is the cluster name with unique suffix appended.

Use the service.name Presto resource parameter to make the service name more predictable. For example setting service.name=test-cluster will cause the Presto Coordinator service address to be http://presto-coordinator-test-cluster.NAMESPACE.svc.cluster.local:8080.

Use the nameOverride parameter in order to set CLUSTER_NAME_UUID to a different value.

You can also change type of the Presto Coordinator Service using the service.type parameter. For more information on Kubernetes Service types, refer to Kubernetes Services types.

You can add additional parameters to spec section of Service by using service.additionalSpecProperties Presto resource parameter, e.g.

service:
  additionalSpecProperties:
    loadBalancerIP: 78.11.24.19
  type: LoadBalancer

Use service.nodePort parameter to specify the port on which Presto Coordinator Service should be exposed when service.type is set to NodePort, e.g.

service:
  type: NodePort
  nodePort: 3001

Presto Coordinator properties

Property name

Example

Description

coordinator.cpuRequest

coordinator.cpuLimit

coordinator:
  cpuRequest: 6
  cpuLimit: 32

Specifies the Presto Coordinator pod’s CPU limit and request.

coordinator.memoryAllocation

coordinator:
  memoryAllocation: 60Gi

Specifies Presto Coordinator pod’s memory usage (both request and limit). If empty Presto Coordinator pod will utilize entire memory available on the node.

coordinator.additionalProperties

coordinator:
  additionalProperties: |
    resource-groups.config-file=filename

Specifies additional config.properties properties.

coordinator.nodeSelector

coordinator.affinity

coordinator:
  nodeSelector:
    role: presto
  affinity:
    podAffinity:
      ...

Specifies Presto Coordinator pod’s node selector and affinity.

Presto Worker properties

Property name

Example

Description

worker.count

worker:
  count: 3

Number of Presto worker pods.

worker.autoscaling

worker:
  autoscaling:
    enabled: true
    minReplicas: 1
    maxReplicas: 100
    targetCPUUtilizationPercentage: 80

Configuration of Presto Workers autoscaling. For more information see Horizontal scaling of a Presto cluster.

worker.deploymentTerminationGracePeriodSeconds

worker:
  deploymentTerminationGracePeriodSeconds: 7200

Specifies termination grace period for Presto Workers pods. Presto Worker pods won’t be terminated until queries running on the pod are finished or grace period passes.

worker.cpuRequest

worker.cpuLimit

coordinator:
  cpuRequest: 6
  cpuLimit: 32

Specifies Presto Worker pod CPU limit and request.

worker.memoryAllocation

worker:
  memoryAllocation: 100Gi

Specifies Presto Worker pod memory usage (both request and limit). If empty Presto Worker pod will utilize entire memory available on the node.

worker.additionalProperties

worker:
  additionalProperties: |
    resource-groups.config-file=filename

Specifies additional config.properties properties.

worker.nodeSelector

worker.affinity

worker:
  nodeSelector:
    role: presto
  affinity:
    podAffinity:
      ...

Specifies Presto Worker pod’s node selector and affinity.

General memory properties

memory section specifies general Presto memory configuration

Property name

Example

Description

memory.nodeMemoryHeadroom

memory:
  nodeMemoryHeadroom: 2Gi

Memory headroom that Presto should leave on a node when Presto pods are configured to use entire node memory (empty memoryAllocation configuration property).

memory.xmxToTotalMemoryRatio

memory:
  xmxToTotalMemoryRatio: 0.9

Ratio between Presto JVM heap size and memory available for a Presto pod.

memory.heapHeadroomPerNodeRatio

memory:
  heapHeadroomPerNodeRatio: 0.3

Ratio between memory.heap-headroom-per-node Presto configuration property and Presto JVM heap size.

memory.queryMaxMemory

memory:
  queryMaxMemory: 1Pi

Value of the query.max-memory Presto configuration property.

memory.queryMaxTotalMemoryPerNodePoolFraction

memory:
  queryMaxTotalMemoryPerNodePoolFraction: 0.333

Value the query.max-total-memory-per-node Presto configuration property expressed as fraction of Presto JVM heap size.

Hive connector properties

Starburst Presto on Kubernetes provides automatic configuration of the Hive connector. Such a connector allows you to either access an external Metastore or use built-in internal Presto cluster Metastore as well.

External Metastore

You can configure Presto to use an external Hive Metastore by setting the hive.metastoreUri property, e.g.

hive:
  metastoreUri: thrift://hive-metastore:9083

Internal Metastore

You can configure Presto to use an internal Hive Metastore by setting the hive.internalMetastore.mySql or hive.internalMetastore.postgreSql properties, e.g.

hive:
  internalMetastore:
    mySql:
      jdbcUrl: jdbc:mysql://mysql-server/metastore-database
      username: hive
      password: hivePassword

or

hive:
  internalMetastore:
    postgreSql:
      jdbcUrl: jdbc:postgresql://postgresql-server/metastore-database
      username: hive
      password: hivePassword

In such a case, an additional Hive Metastore pod will be created which will store data in the provided PostgreSQL or MySQL database.

You can also make the internal Metastore to use an internal ephemeral PostgreSQL database. This can be enabled by setting the hive.internalMetastore.internalPostgreSql.enabled to true, e.g.

hive:
  internalMetastore:
    internalPostgreSql:
      enabled: true

In this case, an additional PostgreSQL pod will be created along with the Hive Metastore pod. This Hive Metastore will use the internal ephemeral PostgreSQL as a relational database.

Note: Using the internal ephemeral PostgreSQL is not recommended for production. Data stored within the internal PostgreSQL will be lost when the Presto cluster is terminated.

Note: You cannot use external and internal Metastore at the same time. You also cannot use external and internal relational database for Metastore at the same time.

Using internal Metastore with custom S3 endpoints

It possible to configure an internal Metastore with S3 like data sources. You can specify a custom S3 endpoint via the hive.internalMetastore.s3Endpoint property, e.g.

hive:
  internalMetastore:
    s3Endpoint: http://my-custom-s3-endpoint

Using AWS credentials

You can provide AWS credentials for your Presto cluster as a Kubernetes secret. The secret name can be configured using the hive.awsSecretName property, e.g.

hive:
  awsSecretName: aws-secret-name

Such secret should contain two files:

  • AWS_ACCESS_KEY_ID

  • AWS_SECRET_ACCESS_KEY

which contain corresponding AWS credentials.

Using Azure Storage account key to access ADLS Gen 2

You can provide an Azure Storage account key for your Presto cluster as a Kubernetes secret. The secret name can be configured using the hive.azureAbfsSecretName property, e.g.

hive:
  azureAbfsSecretName: azure-secret-namme

Such a secret should contain two files

  • AZURE_STORAGE_ACCOUNT

  • AZURE_ACCESS_KEY

which contain the Azure Storage account name and key.

Note Internal Metastore won’t work with an Azure Storage account key secret. This means that you need an external/ Metastore service for Presto to work with Azure storage.

Using Azure credentials to access ADLS Gen 1

You can provide Azure credentials for ADLS Gen 1 as a Kubernetes secret. The secret name can be configured using the hive.azureAdlSecretName property, e.g.

hive:
  azureAdlSecretName: azure-secret-namme

Such a secret should contain three files

  • AZURE_ADL_CLIENT_ID

  • AZURE_ADL_CREDENTIAL

  • AZURE_ADL_REFRESH_URL

See Service-to-service authentication with Azure Data Lake Storage Gen1 using Azure Active Directory for more information.

Using Google Service Account key

You can provide a Google Service Account Key for the Presto Cluster as a Kubernetes secret. The secret name can be configured using the hive.googleServiceAccountKeySecretName property, e.g.

hive:
  googleServiceAccountKeySecretName: google-service-account-key-secret-name

Such secret should contain single file:

  • key.json

which contains Google Service Account Key in JSON format.

Specifying additional Hive connector properties

It is possible to specify additional Hive connector properties via the hive.additionalProperties property, e.g.

hive:
  additionalProperties: |
    hive.allow-drop-table=true

Customizing internal Hive Metastore pod

It possible to customize internal Hive Metastore pod’s Docker image via the hive.internalMetastore.image property, e.g.

hive:
  internalMetastore:
    image:
      name: customized-internal-hive-metastore
      tag: 0.1
      pullPolicy: IfNotPresent

It is possible to change the resources required by Hive Metastore’s pod via:

  • hive.internalMetastore.cpu

  • hive.internalMetastore.memory

  • hive.internalMetastore.nodeSelector

  • hive.internalMetastore.affinity

properties.

Customizing internal PostgreSQL Docker pod

It is possible to customize the internal PostgreSQL pod’s Docker image via the hive.internalMetastore.internalPostgreSql.image property, e.g.

hive:
  internalMetastore:
    internalPostgreSql:
      image:
        name: customized-internal-postgresql
        tag: 0.1
        pullPolicy: IfNotPresent

It is possible to change the resources required by PostgreSQL’s pod via:

  • hive.internalMetastore.internalPostgreSql.cpu

  • hive.internalMetastore.internalPostgreSql.memory

  • hive.internalMetastore.internalPostgreSql.nodeSelector

  • hive.internalMetastore.internalPostgreSql.affinity

properties.

It is possible to specify the storage used by PostgreSQL’s pod via: the hive.internalMetastore.internalPostgreSql.storage property, e.g.

hive:
  internalMetastore:
    internalPostgreSql:
      storage:
        className: db-class
        size: 20Gi
        claimSelector:
          matchLabels:
            release: "stable"
          matchExpressions:
            - {key: environment, operator: In, values: [dev]}

Prometheus support

Presto Kubernetes supports exposing Presto metrics to Prometheus. You can use the Prometheus Operator in order to set up Prometheus in your Kubernetes cluster and collect Presto metrics.

To enable Presto Prometheus metric endpoints set the prometheus.enabled property to true. This will cause the following additional Kubernetes services to be created:

  • prometheus-coordinator-CLUSTER_NAME_UUID

  • prometheus-worker-CLUSTER_NAME_UUID

where CLUSTER_NAME_UUID is the cluster name with a unique suffix. Use the nameOverride property in order to set CLUSTER_NAME_UUID to a different value.

Those services will expose Presto metrics in the Prometheus format. You can use ServiceMonitor resources from the Prometheus Operator to make Prometheus collect metrics from those endpoints. In order to match Presto Prometheus endpoints you can use labels, e.g.

matchLabels:
  instance: CLUSTER_NAME_UUID
  role: prometheus-coordinator

The following Presto and JVM metrics will be exported by default:

  • running_queries - number of currently running queries

  • queued_queries - number of currently queued queries

  • failed_queries - total count of failed queries

  • jvm_gc_collection_seconds_sum – total GC (young and old) time in seconds

  • jvm_gc_collection_seconds_count – total GC (young and old) count

  • jvm_memory_bytes_committed, jvm_memory_bytes_init, jvm_memory_bytes_max, jvm_memory_bytes_used – JVM memory usage metrics (heap and non-heap). For more information see MemoryMXBean.