AWS EKS networking #

This document helps you to understand networking requirements and configuration for Starburst Enterprise platform (SEP) on AWS EKS.

Best practice: Use a single availability zone inside your existing VPC #

When deploying SEP into an AWS EKS cluster, it is important to ensure that your cluster resources are located with the following in mind:

  • cluster communications latency
  • data ingress/egress costs
  • IP address availability

Using your existing VPC is a key cost control measure. It ensures that costs associated with data ingress and egress are kept to a minimum. Every new VPC comes with a NAT gateway. And, while NAT gateways are inexpensive, costs for data transferred through that gateway add up very quickly. As a best practice, placing your SEP inside of your existing VPC, co-resident with as many of your data sources as possible not only keeps costs down, it also greatly simplify networking and security.

Equally as important is performance. No matter if you use an existing or new VPC, SEP must run in a single availability zone (AZ) to ensure the best performance possible. To accomplish this, use node groups. The node groups are then tied to a single AZ using affinity or node selection rules.

For SEP, two managed node groups are required. The SEP coordinator and workers are deployed to one group while support services, such as HMS, Ranger and nginx, are deployed to the second node group.

Spot instances and single AZs #

AWS EC2 Spot instances are sometimes desireable to keep costs down. However, they are inherently unreliable, as they can be recalled by the EC2 platform at any time. Because of how SEP drives performance by dividing query processing for any single query, sudden removal of an expected resource can cause failures. Further, when using a single AZ, Spot instance rebalancing is diminished, as pools in only the single AZ are available to it. Using Spot instances may still be desirable in some cases. In this case, to use Spot instances for the SEP workers, create a third node group to contain them, separating the non-Spot coordinator node group from the Spot worker node group. The remaining node group contains support services as before.

IP address requirements #

An important consideration in using an existing VPC is IP address availability. As part of standing up your SEP cluster, you must ensure that sufficient IP addresses are reliably available for use by your SEP instances.

In EKS clusters, AWS creates one subnet per availability zone. Usually, these are configured as /20 Classless Inter-Domain Routings (CIDR) with 4,091 IP addresses available for use (an additional 5 are reserved by the cluster itself). SEP requires that all hosts, both workers and coordinators, are sized identically. Each of these instances has a maximum number of IP addresses that can be assigned to it, and EKS reserves twice that number of addresses for it.

For purposes of this discussion, we assume an SEP deployment involving six m5.xlarge workers and one m5.xlarge coordinator. Each of this instances can have a maximum of 15 IP addresses in use, one per each of the 15 interfaces it comes with. An additional 15 are reserved, for a total of 30 IP addresses needed per instance. Those seven instances together then require 210 IP addresses:

  ( 7 m5.xlarge instances)
x ( 15 interfaces per instance )
x ( 2 IPs per interface )
= 210 IP addresses needed

In this example, you must ensure that a minimum of 210 IP addresses are reliably available for use by your SEP instances at all times.

Using subnets #

You can use an existing subnet or create a new one. It must be configured with a route out to the Internet either via a NAT gateway or an IGW to allow your EKS cluster to communicate with the AWS EKS management backplane. Cost considerations for these communications are minimal.

Considerations if you must use VPC peering #

If you cannot place SEP within your current VPC because of a scarcity of IP addresses, as an alternative you can create a peering connection with the new EKS cluster’s VPC to avoid the often cost-prohibitive operation of all data passing through the NAT gateway. VPC peering requires additional setup, and comes with potential downsides:

  • Does not scale well.
  • Transitive routing is not available.
  • Peering connections are a resource that must be managed.
  • Firewall rules must be carefully managed.

Additionally, you must ensure that the CIDR you set for the new SEP VPC does not match or overlap with your existing VPC’s CIDR. VPCs with overlapping CIDRs cannot create peering connections.

The following are required when using internet-facing ALBs:

  • Subnets tagged with kubernetes.io/cluster/cluster-name → shared
  • Subnets tagged with kubernetes.io/role/elb → 1

The following is required when using ALBs that are not internet-facing:

  • kubernetes.io/role/internal-elb → 1

Configure DNS #

Install the ExternalDNS add-on to automatically add DNS entries to Amazon Route 53 once ingress resources are created, as shown in the following example external-dns.yaml file:

provider: aws
aws:
  zoneType: public
  region: us-east-2
txtOwnerId: <txtOwnerId>   # Identify this
txtPrefix: <txtPrefix>     # external DNS instance
zoneIdFilters:
  - <HostedZoneID> # ID of AWS R53 Hosted Zone

# This allows external-dns to delete entries as well
policy: sync

Run the following commands to complete the install. Be sure to use the latest artifact of external-dns and specify its version in the command:

$ helm repo add bitnami https://charts.bitnami.com/bitnami
$ helm repo update
$ helm upgrade external-dns bitnami/external-dns --namespace kube-system --install --version <X.Y.Z> \
  --values external-dns.yaml

Best practice: Configure ingress access for SEP and Ranger through a controller #

Accessing your SEP cluster from external clients requires access to the coordinator and Ranger pods within the K8s cluster. This can be handled either through an AWS layer 7 load balancer (ALB) ingress controller, or by deploying an Nginx ingress controller behind a classic layer 4 load balancer.

There are technical considerations for either approach:

  • Using an ALB ingress controller
    • TLS termination is taken care of in the load balancer.
    • Easy to integrate AWS Certificate Manager for publicly-signed certificates.
    • A separate load balancer resource is be created for each service (Starburst, Ranger, etc.)
  • Using an Nginx ingress controller
    • Nginx can route traffic to different services using a single load balancer.
    • Internal or alternate CA issue certificates, as it is not possible to use Amazon Certificate Manager with an Nginx pod.

For either approach, you must first configure and apply changes to your cluster’s ingress.yaml file. When you have verified that the ingress for the cluster is working properly, proceed with configuring and deploying your ingress controller of choice.

Configure ingress.yaml #

The following ingress.yaml file provides the necessary networking configuration for both SEP and Ranger. You can safely ignore the Ranger configuration if your organization does not use it:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: starburst
  annotations:
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/group.name: sep
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80},{"HTTPS": 443}]'
    alb.ingress.kubernetes.io/ssl-redirect: '443'
    alb.ingress.kubernetes.io/success-codes: 200,303
    alb.ingress.kubernetes.io/tags: key1=value1, key2=value2
    alb.ingress.kubernetes.io/certificate-arn: <cert-??>
spec:
  tls:
    - hosts:
        - <starburst-hostname>
  rules:
    - host: <starburst-hostname>
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: starburst
                port:
                  number: 8080
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ranger
  annotations:
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/group.name: sep
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80},{"HTTPS": 443}]'
    alb.ingress.kubernetes.io/ssl-redirect: '443'
    alb.ingress.kubernetes.io/success-codes: 200,302
    alb.ingress.kubernetes.io/tags: key1=value1, key2=value2
    alb.ingress.kubernetes.io/certificate-arn: <cert-??>
spec:
  tls:
    - hosts:
        - <ranger-hostname>
  rules:
    - host: <ranger-hostname>
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: ranger
                port:
                  number: 6080

The following are important to remember as you modify the preceding example with the particulars of your organization:

  • alb.ingress.kubernetes.io/certificate-arn - If AWS ACM contains multiple certificates matching the hostname, specify certificate ARN explicitly, otherwise you can remove this annotation.
  • alb.ingress.kubernetes.io/group.name - SEP forces the ALB controller to combine multiple Kubernetes ingress resources into a single ALB on AWS, reducing costs.
  • alb.ingress.kubernetes.io/success-code - Contains HTTP response codes of the root (/) resource that are used by target group health checks. By default they accept only 200, but SEP responds with 303, and Ranger with 302.
  • alb.ingress.kubernetes.io/listen-ports - Ensures that ALB listens on both HTTP and HTTPS ports
  • alb.ingress.kubernetes.io/ssl-redirect - Redirects from HTTP to HTTPS

Once all changes to ingress.yaml are made, apply the customizations using the following command in the namespace that contains SEP:

$ kubectl apply -f ingress.yaml

You must wait until the ALB status changes from Provisioning to Ready before the networking behaves as expected with the configuration changes.

Using an AWS load balancer controller #

The AWS documentation provides complete instructions for installing the ALB ingress controller on your EKS cluster. The following example commands provide an overview of the process to install ALB controller that automatically discovers ingress resources and propagates them to AWS ALB and target groups.

$ helm repo add eks https://aws.github.io/eks-charts
$ helm repo update
$helm upgrade -i aws-load-balancer-controller eks/aws-load-balancer-controller -n kube-system \
  --set clusterName=<cluster-name>

Once the ALB controller is installed, you must define ingress in your SEP chart:

expose:
  type: "ingress"
  ingress:
    ingressName: "starburst-ingress"
    serviceName: "starburst"
    servicePort: 8080
    tls:
      enabled: true
    host: "starburst.example.com"
    path: "/*"
    annotations:
      kubernetes.io/ingress.class: alb
      alb.ingress.kubernetes.io/scheme: interal
      alb.ingress.kubernetes.io/target-type: ip

With the defined host, the ingress controller automatically associates certificates in ACM with a matching Subject Alternative Name or with a wildcard such as “*.example.com” with this load balancer.

By default, the controller creates an ALB, configures it with TLS certificate stored in the AWS Certificate Manager (private key is not exposed to Kubernetes), and terminate TLS at ALB.

Target groups are configured to send traffic to ports on Kubernetes worker nodes. A NodePort service type must be configured in Kubernetes to expose specific ports across all nodes.

The following traffic flows apply when using “ingress” as the expose: type:

  • Kubernetes pod: Runs the workload and listens on a port
  • kube-proxy: Listens on NodePort, proxies traffic to the pods in cluster
  • Target Groups: Discover EKS worker nodes, run health checks
  • ALB: Terminates TLS, performs HTTP routing
  • Route 53 A record: Points to ALB endpoints in public subnets

Using Nginx #

The Nginx controller is deployed as an additional pod in the EKS cluster. Certificate management for HTTPS connections is implemented using one of the following Helm-managed certificates:

  • A self-signed certificate
  • A internal CA-signed certificate
  • A Let’s Encrypt-issued certificate using cert-manager.io

The following is an example of Nginx chart to control access to an SEP cluster:

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: nginx-ingress
  annotations:
    kubernetes.io/ingress.class: nginx
    service.beta.kubernetes.io/aws-load-balancer-internal: "true"
    service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
spec: tls:
  - hosts:
    - starburst.example.com
    secretName: tls-secret
  rules:
  - host: starburst.example.com
http: paths:
      - backend:
          serviceName: trino
          servicePort: 8080
 controller:
  extraArgs:
    ##   default-ssl-certificate: "<namespace>/<secret_name>"
    default-ssl-certificate: default/cert-example.com

In this case, the default certificate is a Kubernetes secret specified by applying the following YAML:

apiVersion: v1
kind: Secret
metadata:
  name: cert-example.com
  namespace: default
data:
  ## Both tls.crt and tls.key are base64-encoded PEM files on a single line
  tls.crt: <self-signed certificate>
  tls.key: <self-signed private key>

In the SEP values.yaml file, ingress is exposed via the following YAML block:

expose:
  type: "ingress"
  ingress:
    ingressName: "nginx"
    serviceName: "trino"
    servicePort: 8080
    ingressClassName:
    tls:
      enabled: true
    host: "starburst.example.com"
    path: "/"
    pathType: Prefix
    annotations:
      kubernetes.io/ingress.class: "nginx"

All inbound HTTPS traffic to starburst.example.com is now directed to the SEP coordinator. A similar configuration is applied in the Starburst Ranger Helm with ranger.example.com as the host.