How control plane high availability works#

Starburst control plane supports running multiple instances simultaneously to provide fault tolerance and horizontal scalability. All instances connect to the same backend database, which is the shared source of truth for cluster configuration, routing state, and query history. A load balancer or reverse proxy in front of the instances distributes incoming connections across all healthy ones.

Capabilities and limitations#

Control plane high availability (HA) provides server-side reliability during steady-state operation. Query resilience across an instance failure depends on the client type and on shutdown configuration. It does not guarantee zero-downtime version upgrades. See Upgrades.

If one instance fails, the load balancer routes new connections to the remaining healthy instances. Instances running the same version can be restarted one at a time without downtime, for example to apply a configuration change or recover from a failure. Each query is tracked against the control plane instance that submitted it. Query state itself, which is the backend cluster that is serving the query, is resolved from the shared database. Another instance can therefore take over serving the results of a query if the original instance goes down.

Client type determines whether a query survives an instance failure:

  • SQL clients that poll for results (for example, Trino CLI) retry automatically on a connection failure and resume against a different healthy instance. This works cleanly when the failing instance is restarted gracefully. See the shutdown behavior that follows.

  • The web UI retries only across a backend crash, not a lost tab: if the instance handling the UI’s requests crashes, the UI retries and can resume against a healthy instance. But if the browser tab is refreshed, closed, or otherwise abandoned, the UI loses track of the query and does not resume polling on its own.

Shutdown configuration determines whether draining completes gracefully:

  • Graceful shutdown drains in-flight queries, rejects new ones: When shutdown begins, the instance stops accepting new requests immediately, while requests already in flight continue until they complete or http-server.stop-timeout elapses. http-server.stop-timeout controls how long an instance waits for in-flight requests to complete on shutdown before aborting them. The control plane defaults this to 20s so a query poll caught mid-request during a restart has a chance to finish instead of failing in a way SQL clients cannot retry. This value must stay below the pod or process termination grace period. See terminationGracePeriodSeconds for the Helm chart equivalent. Otherwise the process is killed before the drain completes.

  • What happens on SIGKILL: If terminationGracePeriodSeconds elapses before the process finishes draining in-flight requests, which happens whenever http-server.stop-timeout is not kept below it, Kubernetes sends SIGKILL, forcibly terminating the process immediately. Any requests still in flight at that point fail non-gracefully, since the process has no chance to respond before being killed. Clients like the Trino CLI that poll for results can still retry the nextUri poll against a different healthy instance in this case, since the failure surfaces as a connection error they already know how to retry.

What HA does not guarantee: zero-downtime version upgrades. The control plane runs database schema migrations automatically on startup, and it does not guarantee backward compatibility between schema versions. A newer instance can migrate the shared database to a schema that older instances can no longer read. In general, a version upgrade therefore needs every instance to run the same version, which can mean downtime. Restarting one instance at a time is safe when all instances run the same version.