How coordinator high availability works#
Starburst control plane provides coordinator high availability (HA) for Starburst Enterprise platform (SEP) clusters. When a coordinator fails, the control plane automatically promotes a healthy standby coordinator to handle the traffic.
Overview#
In a standard SEP deployment, each cluster has a single coordinator that acts as the entry point for all queries. If this coordinator becomes unavailable, queries fail until you restore the coordinator.
High availability maintains multiple coordinators for each cluster:
One coordinator is active and handles queries.
The other coordinators are on standby.
If the active coordinator fails, the control plane promotes a standby coordinator to handle traffic.
Requirements#
Before you enable coordinator high availability, ensure your deployment meets the following requirements:
Multiple coordinators: Deploy at least two coordinator instances per environment with the same size.
Environment property: All nodes in a cluster must share the same
node.environmentvalue.Network connectivity: Bidirectional network access between the control plane and all coordinators is required for announcements and health monitoring. In Kubernetes, the control plane calls the pod IP address that the coordinator announces. If the control plane and the cluster run in different Kubernetes clusters, the network between them must route to the pod IP addresses.
Consistent configuration: All coordinators in an environment must have identical configurations, including catalogs, connectors, and security settings.
Discovery: The control plane runs coordinator high availability only for clusters that use discovery. Register the cluster in the control plane with discovery turned on. See Register the cluster with discovery.
Announcements: In standalone mode, the coordinators and workers announce themselves with
discovery.uri. In managed mode, they usedeployment.modeanddeployment.manager.uri. See Configure coordinator high availability.
Coordinator roles#
The control plane assigns each coordinator one of the following states:
Role |
Description |
|---|---|
|
The coordinator that is processing queries. |
|
A healthy backup coordinator ready for promotion if the active coordinator fails. |
|
A coordinator removed from the HA pool. Restart the coordinator to return it to the pool. |
|
A stale coordinator record whose URI now serves a different coordinator
instance. This occurs when a coordinator restarts and receives a new
instance identifier. The stale record is excluded from HA management. The
new coordinator instance registers itself automatically as |
The Clusters pane does not show the count of decommissioned coordinators.
Health status#
The control plane assigns one of the following health statuses to each coordinator:
Status |
Condition |
|---|---|
|
The last health check succeeded. |
|
Health check data is older than 30 seconds, or the coordinator has fewer than three consecutive failures over 10 seconds. |
|
More than three consecutive health check failures over 10 seconds. |
In the public cluster API, the health status values are Healthy,
NotHealthy, and Unknown.
When a network issue occurs, health checks fail immediately and the coordinator
status changes to UNKNOWN. After three or more consecutive failures over 10
seconds, the status changes to NOT_HEALTHY and the control plane triggers a
failover.
Coordinator lifecycle#
The following sections describe how the control plane registers coordinators, monitors coordinator health, and handles failover.
Coordinator registration#
When you start a SEP coordinator, it periodically sends announcements to the control plane. The control plane processes these announcements as follows:
Retrieves the coordinator’s server info
Registers the coordinator with a
STANDBYroleRecords the coordinator’s URI and environment
Begins health monitoring for the coordinator
Worker nodes also send announcements to the control plane. The control plane proxies these announcements to the active coordinator. This ensures worker nodes connect to the active coordinator during failover.
Health monitoring#
The control plane monitors all your registered coordinators:
Runs health checks every five seconds
Verifies coordinators are reachable and ready to handle queries
Failed checks increment a failure counter; successful checks reset the failure counter
A health check fails when a coordinator:
Is unreachable (connection refused, timeout, network error)
Returns an HTTP status code other than 200
Reports it is not ready to handle queries
During each health check, the control plane also verifies that the instance
identifier returned by the coordinator matches the stored identifier. If the
identifiers differ, the coordinator at that URI has been replaced by a new
instance (for example, after a pod restart). The control plane immediately
marks the stale record as GONE. The new coordinator instance at the same URI
announces itself and registers as a fresh STANDBY.
Automatic failover#
When the monitor detects three or more health check failures over 10 seconds,
the coordinator status changes to NOT_HEALTHY and the control plane initiates
failover:
The control plane selects a healthy coordinator with a
STANDBYroleThe failed coordinator is marked as
DECOMMISSIONEDThe selected coordinator is promoted from
STANDBYtoACTIVEQuery traffic automatically routes to the new active coordinator
Note
If no active coordinator exists (for example, during initial startup), the
control plane selects a healthy standby and promotes it to ACTIVE.
Multiple active coordinators#
If multiple coordinators become active simultaneously:
The monitor detects multiple active coordinators
The control plane selects one healthy coordinator to remain active
All other active coordinators are demoted to
DECOMMISSIONED
Limitations#
Coordinator high availability operates at the environment level. Each cluster independently maintains its own active coordinator.
The control plane does not retry running queries when a coordinator fails. Configure your client application to retry failed queries.
Failover takes approximately 10-15 seconds because the control plane waits for three consecutive health check failures before promoting a standby. If a coordinator becomes unresponsive without disconnecting, failover takes up to two minutes by default. To reduce this time, configure
clusters.http-client.request-timeout.To return a decommissioned coordinator to the HA pool, restart the coordinator.
When the SEP Helm chart is deployed with
internalTls: true, the SEP init container automatically overrides anydiscovery.urivalue you set inconfig.propertieswithhttps://{node.environment}:{httpsPort}(for example,https://mycluster:8443). Settingdiscovery.uriin your Helm values has no effect in this configuration. To use coordinator high availability withinternalTls: true, you must make the control plane accessible at the address that the init container generates. The Helm chart does not support this, so it requires custom setup. Managed mode needsdiscovery.urito equaldeployment.manager.uri.Send queries through the control plane. If a client connects to the cluster directly, for example through the Kubernetes Service of a Helm installation, the connection can reach the standby coordinator. The Service selects both coordinators, and it cannot tell the active coordinator from the standby coordinator.
The coordinator Deployment of the Helm chart uses the
Recreatestrategy, so a Helm upgrade stops both coordinators. Coordinator high availability does not keep the cluster available during an upgrade.After a coordinator restarts, the control plane can keep calling the old address of the coordinator for several minutes. Restart the control plane pods to fix it.