Verify and troubleshoot load balancing#
Verify query distribution#
Check the Cluster and Cluster groups columns on the Starburst control plane Query insights page to see which cluster and cluster group handled each query.
Troubleshooting#
If you experience unexpected query behavior, check the following common issues.
All queries route to one cluster#
If all queries route to one cluster:
Verify the cluster group contains more than one cluster.
Check if the queries share a transaction ID. Control plane routes all queries in a transaction to the same cluster.
One cluster receives no queries#
If one cluster receives no queries, verify the cluster is active and healthy.
Check that its status is not UNSUPPORTED. See Monitor cluster status.
Queries are not evenly distributed#
If queries are not evenly distributed across clusters:
Allow more time for distribution to balance out.
Check if one cluster is servicing workloads directly, bypassing control plane.
Check the query limit of each cluster. If the clusters in the group have different sizes, set the limit of each cluster to match its capacity. See Set query limits.
Tail latencies are higher than expected#
If tail latencies are higher than expected, consider configuring resource groups or session properties on your clusters.
Queries fail with object not found error#
If queries fail with object not found or similar errors, check the following:
In standalone mode, confirm that all catalog configurations, database objects, authorizations, and privileges are the same on all clusters in the cluster group.
In managed mode, control plane gives every cluster the same catalogs. Confirm that the database objects exist on every cluster in the group, and that the user has the privileges.
Control plane does not start, and the error mentions the statement queue and cluster authentication#
If the error says statement-queue.enabled requires gateway.clusters.authentication.enabled to be true, the smart load balancing
statement queue is on, and it needs cluster authentication. The queue is on by
default. Set gateway.clusters.authentication.enabled to true, with an issuer
and a secret. Or set statement-queue.enabled to false to send statements
straight to the clusters.
Control plane does not start, and the error mentions background work and the statement queue#
If the error says background-work.enabled requires statement-queue.enabled to be true, background work is on, and it needs the statement queue. Set
statement-queue.enabled to true, or turn off background-work.enabled.
Queries wait before they start#
Every cluster in the cluster group is at its query limit. Raise the limit, or add a cluster. See Set query limits. Usually, control plane starts a queued statement almost immediately when a cluster has capacity. In rare cases, for example when there is a problem, it can take up to 10 seconds.
Queries wait although a cluster looks idle#
Control plane counts the queries that a cluster reports, including queries that did not come through control plane, such as the refresh of a materialized view and table maintenance. These queries count against the limit of the cluster. Check the running queries of the cluster, and raise its limit if you need to. See Set query limits.
A query fails after a wait#
The statement waited longer than the maximum queued time, which is 10 minutes by default. Or the queue was full, or the statement was longer than the maximum statement length. See Smart load balancing.
A query fails with the error that a cluster group no longer exists#
If the error says Cluster group <id> no longer exists; retry the query, an
administrator deleted the cluster group that control plane routed your statement
to. Retry the query. Control plane routes it again, to the group that your
routing rules choose.
A query is rejected because no routing rule matches, or no cluster group is the default#
For these errors, see Verify and troubleshoot workload isolation.