Verify and troubleshoot workload isolation#

Verify routing#

To check how the Starburst control plane routed a query, open the Query insights pane. For each query, the pane shows:

  • The cluster group that the query was sent to.

  • The cluster within that cluster group that ran the query.

  • The routing rule that matched the query, in the Routing rule column. For a query that went to the default cluster group, the column shows Default routing.

Use this information to test your routing rules. When a specific rule routed a query, the rule name appears in the query detail view, so you can confirm that the rule you intended applied.

To test a rule from the UI, run a query in the Query editor pane, and then check Query insights. The query editor has a Client tags field. Control plane sends the tags that you enter with the query, and a routing rule can match them with system.request.clientTag.

Troubleshooting#

If your queries do not route as expected, check the following common issues.

Queries route to the wrong cluster group#

If queries go to a different cluster group than you expect:

  • Confirm that you defined your routing rules correctly.

  • Check the order of the rules. Control plane evaluates them in priority order. If the Default routing rule is above other rules, it matches every query, and those rules never apply. Keep it last.

  • Check the tags of the query. A condition must use the exact name of a system tag, such as system.identity.group, or of a custom tag that one of your tagging rules produces. See Auto tag-based routing.

  • A statement that a client sends directly to a cluster, through the address of the cluster, is not routed by your rules. See Statements that name a cluster.

  • Check which cluster group is the default. A query that no rule matches goes to the default cluster group, and the default can differ from adhoc. See The default routing rule.

A query is rejected with HTTP status 503 and the message No routing rule matched this query#

The full message is No routing rule matched this query, so it could not be assigned to a cluster for execution. Please contact your administrator. This happens when no rule matches and the Default routing rule has been deleted. Create a rule that matches the query, or create a rule that matches every query and sends it to the default cluster group.

A tagging rule or a routing rule does not save#

  • If the error says that a name or a tag key must not start with the reserved prefix system., choose another name. See Reserved names.

  • If the error says Cluster group not found, you cannot use the cluster group, or it no longer exists. See Choose the cluster group.

A query is rejected with HTTP status 503 because no cluster group is the default#

A rule can route a query to the default cluster group. If no cluster group is marked as the default, control plane rejects the query with HTTP status 503. Make a cluster group the default. See Set the default cluster group.

Queries wait before they start#

If queries wait in the statement queue, see Verify and troubleshoot load balancing.

A scheduled refresh of a materialized view runs on another cluster than you expect#

Control plane routes a scheduled refresh with your routing rules. A rule can match the source scheduled-mv-refresh and the client tags of the refresh. If background-work.materialized-view-refresh.routing-group.explicit-override is set, control plane sends every refresh to that cluster group, and it does not evaluate the rules. If no rule matches, control plane rejects the refresh with HTTP 503 and tries it again later. Add or fix a routing rule, or restore the Default routing rule.

Inconsistent query performance within a cluster group#

If some queries in a cluster group perform worse than you expect:

  • Check the query history on each cluster in the cluster group to confirm that no workloads bypass control plane and send queries directly to the cluster.

  • Confirm that the queries in the cluster group are compatible and unlikely to interfere with each other. Consider moving incompatible queries to a different cluster group.