Manage cluster groups#

A cluster group is a set of clusters that Starburst control plane treats as one destination for queries. Every cluster belongs to exactly one cluster group, and exactly one cluster group is the default. Use the Cluster groups tab of the Clusters pane to create, edit, and delete cluster groups, and to set the default group. To work with individual clusters, see Manage clusters.

The Cluster groups tab#

The Cluster groups tab lists the cluster groups that you can read. For each group, it shows the name, the number of clusters that you can see in the group, and the concurrency limit for each cluster. You can filter the list.

In managed mode, you can read the groups that you own. If you hold the MANAGE_SECURITY, CREATE_CLUSTER, or MANAGE_QUERY_ROUTING_RULES privilege, you can read every group. If you can read no group, the list is empty. In standalone mode, the tab shows every group.

Create a cluster group#

You need the CREATE_CLUSTER privilege.

  1. On the Cluster groups tab, click Create cluster group.

  2. Enter a name for the group. The name can have up to 255 characters, and it must differ from the name of every other cluster group. Names are case-sensitive, so ETL and etl are two groups.

  3. Optional: enter a description.

  4. Set the concurrency limit for the clusters in the group. See Set query limits.

  5. Click Create cluster group.

Your role owns the group that you create. You can add clusters to the group when you create or edit a cluster. See Manage clusters.

Edit a cluster group#

You can change the name, the description, and the concurrency limit of a cluster group that you own. Control plane identifies a group by its ID, not its name, so routing rules and clusters keep working after you rename a group.

If someone else changes the group while you edit it, your save fails. Reload the group, and make your change again.

Set the default cluster group#

The default cluster group receives a cluster that you add without naming a group. It also receives a query that no routing rule matches.

To change the default, select Set as default from the more_vert menu of another cluster group. You need the MANAGE_SECURITY privilege. In managed mode, owning the group is not enough. When you set a new default, the previous group stops being the default. The change takes effect at once on all control plane instances.

Set query limits#

The concurrency limit of a cluster group is the number of queries that can run at the same time on each cluster in the group. It is a whole number from 1 to 10,000. New groups start with a limit of 100.

A cluster without a limit of its own uses the limit of its group. A limit that you set on a single cluster overrides the limit of the group. See Set query limits.

The limits have an effect only when smart load balancing is on. Smart load balancing is on by default with auto tag-based routing, and it must be off in standalone mode, so the limits have no effect there.

Smart load balancing#

The smart load balancing statement queue holds a SQL statement in control plane until a cluster can run it. It makes the query limits that you set for a cluster group work. When every cluster in the group is at its limit, new queries wait in the queue, so they do not overload a cluster.

The statement queue is on by default when control plane uses auto tag-based routing and cluster authentication is enabled. With the queue, control plane stores each statement and sends it to a cluster later, instead of passing it straight to a cluster. This can change how long a query takes to start, and how its life cycle looks to a client. To go back to sending each statement straight to a cluster, turn the queue off. See Turn smart load balancing on or off.

How smart load balancing works#

When the smart load balancing statement queue is on, control plane handles each statement in the following way:

  1. A client submits a statement to control plane. Control plane rejects the statement if it is longer than the maximum statement length, or if the queue is full.

  2. Control plane selects a cluster group for the statement by using your routing rules, and puts the statement in the queue.

  3. Control plane checks the queue. This usually happens almost immediately, and it can take up to 10 seconds. For each statement, it looks in the cluster group for a cluster that is below its query limit.

  4. If a cluster has capacity, control plane sends the statement to it. The load balancing strategy that you set picks the cluster among those that have capacity. The strategy skips every cluster that is at its limit.

  5. If no cluster in the group has capacity, the statement stays in the queue. Control plane tries again at the next check.

  6. If a statement waits for longer than the maximum queued time, control plane expires it.

A statement is in one of these states: queued, dispatching, dispatched, failed, cancelled, or expired. A statement is dispatching while control plane sends it to a cluster. If that takes longer than the maximum dispatching time, control plane returns the statement to the queue.

Control plane keeps a finished statement, which is one that is dispatched, failed, cancelled, or expired, for the retention period. Then it deletes the statement.

What smart load balancing changes#

  • Query limits. The limits that you set for the clusters in a group have an effect only when the statement queue is on. When the queue is off, the limits have no effect.

  • Load balancing. With the queue, both load balancing strategies skip clusters that are at their limit. See Key use cases.

  • How control plane counts the queries of a cluster. Control plane counts a query against a cluster from the moment that it picks the cluster. Clusters report their running queries every 5 seconds. The count therefore includes queries that did not come through control plane, such as the refresh of a materialized view and table maintenance.

  • The backend database. Control plane stores queued statements in the query_queue table in the backend database. The table holds the text of each statement and the headers that the client sent. Control plane creates the table when it starts.

Statements that name a cluster#

With smart load balancing on, you can still send a statement to one cluster through the address of the cluster, /clusters/<name>/. The query editor and the privilege pickers in the cluster web UI use it for statements such as SHOW CATALOGS. Control plane does not load balance these statements. It queues, tracks, and audits them, and it records the cluster group of the cluster in query history. A request for a cluster that does not exist returns HTTP status 404.

Turn smart load balancing on or off#

The smart load balancing statement queue is available only with auto tag-based routing, which is the default routing type, gateway.routing.type=AUTO_TAGS. It needs cluster authentication: control plane does not start if the queue is on and gateway.clusters.authentication.enabled is not true. The error says: statement-queue.enabled requires gateway.clusters.authentication.enabled to be true. Cluster authentication is off by default.

  • To keep the smart load balancing queue on, set gateway.clusters.authentication.enabled to true, and set gateway.clusters.authentication.issuer and gateway.clusters.authentication.secret. Make sure that the clusters accept the tokens that control plane signs.

  • To turn the smart load balancing queue off, set statement-queue.enabled to false in the control plane’s config.properties.

  • With background work turned on, do not turn the queue off. Control plane does not start if background-work.enabled is true and statement-queue.enabled is false. The error says: background-work.enabled requires statement-queue.enabled to be true.

  • In standalone mode, you must turn the queue off.

The following properties control the queue:

Property

What it controls

Default

statement-queue.max-size

The most statements that the queue holds, queued or dispatching

100000

statement-queue.max-statement-length

The longest statement, in characters, that control plane accepts

1000000

statement-queue.max-queued-time

How long a statement can wait before it expires

10m

statement-queue.retention.period

How long control plane keeps finished statements

3d

Delete a cluster group#

You must own the cluster group to delete it. Control plane refuses to delete a group that something still refers to. Before you delete a group:

  • Move its clusters to another group.

  • Change or delete the routing rules that route to the group.

  • Wait until no statement for the group is queued or being sent. See Smart load balancing.

  • Make another cluster group the default. Control plane refuses to delete the default group.

If control plane routes a statement to the group while you delete it, control plane rejects the statement with the error Cluster group <id> no longer exists; retry the query. Retry the query, and control plane routes it again.

Who can do what#

Task

Requirement in managed mode

Create a cluster group

The CREATE_CLUSTER privilege

Edit or delete a cluster group

You own the group

Read a cluster group

You own it, or you hold MANAGE_SECURITY, CREATE_CLUSTER, or MANAGE_QUERY_ROUTING_RULES

Set the default cluster group

The MANAGE_SECURITY privilege, whether or not you own the group