Workload management#

When you use Starburst Enterprise to process large workloads, you may encounter observability and resource management inefficiencies such as uneven resource distribution and query bottlenecks.

Workload management provides you with detailed insights to efficiently manage and monitor resource groups, check their activity, identify related queries, and pinpoint bottlenecks caused by overly strict limitations, ensuring smoother and more efficient operations.

Note

Workload management is available as a public preview in Starburst Enterprise. Contact Starburst Support with questions or feedback.

Navigate to the Workload management tab in the Starburst Enterprise web UI to access the feature.

Monitor resources#

The workload management tab describes how your your cluster’s resources are handling the current workload of active and queued queries.

Active groups: Name of the resource groups.
Queued queries: Current number of queries waiting to be run.
Running queries: Current number of running queries.
Consumed CPU Quota: Current CPU usage of the running queries adjusted against the quota limits.
Memory: Memory usage of the running queries.
Query overview: Once selected, displays the list of queries that have been assigned to the selected resource group.

Workload group details#

To view a group’s metrics in more detail, click the name of a resource group in the Workload management tab.

The workload group details pane provides a detailed historical view of a resource group’s metrics. In addition, you can apply the following filters to the charted metrics:

Start date
End date

Manage groups#

Built-in resource groups are an improved version of a SEP resource group that let users limit the number of queued and running queries inside a group. Each resource group can either have subgroups or selectors configured but not both. Selectors define how to match queries that are managed by the given resource group.

The built-in resource group’s matching algorithm differs from the algorithm used for SEP resource groups. Built-in groups are matched based on the order of selectors.

Resource groups in the workload management feature are traversed in order to match a group for a given query. The order of root groups and subgroups matters. If the group has selectors configured, the selectors are checked to see if there is a match. If the group has subgroups, the subgroups are checked in order recursively. If no matching group is found, the query will fail.

For more information about resource groups, see the documentation.

To manage built-in resource groups, click the Manage groups button.

Configuration#

Any configuration changes are applied immediately after the configuration is saved, without the need for the coordinator to restart.

Built-in resource groups are enabled by default unless another resource group was configured in your etc/resource-groups.properties file and contains the resource-groups.configuration-manager configuration property.

If this is the case, the SEP resource groups must be disabled and the resource-groups.configuration-manager property must be set to starburst to enable built-in resource groups.

In the workload management tab, click Manage groups to configure new or existing resource groups. The manage resource groups menu has the following configurable attributes:

Attribute	Required	Description
`Name`	required	Name of the resource group. This can be a template. Templates let administrators construct resource group trees dynamically. For example, in the `pipeline_${USER}` group, `${USER}` is expanded to the name of the user that submitted the query. See the resource groups example for more information.
`Max queued`	required	Maximum number of queued queries. Once the limit is reached new queries are rejected.
`Concurrency limit`	required	Maximum number of running queries. Once the limit is reached new queries are queued.
`Selectors`	Optional	For more information, see Selector rules.

To add and configure a new resource group, click Add root group and configure the properties as appropriate for your setup.

Example use case#

The workload management feature lets you monitor cluster utilization for queries and for users that appear to be causing resource bottlenecks.

As an admin, assume you have a group configuration Global/User/${USER}. If the user is utilizing too many CPU resources, you can create a separate group for that user Global/User/Bad_user with a lower concurrency limit.

Migration to built-in resource groups#

If you have SEP resource groups and want to manage them under the Workload management tab, you must migrate them to built-in resource groups.

To migrate from your legacy resource groups, move the selectors inside the matching resource group. Then make the resource groups and subgroup order match the order of the selectors.

Additionally, you must drop schedulingPolicy and any memory or CPU limits as those are not supported by the built-in resource groups.