Starburst Galaxy

  •  Get started

  •  Working with data

  •  Data engineering

  •  Developer tools

  •  Cluster administration

  •  Security and compliance

  •  Troubleshooting

  • Galaxy status

  •  Reference

  • Configure autoscaling #

    Autoscaling adds one or more workers to get the combined CPU usage of all workers below 60%. If the CPU usage continues to climb and exceeds 60%, the process repeats until the specified maximum number of workers is reached. The upscaling process takes approximately four minutes to make the first adjustment.

    Clusters automatically scale down to the minimum number of workers when the combined CPU usage of all workers drops below 60%. Autoscaling removes one or more workers until CPU usage approaches 60%. The downscaling process takes approximately 15 minutes to make the first adjustment.

    Get started #

    To configure autoscaling, you must have the privileges to create and edit a cluster. By default, the accountadmin role has the privileges to configure autoscaling.

    You can configure autoscaling when creating a new cluster or when editing an existing cluster.

    Configure new cluster #

    In the navigation, click Admin > Clusters.

    1. Click Create cluster. See Create a cluster for details.

    2. In the Cluster type section:

      • Use the slider scale to set the minimum and maximum number of workers to scale between. The Cluster size should now be Custom.

          Cluster size custom and sliding scale

    3. Click Create cluster.

    Configure existing cluster: #

    In the navigation, click Admin > Clusters.

    1. Choose an existing cluster from the clusters list and open the Edit cluster panel:

      • Click the options menu, and select Edit cluster. The Edit cluster panel should now be open.

    2. In the Cluster type section:

      • Use the slider scale to set the minimum and maximum number of workers to scale between. The Cluster size should now be Custom.
    3. Click Save changes.

    When to use autoscaling #

    There are many reasons to configure autoscaling on your cluster. Here are some things to consider:

    • Autoscaling can only be used with the Custom cluster size.
    • Default cluster sizes start with the pre-specified number of workers and end with that number, regardless if that many workers are needed for a given query.
    • A default cluster size is better when running quick or short workloads and suspending the cluster once done.
    • Autoscaling is more efficient when workflows are large, vary over time, or are automated. This allows the cluster to adjust resource usage to the needs of the workload.

    Other useful resources: