Performance features #

Starburst products are fast, but you can make them even faster with the performance features described in this topic.

Fault-tolerant execution #

Fault-tolerant execution (FTE) allows a cluster to retry queries or parts of query processing in the event of failures without having to start the whole query from the beginning. This is especially useful for long-running queries that are typical with batch processing and Extract Transform Load (ETL) queries.

When a cluster is in fault-tolerant execution mode, intermediate exchange data is spooled and can be re-used by another worker. Queries that require more memory than what is currently available in the cluster are able to share resources.

Learn how to enable FTE in your Galaxy cluster and in your SEP cluster

Warp speed #

Starburst Warp Speed immediately improves performance for recently used datasets by performing an asynchronous indexing and caching process on new data. Warp Speed automatically creates and maintains these indexes and caches based on the characteristics of the processed queries.

This process of creating the indexes and caches is also called warmup. Warmup is performed individually by each worker based on the processed splits and uses the local high performance storage of the worker. This new data is stored on local storage attached to each worker node in the cluster.

Because the data is now available directly in the cluster, and no longer has to be retrieved from the remote object storage, query processing is accelerated when accessing the same data.

Learn how to enable Warp Speed in your Galaxy cluster and in your SEP cluster.