Starburst | Starburst cluster sizing, and performance tuning

Starburst cluster sizing, and performance tuning

Cast: Dain Sundstrom and Manfred Moser

Video date: September 9, 2020

Running time:

Abstract: This training session is designed to help you tune and size deployments for optimal performance.

Product: Starburst Enterprise, Starburst Galaxy, Trino

Description

This training session is geared towards helping users tune and size their deployment for optimal performance. Delivered by Dain Sundstrom, this session covers the following topics:

Cluster configuration and node sizing
Memory configuration and management
Improving task concurrency and worker scheduling
Tuning your JVM configuration
Investigating queries for join order and other criteria
Tuning the cost-based optimizer

Detailed topics with timestamps

Clicking the timestamp links below will take you to YouTube, or you can scrub to that timestamp in the video player above.

Welcome - 0:00
General Strategy - 5:45
Baseline Advice - 11:53
Cluster Sizing / CPU and Memory - 14:57
Machine Sizing - 34:46
- Memory - 34:58
- Memory Allocations - 39:55
- Shared Join Hash - 46:59
- Distributed Join - 49:11
- Skew - 50:58
- Use bigger machines - 55:03
- Machine Types - 58:40
Additional Thoughts - 1:03:20
- Hash join vs (sort) merge join - 1:03:25
- Spilling - 1:04:43
- Small Clusters - 1:07:54
Tuning the Workload - 1:23:45
- Query Plan - 1:24:52
- Precomputing - 1:30:00
- Connectors - 1:34:53
Hive Data Organization - 1:39:27
- Organize the data for the Hive connector - 1:39:41
- Hive Partitioning - 1:42:13
- Hive bucketing - 1:43:59
- Orc and Parquet - 1:46:26
- File Size - 1:51:02
- Bad Parquet Files - 1:53:13
- Rewrite table with ORC writer - 1:54:29
Making Queries Faster - 1:55:37
- What to look for in a query - 1:57:09
- More hardware - 2:01:18
- Under-utilization - 2:02:32
- Hive Caching - 2:05:29
Sharing Resources / Resource Groups - 2:08:34

Video library

Starburst cluster sizing, and performance tuning

Description

Detailed topics with timestamps