Starburst Admin playbook reference #

Starburst Admin includes numerous playbooks to perform specific actions. You can use them individually or in sequence to satisfy the needs of your specific use case for installing and operating.

The following playbooks are available:

Install cluster #

The install playbook installs the RPM or tarball package on all defined hosts. You are required to place the tar.gz or rpm file in the files directory and specify the version in vars.yml to run the playbook:

ansible-playbook playbooks/install.yml

Errors occurs if RPM and tarball are found or if version values mismatch.

Alternatively to the archive in the files directory on the control machine, you can set the installer_url in vars.yml to point to HTTP or HTTPS URL of the tar.gz or rpm. Ansible then downloads the binary from the URL directly on the hosts. This approach scales is more complex to set up, but scales better since all hosts can download from the URL in parallel. The hosts need to be able to contact the specified URL.

The playbook verifies the availability of the required Java runtime on the host, starting first at the value of JAVA_HOME and then looking at common installation paths. It uses the script files/find_java.sh and fails if no runtime is found.

The playbook installs the RPM or unpack the tarball and create the necessary directories specified via vars.yml.

Ansible can operate as a non-root user, but for some operations, it requires root privileges. By default, sudo commands are used to elevate privileges. Since sudo prompts for the user password, run every ansible-playbook command with the --ask-become-pass parameter as detailed in the become command documentation.

The RPM installation automatically adds a dedicated system user to run SEP as a service. This user owns all configuration files, which should not be readable by other users on the hosts. The tarball installation uses the installation_owner and installation_group defined in vars.yml. The install playbook create the user and group.

Push configurations #

The push-configs playbook generates the configuration files for the coordinator and workers and distributes them to all hosts.

The following resources are used to create the set of configuration files:

  • files/vars.yml - variable definitions to use in the configuration files that are templatized with Jinja2. These files use the extension .j2. Ansible uses the set values and replaces the variable placeholders. For example, the placeholder `` is replaced with the value prodcution from node_environment: production.
  • files/coordinator - the configuration files for the coordinator
  • files/worker - the configuration files for all the workers
  • files/catalog - the catalog properties files to define the connection to any data sources, required on the coordinator and all workers.
  • files/extra/etc - additional files that are placed in the directory specified by the etc_directory variable on all hosts, typically properties files for event listener, security or similar configuration
  • files/extra/lib - additional files that are placed in the lib directory all hosts, typically binary files or configuration files that are added to the servers classpath such as custom user defined function implementations
  • files/extra/plugin - additional files that are placed in the plugin directory all hosts, typically complete directories with binaries used for a custom developed plugin such as a security extension or another connector

Any changes, including deletion of catalog files is synchronized to the hosts.

Other file deletions are not synchronized but can be performed with the Ansible file module. For example, if you made a file files/extra/etc/foo.sh, the file is copied into /etc/starburst/foo.sh on the hosts. You can use the following command to delete it:

ansible all -m file -a "path=/etc/starburst/foo.sh state=absent"

The supported files and configuration methods are identical to the general configuration files and properties from SEP for the identical version you deploy with Starburst Admin.

After you created or updated all the configuration as desired, you can run the playbook with the following command:

ansible-playbook playbooks/push-configs.yml

Keep in mind that most of the configuration files need to be distributed to all nodes in the cluster and that they need to be identical for all workers. Typically a restart is required to activate any configuration.

Start cluster #

The start playbook start the coordinator and all worker processes on the hosts. It starts SEP the service command for RPM-based installations or with the launcher script for tarball installation.

You need to install the cluster and push the configuration before starting the cluster the first time.

ansible-playbook playbooks/start.yml

The playbook relies on the hosts to be up and running and available to Ansible. When restarting the hosts on the operating system or hardware level, the RPM-based installation automatically starts the SEP processes. The tarball installation does not start automatically. You can however configure it to perform the start by using the launcher script as daemon script. Refer to the documentation of your used Linux distribution for details.

Stop cluster #

The stop playbook stops the coordinator and all worker processes on the hosts. It does not take into account if SEP actively processes any workload, and simply terminates.

ansible-playbook playbooks/stop.yml

You can use the graceful shutdown as an alternative.

Restart cluster #

The restart playbook stops and then starts the coordinator and all worker processes on the hosts.

ansible-playbook playbooks/restart.yml

It is equivalent to running the stop and the start playbook sequentially.

Check service status #

The check-status playbook checks the status of the the coordinator and all worker processes and displays the results.

ansible-playbook playbooks/check-status.yml

The playbook uses the init.d script for the RPM-based installation or the launcher script for a tarball installation to get the status of each service. If the service is running, you see a log for each address in your inventory file stating Running as <pid>:

TASK [Print status] ********
ok: [172.28.0.2] => {
    "msg": "Running as 1965"
}
ok: [172.28.0.3] => {
    "msg": "Running as 1901"
}
ok: [172.28.0.4] => {
    "msg": "Running as 1976"
}

If a service is not active, you see Not running:

TASK [Print status] ****
ok: [172.28.0.2] => {
    "msg": "Not running"
}
ok: [172.28.0.3] => {
    "msg": "Not running"
}
ok: [172.28.0.4] => {
    "msg": "Not running"
}

Graceful worker shutdown #

The graceful-shutdown playbook stops the workers processes after all tasks are completed.

ansible-playbook playbooks/graceful-shutdown.yml

The graceful shutdown takes longer than using the stop playbook. It causes the workers to complete any assigned work. If all workers are shut down, no further query processing is performed by the cluster. The coordinator remains running at all times.

Rolling worker restart #

The rolling-restart-workers playbook stops and starts all worker processes sequentially one after the other using a graceful shutdown and a new start.

ansible-playbook playbooks/rolling-restart-workers.yml

You can configured the following variables in files/vars.yml to affect the graceful shutdown:

  • graceful_shutdown_user - user name to pass to the X-Presto-User or X-Trino-User header when issuing the graceful shutdown request via the HTTP API
  • graceful_shutdown_retries - number of times to check for successful shutdown before failing
  • graceful_shutdown_delay- inactive duration between shutdown status checks
  • rolling_restart_concurrency - number of workers to restart at the same time

By default, the playbook waits up to 10 minutes for any individual worker to stop and start. Each operation has a 10 minute timeout. If this timeout is reached, then the playbook execution fails. If you have a longer shutdown grace period configured, you may want to extend this timeout.

Keep in mind that this playbook does not change any configuration of the workers.

Collect logs #

The collect-logs playbook downloads the log files from all hosts to the control machine.

ansible-playbook playbooks/collect-logs.yml

The server, HTTP request, and launcher logs files from each host are copied into logs/coordinator-<hostname> for the coordinator or logs/worker-<hostname> for the workers in the current directory.

Uninstall cluster #

The uninstall playbook removes all modifications made on the hosts by other playbooks. Stop the cluster before running the playbook.

ansible-playbook playbooks/uninstall.yml

The playbook deletes all data, configuration and log files. It also removes the deployed binary packages and the created user accounts and groups.