Usage metrics#

Usage metrics collection enables Starburst Enterprise platform (SEP) to log information about all nodes in the cluster. This feature is mostly used for Mission Control and other management solutions. Users can use the resulting log information for their own monitoring and operational purposes.


Usage metrics collection is configured by a number of properties, that you can add in etc/ to configure details.

Usage metrics configuration properties#

Property name


Default value


Base path of the usage log files



Maximum size of a single usage log file

100 MB, minimum 1MB


Maximum number of usage log files


Path to the directory on the node in which log files are stored, for example, /var/log/presto/cloudwatch. Amazon CloudWatch is automatically enabled by the SEP CFT and the usage metrics data is available in usage-metrics-* log streams.


The initial delay to before tracking usage, allowing the cluster to start up before metrics gathering starts.

1 min


Length of the interval between usage metric log entry creation.

1 min


Number of threads used to gather and write all usage information.

100, minimum 10


Expose the REST API end point for usage metrics, aggregated for all nodes since cluster start, using the end point is a convenient alternative to the usage metrics parser command line tool, the coordinator exposes the result at /v1/cluster/usage


Logging details and analysis#

The logged information includes details about number of nodes, number of cores, start time, cpu time, and other data about the cluster and nodes in the cluster. The log uses JSON format for each entry.

The JSON information can be inspected manually, ingested in any logging analysis or management tool, and can also be parsed with the metrics parser tool available from Starburst Support. The parser is a command line application that can be run directly, or with the java command.

mv starburst-usage-metrics-parser-*-executable.jar usage-metrics-parser
chmod a+x usage-metrics-parser
# or
java -jar usage-metrics-parser

Save the logs to local empty folder and use the parser with appropriate arguments:

usage-metrics-parser --from <from> --to <to> <path>

The parameters <from>/<to> define date-time range and <path> points to folder with the log files. The resulting output shows the analysis results:

cluster restarts: 1, cpu time: 265.55s, available cpu time: 93130.57s, cpu utilization: 0.29%, min cores: 16, max cores: 16

Persisting Metrics#

Understanding the usage of your cluster requires persisting the usage metrics. By default the metrics are not persisted.

The simplest mechanism to persist metrics is to configure the path of the log file to a location that is separate from the filesystem of the coordinator. Typically this is a network location that is mounted.

The CFT setup automatically configures persisting the metrics in Amazon CloudWatch. Users of Amazon EKS can also use CloudWatch as well.

Logs exported from CloudWatch are decorated and have to be pre-processed to limit information to the raw format displayed in the preceding section to process it with the metrics parser.

Log management solutions integrated with other cloud platforms, or available separately, allow persisting the metrics by capturing the log file or regularly inspecting the REST endpoint.