Insights introduction#

Starburst Insights provides a visual analysis of important metrics about your cluster for all types of users, from platform administrators to data consumers. From the Insights interface, you can access cluster performance information from a selected date range, such as:

  • CPU usage

  • Memory usage

  • Worker statistics

  • Cluster size

  • Query load

Additionally, users can drill down into query history, including:

  • Most accessed tables, and most active users by CPU time and queries

  • Most and least active queries

  • Completed query history

  • Query details, including query statistics and execution plan

Insights also includes an ad hoc querying tool, Worksheets.

Note

Insights requires a valid Starburst Enterprise license.

Installation#

Insights does not need to be separately installed. It is automatically included and activated on your coordinator, similar to how the Web UI is managed.

If the Web UI is disabled, Insights is also disabled.

Note

Usage metrics and telemetry are surfaced and documented in Insights. However, usage metrics continue to be collected even when Insights or the Web UI are disabled.

Requirements#

Insights requires:

  • A configured and operational query logger setup for full functionality, because Insights uses the same database. Without the query logger database, only limited information is available.

  • Configuration properties on the coordinator, described next.

Configuration#

Insights is configured in the config.properties file on the coordinator only. To avoid startup failures, do not add the properties to worker configuration files.

You must explicitly enable Insights to use the persisted data in the query logger database with insights persistence-enabled=true. This provides the information needed for the query and cluster history features of Insights. Additionally, you can configure Insights to persist cluster metrics to the same database with the second property in the table.

To use the usage metrics feature, and to grant unrestricted access to query history, you must assign those rights to authorized users and/or groups, or enable everyone with a wildcard. See examples in Authorization examples.

Cluster and query history-related configuration properties#

Property name

Description

insights.persistence-enabled

Enable the query history functionality. Defaults to false.

insights.metrics-persistence-enabled

Enable the cluster history functionality. Defaults to false.

insights.jdbc.url

JDBC connection string URL for the query logger database.

insights.jdbc.user

Username of the user with read/write access to the query logger database.

insights.jdbc.password

Password of the user for database access.

insights.authorized-users

Regular expression to match user names granted unrestricted access to the query history and to the usage metrics feature.

insights.authorized-groups

Regular expression to match user groups granted unrestricted access to the query history and to the usage metrics feature.

The following optional properties allow you to fine-tune the functionality and behavior of Insights.

Optional Insights configuration properties#

Property name

Description

insights.jdbc.connection-pool.max-size

Maximum number of connections to the query logger database in the connection pool in SEP. Default is 10.

insights.jdbc.connection-pool.min-size

Minimum number of connections to the query logger database in the connection pool in SEP. Default is 1.

insights.metrics-collection-interval

How often query and cluster metrics are sampled for the overview page. Default is 15s.

insights.metrics-persistence-interval

How often query and cluster resource metrics are persisted for the cluster history page. Default is 60s.

insights.max-samples

The number of sample data points to store in memory for the graphs on the overview page. Default is 120.

insights.authorization-application.url

UI URL of an authorization app associated with this cluster, such as Apache Ranger, the Privacera Platform, or Immuta.

insights.authorization-application.label

Name of the associated authorization app to appear in the Resources menu, as discussed in Customize authorization app. Default is Authorization UI.

Instructions for configuring Insights for Kubernetes deployments are available in the Kubernetes documentation.

Example configuration with a PostgreSQL query logger database

insights.persistence-enabled=true
insights.metrics-persistence-enabled=true
insights.jdbc.url=jdbc:postgresql://postgresql.example.com:5432/sepquerylogger
insights.jdbc.user=test_psql
insights.jdbc.password=test12

It is not good practice to use the public schema. Configure the user to use a different schema by default. See the PostgreSQL documentation for details.

Example configuration with a Oracle query logger database

insights.persistence-enabled=true
insights.metrics-persistence-enabled=true
insights.jdbc.url=jdbc:oracle:thin:@oracle.example.com:1521/sepquerylogger
insights.jdbc.user=test_ora
insights.jdbc.password=test12

Example configuration with a MySQL query logger database

insights.persistence-enabled=true
insights.metrics-persistence-enabled=true
insights.jdbc.url=jdbc:mysql://mysql.example.com:3306/sepquerylogger?sessionVariables=sql_mode=ANSI
insights.jdbc.user=test_mysql
insights.jdbc.password=test12

You must specify sql_mode=ANSI in the insights.jdbc.url configuration. The configured user must have sufficient rights to create tables and insert data in the configured schema.

Authorization examples#

Grant unrestricted access to users alice, bob, and charlie:

insights.authorized-users=alice|bob|charlie

Grant unrestricted access to user alice and any user in the admin group:

insights.authorized-users=alice
insights.authorized-groups=admin

The insights.authorized-groups property uses groups configured with File group provider or LDAP group provider in etc/group-provider.properties.

Grant unrestricted access to everyone:

insights.authorized-users=.*

Note

If both insights.authorized-users and insights.authorized-groups are empty, no user has access to the usage metrics feature or unrestricted access to query history.

Login and access#

Access Insights directly with the URL of your coordinator. This redirects you to the context /ui/insights. The legacy Web UI remains accessible at the /ui context, and is used if no valid license is configured.

Following are some examples:

URLs to access Insights and the Web UI#

Insights

Web UI

https://sep.example.com/ or https://sep.example.com/ui/insights

https://sep.example.com/ui/

http://sep.example.com:8080/ or http://sep.example.com:8080/ui/insights

http://sep.example.com:8080/ui/

https://localhost:8080/ or https://localhost:8080/ui/insights

https://localhost:8080/ui/

Use the same authentication details for Insights as used for the Web UI. With no authentication configured for the cluster, use any username to log in.

Left-hand navigation#

With a successful login, Insights opens with the overview screen and a left-hand navigation bar that includes the following elements:

  • Overview to access the Overview tab.

  • Query history to access the query history tab.

  • Cluster history to access the cluster history tab.

  • Usage metrics to access the usage metrics tab.

  • Worksheets to access the Worksheets tab.

  • Resources, which includes links to sites that open in a new browser tab:

    • Authorization UI opens the URL of the associated authorization app, as described in Customize authorization app.

    • Web UI opens the default Web UI.

    • Documentation opens the Insights documentation.

  • SEP version used by the cluster.

  • Environment name of the cluster.

  • Uptime of the cluster.

  • Sign out to sign out of Insights and the Web UI.

Customize authorization app#

The Resources menu only contains the Authorization UI link if you configure the coordinator’s config.properties with the insights.authorization-application.url property, using a value pointing to a valid URL for an authorization app such as Apache Ranger, the Privacera Platform, or Immuta. This submenu provides a convenience link in Insights to open the UI of the associated authorization app.

You can replace the default Authorization UI text of this submenu with the insights.authorization-application.label property, using a short string that does not overrun the allotted space.

The config.properties must still contain a valid ranger.policy-rest-url property for the cluster, which is the primary way to associate an authorization app with the cluster.

For example:

insights.authorization-application.url=https://ranger.example.com:6182/
insights.authorization-application.label=Lab Ranger