Insights introduction#

Starburst Insights provides a visual analysis of important metrics about your cluster for all types of users, from platform administrators to data consumers. From the Insights interface, you can access cluster performance information from a selected date range, such as:

  • CPU usage

  • Memory usage

  • Worker statistics

  • Cluster size

  • Query load

Additionally, users can drill down into query history, including:

  • Most accessed tables, and most active users by CPU time and queries

  • Most and least active queries

  • Completed query history

  • Query details, including query statistics and execution plan

Insights also includes an ad hoc querying tool, Worksheets.

Note

Insights requires a valid Starburst Enterprise license.

Installation#

Insights does not need to be separately installed. It is automatically included and activated on your coordinator, similar to how the Web UI is managed.

If the Web UI is disabled, Insights is also disabled.

Note

Usage metrics and telemetry are surfaced and documented in Insights. However, usage metrics continue to be collected even when Insights or the Web UI are disabled.

Requirements#

Insights requires:

  • A configured and operational event logger setup for full functionality, because Insights uses the same database. Without the event logger database, only limited information is available.

  • Configuration properties on the coordinator, described next.

Configuration#

Insights is configured in the config.properties file on the coordinator only. To avoid startup failures, do not add the properties to worker configuration files.

You must explicitly enable Insights to use the persisted data in the event logger database with the first property in the following table. This provides the information needed for the query and cluster history features of Insights. Additionally, you can configure Insights to persist cluster metrics to the same database with the second property in the table.

To use the usage metrics feature, and to grant unrestricted access to query history, you must assign those rights to authorized users and/or groups, or enable everyone with a wildcard. See examples in Authorization examples.

Cluster and query history-related configuration properties#

Property name

Description

insights.persistence-enabled

Enable the query history functionality. Defaults to false.

insights.metrics-persistence-enabled

Enable the cluster history functionality. Defaults to false.

insights.jdbc.url

JDBC connection string URL for the event logger database.

insights.jdbc.user

Username of the user with read/write access to the event logger database.

insights.jdbc.password

Password of the user for event logger database access.

insights.authorized-users

Regular expression to match user names granted unrestricted access to the query history and to the usage metrics feature.

insights.authorized-groups

Regular expression to match user groups granted unrestricted access to the query history and to the usage metrics feature.

The following optional properties allow you to fine-tune the functionality and behavior of Insights.

Optional Insights configuration properties#

Property name

Description

insights.jdbc.connection-pool.max-size

Maximum number of connections to the event logger database in the connection pool in SEP. Default is 10.

insights.jdbc.connection-pool.min-size

Minimum number of connections to the event logger database in the connection pool in SEP. Default is 1.

insights.metrics-collection-interval

How often query and cluster metrics are sampled for the overview page. Default is 15s.

insights.metrics-persistence-interval

How often query and cluster resource metrics are persisted for the cluster history page. Default is 60s.

insights.max-samples

The number of sample data points to store in memory for the graphs on the overview page. Default is 120.

insights.authorization-application.url

UI URL of an authorization app associated with this cluster, such as Apache Ranger, the Privacera Platform, or Immuta.

insights.authorization-application.label

Name of the associated authorization app to appear in the Resources menu, as discussed in Customize authorization app. Default is Authorization UI.

Instructions for configuring Insights for Kubernetes deployments are available in the Kubernetes documentation.

Authorization examples#

Grant unrestricted access to users alice, bob, and charlie:

insights.authorized-users=alice|bob|charlie

Grant unrestricted access to user alice and any user in the admin group:

insights.authorized-users=alice
insights.authorized-groups=admin

Grant unrestricted access to everyone:

insights.authorized-users=.*

Note

If both insights.authorized-users and insights.authorized-groups are empty, no user has access to the usage metrics feature or unrestricted access to query history.

Login and access#

Access Insights by adding /insights to the URL of the Web UI of your coordinator:

Following are some examples:

URLs to access Web UI and Insights#

Web UI

Insights

https://sep.example.com/ui

https://sep.example.com/ui/insights

http://sep.example.com:8080/ui

http://sep.example.com:8080/ui/insights

https://localhost:8080/ui

https://localhost:8080/ui/insights

Use the same authentication details for Insights as used for the Web UI. With no authentication configured for the cluster, use any username to log in.

Left-hand navigation#

With a successful login, Insights opens with the overview screen and a left-hand navigation bar that includes the following elements:

  • Overview to access the Overview tab.

  • Query history to access the query history tab.

  • Cluster history to access the cluster history tab.

  • Usage metrics to access the usage metrics tab.

  • Worksheets to access the Worksheets tab.

  • Resources, which includes links to sites that open in a new browser tab:

    • Authorization UI opens the URL of the associated authorization app, as described in Customize authorization app.

    • Web UI opens the default Web UI.

    • Documentation opens the Insights documentation.

  • SEP version used by the cluster.

  • Environment name of the cluster.

  • Uptime of the cluster.

  • Sign out to sign out of Insights and the Web UI.

Customize authorization app#

The Resources menu only contains the Authorization UI link if you configure the coordinator’s config.properties with the insights.authorization-application.url property, using a value pointing to a valid URL for an authorization app such as Apache Ranger, the Privacera Platform, or Immuta. This submenu provides a convenience link in Insights to open the UI of the associated authorization app.

You can replace the default Authorization UI text of this submenu with the insights.authorization-application.label property, using a short string that does not overrun the allotted space.

The config.properties must still contain a valid ranger.policy-rest-url property for the cluster, which is the primary way to associate an authorization app with the cluster.

For example:

insights.authorization-application.url=https://ranger.example.com:6182/
insights.authorization-application.label=Lab Ranger