Data source management#

Data sources define the connection of Starburst Enterprise platform (SEP) to the various external databases and other systems, that are queried by the SEP user. With Mission Control the creation and management of these data sources is available via a simple user interface.

Create and edit a data source#

The data source management is suitable to create, update, and delete these data sources. Any user can access the data source management by clicking on the Data Sources item in the left hand panel.

The data source management view displays a grid of all existing data sources.

Press the New data source or New button on the screen to add a new data source.

The following data sources can be configured with a rich user interface for the necessary properties.

  • Amazon S3

  • Amazon Redshift

  • Apache Kafka

  • Azure Storage

  • Delta Lake

  • HDFS

  • Microsoft SQL Server

  • MongoDB

  • MySQL

  • Oracle

  • PostgreSQL

  • Teradata

  • Snowflake

Creation and configuration of the data source and the related properties is performed in a tabbed user interface for basic information, connection details, advanced configuration and others. Specific sections differ based on the data source type.

If you can not find your desired data source in the list, try the custom data source configuration.

Data source configuration creates a catalog property file, which configures a specific connector and the relevant properties.

Clicking on a data source name, or using the Edit link in the actions drop down, allows you to edit the data source.

As a next step you can move on to create a cluster, in which the data source is used and available for SEP users to query against.

Data source properties#

Data sources are defined by the following properties:

Type:

Defines the type of external data source to connect to. Type is closely related to the connector used to access the data in the external data source.

Name:

The name of the data source. A well chosen name is important, since it details the data source to the user in SEP. For example, your data source is configured to access a specific database on a specific PostgreSQL server instance. Choosing a name postgresql is therefore not really sufficient. It can be more useful for your users to choose a name of postgresql-customerdb. Potentially, if your users do not really have to know anything about the underlying database, it can also be appropriate to just use a name such as customerdb.

Description:

The description allows you to compose a short paragraph that provides more details about the data source than the name alone.

Connection properties:

After choosing the data source type the most important, required properties to connect to the data sources are displayed. These include properties required by connector as well as properties that enable connection to the specified data source, such as access keys.

Data source types and connectors#

Data source types define the connector SEP uses for the specific data source. The connector in turn, defines the configuration properties to use.

Refer to the following table to determine what connector configuration properties are used for a specific data source type:

Data source type to connector mapping#

Data source type

Connector

Amazon S3

Starburst Hive connector

Amazon Redshift

Redshift Connector

Apache Kafka

Starburst Kafka connector

Delta Lake

Starburst Delta Lake connector

HDFS

Starburst Hive connector

Microsoft SQL Server

Starburst SQL Server connector

MongoDB

MongoDB Connector

MySQL

Starburst MySQL connector

Oracle

Starburst Oracle connector

PostgreSQL

Starburst PostgreSQL connector

Teradata

Starburst Teradata connectors

Snowflake

Starburst Snowflake connector

Custom data sources#

Custom data sources allow you to configure a catalog with a connector that Mission Control does not support. Only use this feature if you have expert knowledge of SEP connectors and catalog configuration.

You can use this interface in the following situations:

  • Configure a catalog using a connector with no related data source support in Mission Control.

  • Manually control the content of the catalog file as an alternative configuration method.

  • Configure a catalog for a custom connector.

Access the configuration with the following steps:

  • Press the + symbol on the bottom right corner to add a new data source.

  • Click on the Don’t see your data source? in the notes section beside the list of data source types

  • Click on the Add one manually link.

  • Confirm that you understand the risk.

  • Configure catalog name, description and connector name.

  • Add the content property definitions for the catalog file in the text input.

  • Press Create to finish.

The supported properties differ for each connector and are identical to the properties used in the data source configuration file, documented with the connectors.