Starburst Documentation
Starburst Galaxy
  •  Get started
  •  Working with data
  •  AI workflows
  •  Data engineering
  •  Developer tools
  •  Cluster administration
  •  Security and compliance
  •  Troubleshooting
  • Galaxy status
  •  Reference

    • starburst galaxy > ai workflows > Configure models

    Configure models #

    Starburst Galaxy supports a variety of embedding and language models. To use Starburst AI and the supported models and functions with Galaxy, configure external large language models (LLMs) or embeddings with a supported integration.

    The following topics cover configuration of integrations, models, and functions.

    Prerequisites #

    To use and configure external AI models, consider the following:

    • To configure an external model, the role that you assume to connect to a new model must have the Create AI Models account-level privilege. The role that creates the connection to a model is granted ownership of that model, and is also assigned the Use AI Model privilege at the time of creation.
    • The role that you assume to use any AI model must have the Use AI Model privilege for a specific model.
    • The role that you assume to use the AI functions must have separate privileges at the starburst catalog level to execute each table function that you want to use.
    • For AWS Bedrock connections:
      • The AWS account must have Bedrock service enabled and access to the desired models.
      • Users must have the "Action": ["bedrock:InvokeModel"] AWS IAM permission to use models.
      • To manage models and perform operations as an AWS administrator, you must have the "Action": ["bedrock:*"] permission.
      • Some models require the use of an inference profile. AWS should indicate this requirement in the model card when requesting access. For language models, set the Model ID to the inference profile ID. Starburst recommends using the inference profile ID over the ARN. For embedding models, set the Model ID to the functional model ID, and specify the inference profile ID in the Advanced configuration options section. This is required because each model may have custom encoding requirements.

    Model integration #

    AI functions invoke an external LLM. Access to the LLM must be configured in the AI models pane. Performance, output, and cost of all AI function invocations are dependent on the LLM integration and the model used. Choose a model that aligns with your specific use case and performance requirements.

    Starburst supports AWS Bedrock and OpenAI-compatible APIs.

    AWS Bedrock #

    AWS Bedrock offers access to a suite of foundational models hosted on AWS. Integrating AWS Bedrock models requires configuring AWS credentials and specifying the models that you want to use for different AI functions. To use AWS Bedrock models, configure the necessary model connection details.

    The following models are tested by Starburst:

    Model name Description
    amazon.titan-embed-text-v2:0 The AWS Bedrock Titan Text Embedding v2 embedded model.
    cohere.embed-multilingual-v3 The AWS Bedrock Cohere embed multilingual model.
    anthropic.claude-3-5-haiku-20241022-v1:0 The AWS Bedrock provided Claude3.5 LLM.
    meta.llama3-2-3b-instruct-v1:0 The AWS Bedrock provided Llama 3.2 3B LLM.

    Starburst supports AWS Bedrock models that support the Converse API. This includes most AWS Bedrock models, but may not include custom models and newer OS models.

    For more information about AWS Bedrock’s supported models, read the documentation.

    OpenAI #

    OpenAI offers access to a number of language and embedding models. To use OpenAI models, configure the necessary model connection details.

    The following models are tested and supported by Starburst:

    Model name Description
    text-embedding-3-small The OpenAI text embedding 3 model.
    text-embedding-3-large The OpenAI text embedding 3 model.
    gpt-4o-mini The OpenAI GPT-4o mini. A compact version of the standard GPT-4o LLM.

    For more information about compatible OpenAI models and APIs, read the documentation.

    Compatible APIs #

    Galaxy supports a number of OpenAI models through OpenAI-compatible APIs. This lets you connect to external LLMs for inferencing tasks and embedding models to generate vector embeddings for AI search.

    Azure OpenAPI #

    Azure OpenAI lets you connect to OpenAI models hosted in the Microsoft Azure cloud. You can use Azure OpenAI for both language and embedding tasks by connecting through the OpenAI-compatible API.

    To configure Azure OpenAI with Galaxy, select OpenAI as the integration during model configuration. Then, log in to your Azure OpenAI portal, copy the deployment URL and API key from Azure OpenAI foundry portal, and enter them in the appropriate fields in the connection details section of the Galaxy model configuration.

    Gemini API #

    Gemini provides large language and embedding models accessible through an OpenAI-compatible API. To use Gemini with Galaxy, select OpenAPI as the integration during model configuration, get the API key from your cloud console in Google, and use the following endpoint: https://generativelanguage.googleapis.com/v1beta/openai/

    Model configuration #

    To configure an external model in Galaxy, the role you assume must have the account-level Create AI models privilege. After a role is granted this privilege, the AI models icon becomes visible in the Galaxy navigation menu. Click the AI models icon to access the AI models pane, where you can click Connect to an external model to configure an external model.

    The following sections provide more detail for connecting to an AI model.

    Select a model integration and type #

    Follow these steps to select one of the supported integrations and connect to a model using a compatible API.

    1. From the Model integration drop-down menu, select Amazon Bedrock or OpenAI & OpenAI API Compatible.
    2. From the Model type drop-down menu, select LLM or Embedding.
    3. In the Model ID text field, enter the Model ID from the provider. For example, if using the AWS Bedrock provided Llama 3.2 LLM, specify meta.llama3-2-3b-instruct-v1:0. The Amazon Model ID property value can be a foundation model ID or an inference profile. When the integration is Amazon Bedrock and the model type is Embedding, the Model ID field becomes a drop-down menu.

    For more information about AWS Bedrock’s integrations and models, read the documentation.

    Model alias and description #

    • The Model alias is the identifier used to reference the model in an AI function writing SQL and/or showing the catalog and its nested schemas and tables in client applications. The model alias has the following requirements:
      • Max length of 64 characters.
      • Only letters, numbers, underscores, and hyphens are allowed.
    • If you are configuring a model with Amazon Bedrock, read the prerequisites section.
    • The Description is a short optional paragraph that provides further details about the AI model.

    Connection details #

    Specify the connection credentials for a cross account AWS IAM role, AWS access key, or OpenAI.

    Amazon Bedrock connection details #

    • Authenticate with a cross account AWS IAM role:
      • From the drop-down list, select the Cross account IAM role. See Cross account IAM role to learn how to configure an AWS cross account IAM role for your AWS account.
      • From the drop-down list, select the AWS Bedrock region. For example, the region code for US East (Ohio) is us-east-2.
    • Authenticate with an AWS access key:
      • Specify the AWS access key and AWS secret key. See AWS access and secret key to learn how to configure these keys for your AWS account. Your browser might try to fill the two key fields with your browser login name and password. If so, overwrite these fields with the correct values.
      • From the drop-down list, select the AWS Bedrock region. For example, the region code for US East (Ohio) is us-east-2.

    For more information about AWS regions and region codes, read the AWS documentation.

    OpenAI connection details #

    • Specify the URL for the OpenAI API endpoint. For example, https://api.openai.com/v1.
    • Specify the OpenAI API key value for OpenAI API access. This property is optional when using a compatible OpenAI API that does not require a key.

    Advanced configuration options #

    In addition to the basic connection details for an external AI model, you can specify additional configuration options to fine-tune and control model behavior. The advanced configuration options available change depending on the Model type.

    When the model type is set to LLM, the following options are available:

    • Max Tokens: The maximum number of tokens expected to be returned by the model in its output. This generally includes words, punctuation, spaces, and special formatting.
    • Temperature: Given a value from 0.1 and higher, this controls the perceived randomness or creativity of the model’s output. A lower value makes the output more deterministic.
    • Top P: Controls the diversity of the output. An alternative to temperature sampling. The model selects tokens from the smallest group whose cumulative probability mass exceeds the specified threshold. For example, if the specified value is 0.1, only the combined tokens that make up the top ten percent of the probability mass are considered for the next word generation.
    • Use developer for system role: The developer role replaced the system role in OpenAI o-series models. For more information on the model chain of command, see the OpenAI documentation.

    When the model type is set to Embedding, the following options are available:

    • Dimensions: The number of numerical components in each embedding vector. This determines the size and complexity of the vector space.
    • Inference profile: A profile of preset configuration settings. Read the AWS documentation to learn more about inference profiles in AWS Bedrock.

    General system prompts #

    Use a system prompt to customize and improve the output of your language model by providing instructions or messages that define its behavior, role, tone, or context before any interaction takes place.

    To add a general system prompt, follow these steps:

    1. Click Add general system prompts.
    2. Click Add general system prompt.
    3. In the text field, define a specific action or behavior for the LLM.
    4. If you want to add additional system prompts, click Add general system prompt.
    5. Click Save.

    For more information about system prompts, see the general system prompts section.

    Prompt overrides #

    Prompt overrides let you override general system prompts or use a different model to further refine the output from your LLM.

    To add a prompt override, follow these steps:

    1. Click Add prompt override.
    2. Select the function you would like to add the override to, and specify the parameters for either User or System. You can add overrides to multiple functions at the same time.
    3. Click Save.

    For more information about prompt overrides, see the prompt overrides section.

    Read more about AI functions to see a list of the supported functions and use cases.