External security in AWS #

You must configure sufficient permissions to allow Starburst Galaxy to access your data sources in AWS. The details depend on the connected data source.

The following sections detail the different methods, and explain the necessary steps to configure security and permissions in AWS. You can then use the provided details in your catalog configuration in Starburst Galaxy.

Cross account IAM role #

You can use an AWS cross account IAM role to configure access to data in S3 and the metadata in Amazon Glue in your S3 catalogs. Create one role for all S3 catalogs and Amazon Glue instances, or split it up into multiple roles, as desired.

Use the following steps to create the role or roles with suitable permissions in your AWS console:

  1. Access the IAM service configuration from the Security, Identity, & Compliance section of the list of all services.

  2. Select Roles from the Identity and Access management menu.

  3. Click on the Create role button.

  4. Select AWS account in the Select trusted entity section.

      AWS select trusted entity section

  5. Select Another AWS account in the An AWS account section.

  6. Copy the Starburst AWS account ID value from the Starburst Galaxy user interface, and paste it in the Account ID field.

  7. Check Require external ID in options.

  8. Copy the value External ID from the Starburst Galaxy user interface, and paste it in the External ID field.

      AWS account allow entities section

  9. Click on the Next button.

  10. Select one or more access policies to enable the necessary access to S3 and AWS Glue.

  11. Click on the Next button.

  12. Optionally configure any tags.

  13. Click on the Next button.

  14. Add a meaningful Role name, such as starburst-galaxy-access, and provide details in the Description.

  15. Review permissions in the Permissions policy summary section.

      review permissions section

  16. Click on the Create role button to finish.

Once the role is created, you can access the necessary details or update the role as desired:

  • Use the Search filter to locate the role by name.
  • Click on the role name in the list of roles.
  • Copy the value of the ARN from the Summary section to provide the value in Starburst Galaxy. The ARN value looks similar to arn:aws:iam::youraccountid:role/starburst-galaxy-access.

      role summary section

Now you can use the ARN value to configure the authentication for S3 and AWS Glue in cloud settings.

AWS access and secret key #

You can use AWS access and secret keys to configure access to data in S3 and the metadata in Amazon Glue in your S3 catalogs.

Use the following steps to create a user with the necessary access:

  1. Access the IAM service configuration from the Security, Identity, & Compliance section of the list of all services.

  2. Select Users from the Access management menu.

  3. Click on the Add users button.

  4. Provide a user name.

  5. Select Access key - Programmatic access in Select AWS credential type.

      AWS set users window

  6. Click on the Next: Permissions button.

  7. Select Attach existing policies directly in Set permissions.

      AWS set permissions window

  8. Select one or more access policies to enable the necessary access to S3 and AWS Glue.

  9. Click on the Next: Tags button.

  10. Optionally configure any tags.

      AWS users review window

  11. Click on the Next: Review button.

  12. Review and click on the Create user button.

The success page provides the Access key ID directly on screen and allows you to access the Secret access key with the Show link. You need to make sure to copy both values for your use. The secret access key value is only available at this stage.

  AWS add user success message

If you do not have the secret access key for the user, you need to use Create new access key in the Security credentials tab of the user Summary.

Now you can use the access key and secret key to configure the authentication for S3 and AWS Glue in your S3 catalogs.

AWS privileges #

You need S3 access to get to the files in your buckets. If you are using AWS Glue for the metadata, you also need access to AWS Glue. The level of access varies depending on whether require read-only or write access to data.

The simplest approach is to use predefined policies from AWS. Depending on your use case, you can use the following policies:

  • AmazonS3ReadOnlyAccess, if the role or user is aimed for read access to S3 only.
  • AmazonS3FullAccess, if the role or user is aimed for read and write access to S3.
  • AWSGlueConsoleFullAccess, if the role or user is aimed to provide access to AWS Glue.

If you want to set up minimal privileges rather than use some of the more permissive read-only policies provided by AWS, the following shows a custom privilege example for read-only access to S3:

{
  {
    "Sid": "s3",
    "Effect": "Allow",
    "Action": ["s3:GetObject"],
    "Resource": ["arn:aws:s3:::example/demo/*"]
  },
  {
    "Sid": "s3list",
    "Effect": "Allow",
    "Action": ["s3:ListAllMyBuckets"],
    "Resource": ["arn:aws:s3:::*"]
  }
}