Manage streaming data resources #

This page provides information on how to perform DML and DDL operations on live tables, how to query errors tables, and how delete and updte your streaming data resources.

Manage live tables #

The following sections detail important management and operational information about live tables.

DDL and DML #

You cannot directly modify the live table definition or delete or update data with SQL statements. See Update schema to learn how to alter the schema through the UI.

To delete data, set a data retention threshold to purge data from the live table at a specified interval. To filter out unwanted data, create a view on the live table with a predicate to filter out the data you do not want.

If you still need to perform DML and DDL operations as done on any other Iceberg table, you can decommission the live table. Note that once you decommission the table, you can no longer use it with streaming ingestion.

Adjust throughput size #

To adjust the read throughput limit per partition, click themore_vertoptions menu, select a size from the drop-down menu, and click Update throughput.

Live table maintenance #

Learn how Galaxy handles live table maintenance.

Errors table #

Every live table is associated with an errors table that serves as the dead letter table. When Galaxy is unable to parse a message according to the schema or if Galaxy cannot read the message due to its large size, a new row is added for that message.

You can query the errors table the same way you query a live table. The table is hidden and does not show up when running SHOW TABLES. The table name follows the convention: "table_name__raw$errors". You must enclose the name in quotes or the query engine fails to parse the table name.

Decommission a live table #

Decommissioning a live table deletes it from streaming ingestion and stores it as an unmanaged Iceberg table.

To decommission a live table, follow these steps:

• Locate the table of interest.
• Click themore_vertoptions menu, and select deleteDelete live table.
• In the dialog, click Yes, delete.

Update schema #

The update schema option is only availabale for tables containing JSON data.

To make changes to the schema, click themore_vertoptions menu, and select Update schema to go to Create a transform table and schematize data.

When you click Save changes, the option to roll back your ingested table to a previous state and backfill data appears in the Backfill Options dialog.

• In the Backfill Options dialog, choose one of the following:

  • Apply changes without backfill: Previously ingested data remains unchanged and the schema is updated with newly ingested data only.

  • Rewind table to savepoint and backfill: Select a date within the last 30 days and a savepoint from the drop-down menus. The table now appears as it did at the chosen time. Messages that were previously read are reread and rewritten with the updated schema.

  The backfill process carries out across the ingestion processes, the output table, and the errors table, ensuring data is not lost or duplicated.

  • Click Save.

When changing columns, Galaxy automatically performs the Iceberg DDL operations on your live table to alter the schema. Rows present in the table prior to the column changes have NULL values in the newly added columns. Removed columns are no longer accessible to query.

Column changes may take 1-2 minutes to become active.

Manage ingest sources #

You can manage ingest sources in the Data ingest pane. Locate the ingest source of interest, then perform edit or delete actions using the following steps:

Edit an ingest source #

• Click themore_vertoptions menu.
• SelecteditEdit ingest source.
• In the dialog, change source details or select a new authenticationm, then click Update source.

Delete an ingest source #

To delete an ingest source, you must first decommission all live tables associated with it, then follow these steps:

• Click themore_vertoptions menu.
• SelectdeleteDelete ingest source.
• In the dialog, click Yes, delete.

Manage schema registries #

To delete a schema registry, you must first decommission all live tables associated with it, then follow these steps:

• Click themore_vertoptions menu.
• SelectdeleteDelete schema registry.
• In the dialog, click Yes, delete.