From the Starburst Galaxy navigation menu, select Data > Data maintenance.
Data maintenance jobs run tasks that improve performance and reduce storage in Apache Iceberg tables. Supported tasks include data file compaction, statistics collection, and deletion of outdated snapshots and orphaned files.
To perform data maintenance operations on live tables, see data maintenance for Kafka streaming ingestion and data maintenance for file ingestion
The Data maintenance pane has the following levels:
To create a data maintenance job, click Create maintenance task in the top, catalog, or schema details levels.
Provide the following information in the Configure data maintenance dialog:
Maintenance task | Description |
---|---|
Compaction | Improves performance by optimizing your data file size. |
Profiling and statistics | Improves performance by analyzing the table and collecting statistics about your data. |
Snapshot expiration | Reduces storage by deleting data snapshots. |
Delete orphan files | Reduces storage by deleting orphaned data files. This rule includes files that are not part of a table. |
In the Execution details section, select an executing role and a cluster from the Select cluster the respective drop-down menus.
In the Job schedule section:
For Select frequency: Choose an hourly, daily, weekly, monthly, or annual schedule from the drop-down menu. The corresponding values depend on the schedule:
hh:mm
, then specify AM or PM.hh:mm
, specify AM or PM, then
select one or more days of the week.hh:mm
, specify AM or PM, then
select a date.MM/DD
hh:mm
. Specify AM or PM.For Enter cron expression: Enter the desired schedule in the form of a UNIX cron expression. For example, a cycle scheduled to run weekly at 9:30 AM on Monday, Wednesday, and Friday:
30 9 * * 1,3,5
Click Save.
All scheduled data maintenance jobs are listed in the Data maintenance pane beginning at the top details level.
The header section includes a maintencance task legend, which explains the task icons:
The following icons show the status of the data maintenance jobs:
The Last run status drop-down menu at the catalog and schema details levels lets you restrict the list to jobs that are scheduled, running, completed, or failed.
The Maintenance task drop-down menu at the catalog and schema detail levels lets you restrict the list to a single task type.
The Search field at the catalog and schema detail levels lets you restrict the list to the search criteria.
On the catalog and schema detail levels, an subdirectory_arrow_right arrow indicates that maintenance tasks for the entity are inherited from its parent schedule.
The options menu on the catalog and schema details levels provides a play_circle run now action, lets you edit the data maintenance job, and presents an option to create a independent data maintenance job for child entities.
The Maintenance, Schedules, and Errors tabs in the top level details pane provide high-level insight into your data maintenance jobs.
The Maintenance tab displays the list of catalogs and the following information:
Next run: The date and time that the data maintenance job is scheduled to run.
The Schedules tab presents a quick look at all scheduled data maintenance jobs and provides the following information:
Next run: The date and time that the data maintenance job is scheduled to run.
The Errors tab provides the following information for the last 100 failed data maintenance jobs:
Timestamp: Date, time, and timezone in which the error occurred.
To view catalog level details, click the name of a catalog from the top details level.
As with other panes in Galaxy, the top row of this pane provides catalog > schema > table breadcrumbs to show which details level you are on. Click the names in the breadcrumb list to navigate among the levels.
Catalog level details are organized in the following columns:
Next run: The date and time that the data maintenance job is scheduled to run.
To view schema level details, click the name of a schema from the catalog details level. The schema level details list can include individual tables or maintenance tasks set up to run for all tables in a schema.
Schema level details are organized in the following columns:
The Status and Tasks columns remain blank until a task has been executed at least once.
For more information on individual data maintenance jobs, click a table name at the schema details level.
The title of the table details level task pane is the name of the table. The top portion at the pane provides a summary of the selected data maintenance job and a Run now button.
The Task history section is organized in the following columns:
Elapsed time: The duration of data maintenance job.
Manage your data maintenance jobs on the catalog and schema details levels, and on the schedules tab at the top details level.
To make edits to a data maintenance job, follow these steps:
Existing data maintenance jobs exclude tables that have separate maintenance schedules. To include these tables, delete the data maintenance job associated with them. After you delete the job, the previously excluded tables are included in the data maintenance job.
To delete a data maintenance job, follow these steps:
Is the information on this page helpful?
Yes
No