Starburst Galaxy’s Icehouse LakeOps service performs automatic table maintenance tasks on registered catalogs.
Icehouse LakeOps runs automatic table maintenance at regular intervals and executes the following optimization tasks when needed:
| Maintenance Task | Description |
|---|---|
| Snapshot expiration | Reduces storage by deleting older data snapshots. This action runs daily and expires snapshots older than 30 days. |
| Compaction | Improves performance by optimizing your data file size. This action runs hourly. |
| Rewrite manifest files | Reorganizes data file references to improve query performance and efficiency. This action runs daily. |
| Orphan file removal | Reduces storage by deleting files that are no longer linked to active data. This action runs daily. |
Icehouse LakeOps is only available to users assigned to roles that have the Manage Icehouse Ops account-level privilege.
Icehouse LakeOps requires exclusive access to registered tables. Any third-party table maintenance services or operations must be fully disabled prior to using LakeOps.
Automatic table maintenance only supports data stored in Parquet format.
To register a catalog for managed maintenance, select Managed Icehouse > LakeOps from the Starburst Galaxy navigation menu.
On the LakeOps pane, click Register a catalog to open the Register catalogs dialog.
In the Catalog selection:
In the Configuration options section:
Choose the Select frequency or Enter cron expression recurring interval format.
For Select frequency, choose from hourly, daily, weekly, monthly, or annually. In the time picker field, input values manually or click the schedule icon to use the time picker wheel. Time intervals are available in five minute intervals, starting at the top of the hour. For manual input, refer to the following frequency and time formatting examples:
| Frequency | Input |
|---|---|
| Hourly | Input a value between 1 minute and 59 minutes. |
| Daily | Input a time in the format hh:mm, then specify AM or PM. |
| Weekly | Input a time in the format hh:mm, specify AM or PM, then select a day of the week. |
| Monthly | Input a time in the format hh:mm, specify AM or PM, then select a date. |
| Annually | Input a month, day, hour, and minutes in the format MM/DD hh:mm. Specify AM or PM. |
Tasks still running when the period starts are abandoned without committing their work.
For Enter cron expression, enter the desired schedule in the form of a UNIX cron expression. For example, a cycle scheduled to run weekly at 9:30 AM on Monday, Wednesday, and Friday:
30 9 * * 1,3,5
The LakeOps pane includes a Register catalogs button and tabs for viewing Managed catalogs and Live tables.
The Search field lets you filter lists based on your search criteria.
Filters display the number of managed resources in each category and the number of errors detected during maintenance runs. Use the filters to view All resources or resources by category, including Registered, Unregistered, and Errors.
The Managed catalogs tab lists all catalogs registered for managed maintenance. From here, you can navigate through the levels:
Refer to the navigation breadcrumbs at the top of each pane to see which level you are on.
The top level displays a maintenance summary for your registered catalogs.
The header section provides a Register catalogs button and the following details:
This level also shows a list of registered catalogs with details organized in the following columns:
The catalog level displays a maintenance summary for your registered schemas.
The header section shows the following details:
This level also shows a list of registered schemas with details organized in the following columns:
Use themore_vertoptions menu to manage registered schemas.
The schema level displays a maintenance summary for your registered tables.
The header section displays the following details:
This level also shows a list of registered tables with details organized in the following columns:
Use themore_vertoptions menu to manage registered tables.
The table level displays a maintenance summary and key metrics for the selected table.
Table stats and data volume details include the Table size and Record count according to the timestamp, the number of Files compacted and the amount of Storage reclaimed in the last 30 days, and any Errors discovered during the last maintenance run.
Table history and storage details include the Current number of data files, the Current number of delete files, the Number of snapshots, the Snapshot retention threshold, and the location of the table in Destination.
The Optimization pane displays the following maintenance task details:
Table metrics are also shown in the following charts:
The Live tables tab displays all live tables registered for managed maintenance with details organized in the following columns:
Live tables cannot be unregistered. Unregistering a catalog or schema, or unmanaged (non-live) table from managed maintenance automatically stops maintenance operations for the selected resource. Previously completed maintenance operations are not reverted.
To remove a catalog, schema, or unmanaged table from managed maintenance:
To re-register a schema or unmanaged table:
To re-register a catalog, follow the instructions for registering a catalog.
To edit configurations for a registered catalog:
Is the information on this page helpful?
Yes
No