Manage catalogs in Icehouse LakeOps#

After you install Icehouse LakeOps, register the catalogs that you want it to maintain.

Register a catalog#

To register a catalog for managed maintenance, select Managed Icehouse > LakeOps from the Control Plane navigation menu.

On the LakeOps pane, click Register a catalog to open the Register catalogs dialog. If you already registered a catalog, click Register catalogs instead.

In the Catalog selection:

  • Select a single catalog from the list of eligible catalogs or select all catalogs by checking the Select all catalogs box.

  • Use the Search catalogs field to find catalogs by name.

In the Configuration options section:

  • In the Optimization tasks section, under Snapshot expiration, enter a threshold between 1 and 90 days for retaining snapshots in the Days field.

  • For additional configuration options, hover over a selected catalog in the Catalog selection section, and click the edit icon. Then expand the Additional options section.

    • Schema/table exclusions lets you exclude any schemas or tables from managed maintenance. All newly created schemas and tables are automatically maintained.

      • Enable to select schemas.

      • Expand a schema to expose its tables.

      • The Select all filter selects all schemas and their tables. Deselect all unchecks all boxes and allows for individual schema and table selection.

      The top of this section shows how many schemas and tables are included or excluded.

    • Blackout period lets you configure a recurring schedule to pause optimization tasks.

      • Enable to set a recurring schedule.

      • Select a Time zone and Duration from their respective drop-down menus. Duration timeframes are available in one-hour intervals.

      • Choose the Select frequency or Enter cron expression recurring interval format.

        For Select frequency, choose from hourly, daily, weekly, monthly, or annually, and then select a time. For Enter cron expression, enter the desired schedule in the form of a UNIX cron expression.

      Tasks still running when the period starts are abandoned without committing their work.

Click Save to register catalogs.

View registered resources#

The LakeOps pane includes a Register catalogs button and the Managed catalogs tab.

The Search field lets you filter lists based on your search criteria. Filters display the number of managed resources in each category. Use the filters to view All resources or resources by category, including Registered and Unregistered.

The Managed catalogs tab lists all catalogs registered for managed maintenance. From here, you can navigate through the levels:

  • Top: shows a list of registered catalogs.

  • Catalog: shows the registered catalog’s schemas.

  • Schema: shows the registered schema’s tables.

  • Table: shows optimization details about the selected table.

Refer to the navigation breadcrumbs at the top of each pane to see which level you are on.

Top-level details#

The top level displays a maintenance summary for your registered catalogs.

The header section provides a Register catalogs button and the following details:

  • Registered catalogs: the total number of catalogs in the Control Plane and what percentage of those catalogs are registered.

  • Managed tables: the number of tables in the registered catalogs.

  • Storage reclaimed: the amount of storage recovered in the last 30 days as a result of maintenance operations like orphan file removal.

  • Files compacted: the number of files that were compacted during the maintenance run in the past 30 days.

This level also shows a list of registered catalogs with details organized in the following columns:

  • Catalog: the name of the registered catalog. Click the name to see catalog-level details.

  • Optimized schemas: the number of schemas included in the maintenance run.

  • Optimized tables: the number of tables included in the maintenance run.

  • Metastore: the metastore used by the registered catalog.

  • Use the options menu to manage registered catalogs.

Catalog-level details#

The catalog level displays a maintenance summary for your registered schemas.

The header section shows the following details:

  • Managed schemas: the number of schemas registered for managed maintenance.

  • Managed tables: the number of tables registered for managed maintenance.

  • Files compacted: the number of files that were compacted during maintenance runs in the past 30 days.

  • Storage reclaimed: the amount of storage recovered in the last 30 days as a result of maintenance operations like orphan file removal.

  • Maintenance success: the percentage of successful and failed maintenance runs.

This level also shows a list of registered schemas with details organized in the following columns:

  • Schema: the name of the schema. Click the name to see schema-level details.

  • Size: the amount of data being optimized.

  • Status: the status of the maintenance run: Initializing, Success, or Failed.

  • Last maintenance: the approximate time since the last maintenance run.

  • Use the options menu to manage registered schemas.

Schema-level details#

The schema level displays a maintenance summary for your registered tables.

The header section displays the following details:

  • Managed tables: the number of tables registered for managed maintenance.

  • Files compacted: the number of files that were compacted during maintenance runs in the past 30 days.

  • Storage reclaimed: the amount of storage recovered in the last 30 days as a result of maintenance operations like orphan file removal.

  • Maintenance success: the percentage of successful runs and failed runs.

This level also shows a list of registered tables with details organized in the following columns:

  • Table: the name of the table. Click the table name to see table-level details.

  • Size: the size of the data being processed.

  • Status: the status of the maintenance run: Initializing, Success, or Failed.

  • Last maintenance: the approximate time since the last maintenance run.

  • Use the options menu to manage registered tables.

Table-level details#

The table level displays a maintenance summary and key metrics for the selected table.

Table stats and data volume details include the Table size and Record count according to the timestamp, and the number of Files compacted and the amount of Storage reclaimed in the last 30 days.

Table history and storage details include the Current number of data files, the Current number of delete files, the Number of snapshots, the Snapshot retention threshold, and the location of the table in Destination.

The Optimization pane displays the following maintenance task details:

  • Compaction: the timestamp indicates when the last compaction task was executed and how many files were removed or added during the maintenance run.

  • Snapshot expiration: the timestamp indicates when the last snapshot expiration task was executed and how many snapshots expired.

  • Orphan file removal: the timestamp indicates when the last orphan file removal task was executed.

Table metrics are also shown in the following charts:

  • Data files compacted: shows how many data files were compacted daily over the past seven days.

  • Delete files compacted: shows how many delete files were compacted daily over the past seven days.

  • Snapshot history: shows how many snapshots were created and how many snapshots expired per day over the past seven days.

  • Table size: shows the size of the table.

  • File composition: shows the total number of data files and delete files.

Manage registered resources#

Unregistering a catalog, schema, or table from managed maintenance automatically stops maintenance operations for the selected resource. Previously completed maintenance operations are not reverted.

Register and unregister resources#

To remove a catalog, schema, or table from managed maintenance:

  1. Go to the Managed catalogs tab, and locate the resource in the catalog, schema, or table list.

  2. Click the options menu at the end of the row, then select Unregister {managed resource}.

  3. In the dialog, click Yes, unregister.

To re-register a schema or table:

  1. Go to the Managed catalogs tab, and locate the resource in the schema or table list.

  2. Click the options menu at the end of the row, then select Register {managed resource}.

  3. In the dialog, click Yes, register.

To re-register a catalog, follow the instructions for registering a catalog.

Edit registered catalogs#

To edit configurations for a registered catalog:

  1. Click the options menu of the desired catalog.

  2. Click Edit catalog to open the edit dialog.

Troubleshooting#

Maintenance errors don’t appear in the Control Plane UI. To troubleshoot maintenance, read the log of the LakeOps coordinator:

kubectl logs deploy/starburst-icehouse-coordinator