File system cache#

Trino accesses files directly on object storage and remote file system storage. This often involves the transfer of large amounts of data. The files are retrieved from HDFS, or any other supported object storage, by multiple workers and processed on these workers. Repeated queries with different parameters, or even different queries from different users, often access, and therefore transfer, the same objects.

Trino caches these files with a cache manager, configured on every node of the cluster. Two cache managers are available:

  • alluxio caches files on local disks with the help of the open source Alluxio libraries, and is the cache manager to use for caching table data.

  • memory caches files on the Java heap, and is used by the coordinator to cache the metadata files it reads during query planning.

Caching of table data is available for catalogs using the following connectors:

Distributed caching#

File system caching is distributed in Trino as part of the mechanism of any other query processing. Query processing, detailed more in Trino concepts is broken up into different stages, where tasks and splits are processed by different nodes in the cluster. The lowest level splits retrieve data from the data source with the help of the connector of the specific catalog. For file system caching, these splits result in the retrieval of files from object storage.

Different nodes process splits with data from objects storage randomly over time, but with preference for using a fixed set of nodes for a given file. If the preferred nodes are too busy, the split, and hence the caching, takes place on a non-preferred, less busy node. File system caching keeps copies of the retrieved files on a local cache storage, separate for each node. Over time the same files from object storage are cached on any nodes that require the data file for processing a specific task. Each cache on each node is managed separately, following the TTL and size configuration, and cached files are evicted from the cache.

You can limit the number of hosts that are preferred to process these tasks with the node-scheduler.cache-preferred-hosts-count property in the coordinator’s etc/config.properties. Query processing still uses all other nodes as required for the parallel processing of tasks, and therefore potentially caches files on more nodes than the preferred hosts only. A low setting, such as the default 2, can reduce the overall size of the cache because it can reduce how often the same file is cached on multiple nodes. A higher setting, up to the number of nodes in the cluster, distributes the workload across more workers by default, and leads to more resilience against node failures at the expense of effective cache size.

Benefits#

Enabling caching can result in the following significant benefits:

Reduced load on storage

Every retrieved and cached file avoids repeated retrieval from the storage in subsequent queries on the same worker. As a result the storage system does not have to provide the file again and again.

For example, if your query accesses 100MB of files from the storage, the first time the query runs 100MB are downloaded and cached. Any following query uses these files. If your users run another 100 queries accessing the same files, your storage system does not have to provide all data repeatedly. Without caching it has to provide the same files again and again, resulting in up to 10GB of total files to serve.

Increased query performance

Caching can provide significant performance benefits, by avoiding the repeated network transfers and instead accessing copies of the files from a local cache. Performance gains are more significant if the performance of directly accessing the storage is low compared to accessing the local cache.

For example, if you access storage in a different network, different data center, or even different cloud-provider region query performance is slow. Adding caching using fast, local storage has a significant impact and makes your queries much faster.

On the other hand, if your storage is already running at very high performance for I/O and network access, and your local cache storage is at similar speeds, or even slower, performance benefits can be minimal.

Reduced query costs

A result of the reduced load on the storage, mentioned earlier, is significantly reduced network traffic and access to storage. Network traffic and access, often in the form of API access, are often a considerable cost factor, specifically also when hosted in public cloud provider systems.

Configuration#

Configuring the cache takes two steps:

  1. Configure one or more cache managers on every node of the cluster, as described in Cache managers. All nodes must use the same configuration.

  2. Enable caching in each catalog that must use it, as described in Catalog configuration.

Cache managers#

Cache managers are configured per node, not per catalog. Set cache-manager.config-files in the node’s etc/config.properties to a comma-separated list of properties files, one per cache manager:

cache-manager.config-files=etc/cache-manager-alluxio.properties,etc/cache-manager-memory.properties

Every listed file must set cache-manager.name to the name of the cache manager, followed by that manager’s own properties. For example etc/cache-manager-alluxio.properties:

cache-manager.name=alluxio
fs.cache.directories=/tmp/trino-cache
fs.cache.max-sizes=500GB

A cluster can load at most one cache manager for each kind of caching. Loading both alluxio and memory therefore uses alluxio for table data, since it is the only one that can cache more data than fits on the heap, and memory for coordinator metadata caching.

If cache-manager.config-files is not set, Trino loads the memory cache manager with default configuration, so that the coordinator caches metadata files. Once the property is set, only the listed cache managers are loaded, and metadata caching requires listing a memory cache manager explicitly.

Alluxio cache manager properties#

Property

Description

fs.cache.directories

Required, comma-separated list of absolute paths to directories to use for caching. All directories must exist on the coordinator and all workers. Trino must have read and write permissions for files and nested directories. A valid example with only one directory is /tmp/trino-cache.

All catalogs with caching enabled share these directories, and entries are separated by catalog within the cache.

fs.cache.max-sizes

Comma-separated list of maximum data sizes for each caching directory. Order of values must be identical to the directories list. Configuring either fs.cache.max-sizes or fs.cache.max-disk-usage-percentages is required.

fs.cache.max-disk-usage-percentages

Comma-separated list of maximum percentage values of the used disk for each directory. Each value is an integer between 1 and 100. Order of values must be identical to the directories list. If multiple directories use the same disk, ensure that total percentages per drive remains below 100 percent. Configuring either fs.cache.max-sizes or fs.cache.max-disk-usage-percentages is required.

fs.cache.ttl

The maximum duration for objects to remain in the cache before eviction. Defaults to 7d. The minimum value of 0s means that caching is effectively turned off.

fs.cache.page-size

The page data size used for caching data. Each transfer of files uses at least this amount of data. Defaults to 1MB. Values must be between 64kB and 15MB. Larger value potentially result in too much data transfer smaller values are less efficient since they result in more individual downloads.
Memory cache manager properties#

Property

Description

fs.memory-cache.max-size

Maximum total data size of the cache on the heap. Defaults to two percent of the maximum heap size.

fs.memory-cache.max-content-length

Maximum data size of a single cached file. Larger files are read directly from storage and never cached. Defaults to 8MB, and must not exceed 15MB.

fs.memory-cache.ttl

The maximum duration for files to remain in the cache before eviction. Defaults to 1h.

Catalog configuration#

Set fs.cache.enabled to true in the catalog properties file of every catalog that must cache table data:

fs.cache.enabled=true

Every node must have a cache manager that can cache table data, alluxio, when a catalog enables caching. Trino fails to start the catalog otherwise.

Enabling table data caching does not replace metadata caching on the coordinator. If a memory cache manager is loaded as well, the coordinator looks up that cache first and reads through to alluxio on a miss, so planning keeps the lower latency of the heap cache.

Monitoring#

The Alluxio cache manager exposes the Alluxio JMX client metrics under the org.alluxio package, and metrics on external reads and cache reads per catalog under io.trino.blob.cache.alluxio:type=AlluxioCacheStats,name=<catalog>.

The memory cache manager exposes per-catalog metrics under io.trino.blob.cache.memory:type=MemoryBlobCacheStats,name=<catalog>, and metrics for the shared cache under io.trino.blob.cache.memory:name=MemoryBlobCache.

The cache code uses OpenTelemetry tracing.

Migration from catalog-level caching#

In earlier versions the Alluxio cache was configured with fs.cache.* properties in each catalog properties file. When upgrading:

  • Move fs.cache.directories, fs.cache.max-sizes, fs.cache.max-disk-usage-percentages, fs.cache.ttl, and fs.cache.page-size out of the catalog properties files and into an alluxio cache manager properties file on every node. Catalogs keep only fs.cache.enabled.

  • Catalogs no longer need separate cache directories. All catalogs that enable caching share the directories of the node’s cache manager.

  • Existing cached data on disk is not reused, because cache entries are identified differently. The cache repopulates from storage on the first queries after the upgrade, and the previous cache directory contents are evicted by the configured size limit.

  • JMX object names changed. The Alluxio cache statistics moved from io.trino.filesystem.alluxio:type=AlluxioCacheStats,name=<catalog> to io.trino.blob.cache.alluxio:type=AlluxioCacheStats,name=<catalog>, and the Hive connector’s io.trino.plugin.hive.fs:type=TrinoFileSystemCache bean no longer exists. Update dashboards and alerts accordingly.

Recommendations#

The speed of the local cache storage is crucial to the performance of the cache. The most common and cost-efficient approach is to attach high-performance SSD disks or equivalent storage. Fast cache performance can also be achieved with a RAM disk used as an in-memory cache.

In all cases, avoid using the root partition and disk of the node. Instead attach one or more dedicated storage devices for the cache on each node. Storage should be local, dedicated on each node, and not shared.

Your deployment method for Trino decides how to attach storage and create the directories for caching. Typically you need to connect a fast storage system, like an SSD drive, and ensure that it is mounted on the configured path.