Starburst was founded around the Trino open source project. Many contributors and maintainers of the project are part of our teams. Our products, Starburst Enterprise platform (SEP) and Starburst Galaxy, both use Trino as their beating heart and essential engine. And in the same way, Trino is at the heart of our efforts to improve our products. Delta Lake support is the latest example.
{Starburst Enterprise platform (SEP) customers have been enjoying ability to use the Delta Lake connector to query their object storage for nearly two years. In that timeframe, we had over 25 engineers learn the codebase and contribute improvements. At this stage the connector is production-tested and full of great features. Now we are giving it back to the Trino community to work with us to make it even better. It’s a real privilege to work with such dedicated communities.
Everyone can now use the open source projects from Delta Lake and Trino to fully unlock the data in their lakehouse.
Technical details
Delta Lake is an open source storage format that provides ACID guarantees on object storage. With the new connector, we support use cases such as reading data, writing data, and more.
Let’s take a quick look at the capabilities:
CREATE SCHEMA- Create schemas for both new tables, and existing tables that you may have created with another engine.CREATE TABLE (AS)- Create empty tables or pre-load them with the result of an other query. With the power of Trino this query can retrieve data from any other configured catalog, and therefore data source, and to populate the table.- Read operations - Retrieve and analyze your data with raw SQL or your client tool of choice.
INSERT/UPDATE/DELETE- Load data, remove old records, and keep current with changes.VACUUM and OPTIMIZE- Manage your data in Delta Lake to keep it performant even after heavy use.
Next steps
The connector and the documentation are in Trino, so you can all start using it! Get the latest Trino 373 release, and you’ve got everything you need. However, this is just the beginning.
Our next steps include rolling the connector into Starburst Galaxy to empower Delta Lake users even more. From here on out we are committed to keeping it current and improving it, and we have a few things in mind already. And of course, we are also looking forward to see what the community wants and contributes.
If you want to find out more, make sure to join us on Trino Community Broadcast episode 34 on Thursday this week.
Starburst engineering and product teams