Decouple Trino from Hadoop and Hive codebases

[electrum]

Background

The Trino Hive connector, along with Iceberg, Delta Lake, and Hudi, were built on top of the Hadoop and Hive libraries for features like Hive Metastore client for catalog metadata, clients for HDFS and cloud object stores, reading and writing data files, and so on. Over the past 10 years, we have slowly been replacing or extending many of these core features to fix bugs, improve performance, add monitoring, and improve reliability. At the same time the Hive and Hadoop libraries have slowed in maintenance and feature development which has made the maintenance of the Trino Hive connector more difficult (e.g., fixing CVEs).

Proposal

Replace all uses of Hadoop and Hive libraries with native Trino code. Trino will no longer depend on the Hive library at all, and Hadoop will be an optional dependency that is only used for accessing HDFS. The major areas that will require rewrites, porting, and new code are described below.

Metadata

Metadata has been abstracted for a long time now with direct implementations for Glue, File, and a bridge to the Hive Thrift Metastore. Recently the HMS Thrift client implementation has been replaced with a Trino specific client generated using the newest Thrift code (the old version used by Hive had CVEs).

File Systems

The Hive plugin uses the Hadoop file system API for accessing files, but the usage has been highly controlled since the custom S3 file system was introduced. Recently, we created the TrinoFileSystem API to abstract away from Hadoop APIs and this abstraction is already integrated into the Iceberg and Delta connectors. For this project, we will add Trino file system implementations for S3, GCS, and Azure storage system. HDFS will continue to be supported, of course, but via an implementation of the Trino file system API.

File Formats

There are Trino native implementations of ORC, Parquet, and RCFile. For this project, we will write native implementations for the other formats supported by Trino (CSV, JSON, Regex, TextFile, SequenceFile, and Avro). The old implementations will be available as a fallback during a transition period. Eventually, we will remove support for Hive input formats and serdes. This means that custom formats, which were never officially supported in Trino, will no longer work.

Hive Type

The Hive plugin uses the internal Hive type system in several places for convenience. The single biggest user of Hive types is during interactions with Hive file formats, and the transition to native implementation will eliminate these.

Tasks

• Migrate more code to TrinoFileSystem in Delta Lake #15071
• Migrate more code to TrinoFileSystem in Iceberg #15482
• Remove various Hive dependencies #15630
• Rcfile cleanup #15775
• Use Trino version of Hive Thrift metastore API #15833
• Native implementations of CSV, JSON, Regex, Text File, and Sequence File formats #15918
• Convert Hive AcidUtils.getAcidState to Trino #16004
• Migrate the Delta Lake connector to use TrinoFileSystem #16020
• Replace various Hive usages #16053
• Add native OpenX JSON format #16073
• Replace usages of Hive TypeInfo #16164
• Use native ParquetWriter in Delta CheckpointWriter #16172
• Use Parquet CompressionCodec instead of CompressionCodecName #16174
• Remove usages of HiveChar and HiveVarchar #16175
• Remove unnecessary directory checks in delta lake #16200
• Minor TrinoFileSystem improvements #16226
• Migrate to TrinoFileSystem #16251
• Replace usage of hadoop.fs.Path with String in Delta Lake #16256
• Add REGEX to storage formats and missing table properties #16271
• Use Instant for Trino FileSystem last modified time #16373
• Remove Hadoop usages from Iceberg #16454
• Replace usage hadoop.fs.Path with String in Delta part 2 #16469
• Update to Iceberg 1.2.1 #16557
• Fix handling of Hive ACID tables with hidden directories #16773
• Remove unnecessary enforcer configuration for Iceberg #17111
• Cleanup Hadoop Path usages for Hive #17190
• Migrate Hudi connector to use TrinoFileSystem #17228
• Remove usages of hadoop.fs.FileStatus in Hudi connector #17229
• Add initial Azure TrinoFileSystem #17237
• Clean up usages of Hadoop Path, Configuration in Hudi #17291
• Use Location in file system APIs #17304
• File system location improvements #17312
• Migrate Hive DirectoryLister to TrinoFileSystem  #17323
• Clean up hadoop.mapred usages in Hudi #17326
• Clean up usages of hive and some minor utils in Hudi #17337
• Remove usages of Path from Delta Lake #17378
• Move Hadoop file system code to HDFS module #17379
• Inline hudi code and remove hadoop dependencies #17392
• Enable Hive optimized Parquet writer by default #17393
• Move Rubix to HDFS module #17400
• Add S3 implementation of TrinoFileSystem and file system manager #17498
• Support S3 Select on native readers #17522
• Fix CI job for trino-filesystem-s3 #17739
• Remove unnecessary dependency in Hudi connector #17741
• Fix correctness issue with double slash and add test for S3 and Glue metastore #17804
• Add regression test for HudiMetadata.getTableMetadata when metastore contains views #17910
• Disallow tables with s3://bucket location (without trailing slash after bucket name) #17921
• Invalid Location.appendPath result when location has no authority and single slash after scheme #17931
• Add GCS filesystem #18143
• Remove Configuration argument from HivePageSourceFactory #18146
• Prepare Iceberg MV for lack of classloader isolation #18148
• Test Trino S3 filesystem with MinIO #18211
• Use Hive native writers for creating empty bucket files #18235
• Remove legacy Hive readers and writers #18241
• Native TEXTFILE reader produces incorrect results when varchars contain the field separator escape character #18215
• trino count(*) less hive #18223
• Remove unused Hive SERIALIZATION_DDL property #18261
• Fix HiveType.getRetainedSizeInBytes for caching removal #18296
• Use pre-existing files in AbstractTestParquetPageSkipping #18407
• Hive cleanup #18418
• Remove support for legacy Hive Parquet writer #18420
• Use shaded version of Coral #18444
• Use direct dependency on Parquet #18445
• Hadoop cleanup #18556
• Move Delta Lake GCS handling into HDFS module #19314
• Remove Hadoop usage from SemiTransactionalHiveMetastore #19445
• Remove remaining Hadoop usages from Hive #19566
• Move Hadoop filesystem modules into filesystem manager #19609
• Test Hive and Hudi on native S3 #19634
• Move Hadoop S3 filesystem tests to HDFS module #19635
• Cleanup Hive tests #19639
• Use memory file system in some Hive tests #19664
• Load HDFS file system in separate class loader #19667
• Use native S3 for connector tests #20288
• Enable native file systems #20624

[jordandakota]

Does this decouple the need to use a hive metastore standalone along with Trino? Otherwise to use dynamic catalog creation with connectors like iceberg and delta, you still need also add config to hive and restart it.

[electrum]

@jordandakota No, that's unrelated to this project. Please feel free to file an issue explaining your interest in detail.