Support Hudi MOR snapshot #14786

willzgw · 2022-10-27T06:57:25Z

Description

Non-technical explanation

Support Hudi MOR Tables
Support MOR snapshot querying for Hudi tables

Release notes

( ) Support Hudi MOR Tables
( ) Support MOR snapshot querying for Hudi tables

# Section
* Support Hudi MOR Tables. ({issue}`5132`)
* Support MOR snapshot querying for Hudi tables. ({issue}`14432`)
* HUDI connector error with loading compaction file. ({issue}`20423`)

codope

@willzgw Thanks for adding the MOR snapshot query support. This would be very helpful for the community. Please edit the PR description accordingly and point to issue#14432.
Also, if you could add integ tests for MOR snapshot query that would be great. You can extend the existing setup. Note that you will have to add an upsert operation (folowwing after insert) in TpchHudiTablesInitializer#load.

plugin/trino-hudi/src/main/java/io/trino/plugin/hudi/HudiSplit.java

plugin/trino-hudi/src/main/java/io/trino/plugin/hudi/HudiPageSourceProvider.java

willzgw · 2022-10-28T11:02:51Z

I am trying to add some unit test cases for this PR.

codope

@willzgw This is looking more and more in shape. It would be great if you could add a few tests as well.

plugin/trino-hudi/pom.xml

plugin/trino-hudi/src/main/java/io/trino/plugin/hudi/HudiTableType.java

plugin/trino-hudi/src/main/java/io/trino/plugin/hudi/partition/HiveHudiPartitionInfo.java

plugin/trino-hudi/src/main/java/io/trino/plugin/hudi/HudiFile.java

plugin/trino-hudi/src/main/java/io/trino/plugin/hudi/HudiModule.java

codope · 2022-11-14T09:17:02Z

plugin/trino-hudi/src/main/java/io/trino/plugin/hudi/HudiRecordCursors.java

+        Path path = new Path(baseFile.getPath());
+
+        HdfsContext context = new HdfsContext(session);
+        Configuration configuration = hdfsEnvironment.getConfiguration(context, path);


HudiSplit already carry configuration. See if we can pass from the split in page source provider.

codope · 2022-11-14T09:19:14Z

plugin/trino-hudi/src/main/java/io/trino/plugin/hudi/HudiRecordCursors.java

+        jobConf.set(READ_COLUMN_NAMES_CONF_STR, join(dataColumns, HiveColumnHandle::getBaseColumnName));
+        schema.stringPropertyNames()
+                .forEach(name -> jobConf.set(name, schema.getProperty(name)));
+        refineCompressionCodecs(jobConf);


Just for my understanding, what's the intention of setting compression codecs here?

plugin/trino-hudi/src/main/java/io/trino/plugin/hudi/HudiSplitSource.java

plugin/trino-hudi/src/main/java/io/trino/plugin/hudi/query/HudiDirectoryLister.java

ebyhr · 2022-12-07T06:36:59Z

Could you rebase on upstream to resolve conflicts and add product tests?

codope · 2023-03-04T16:03:17Z

@willzgw I have a PR for async split processing #16034 that improves the Hudi connector performance. On top of it, I have MoR support in my fork - https://github.com/codope/trino/tree/mor-snapshot
Can you please review the code changes in both? Eventually, if we go with this PR, we would want it to be rebased with #16034

willzgw · 2023-03-04T16:34:20Z

@willzgw I have a PR for async split processing #16034 that improves the Hudi connector performance. On top of it, I have MoR support in my fork - https://github.com/codope/trino/tree/mor-snapshot Can you please review the code changes in both? Eventually, if we go with this PR, we would want it to be rebased with #16034

Hi @codope Let's continue with this PR, I will rebase this branch with #16034.
And, I recently did something similar to your PR on shopee's Trino to optimize the performance of hudi-connector, yours is much better, I just modified HudiReadOptimizedDirectoryLister and the batch-size of the load partition is a fixed value and not configurable.

shan-shanmugam · 2023-08-17T04:06:24Z

Hi @codope / @electrum, When can we anticipate the merge of this request?

codope · 2023-08-18T06:46:36Z

@shan-shanmugam This PR is currently held back due to an ongoing effort to decouple Trino from Hadoop APIs - #15921
In Hudi 0.14.0, we have added a MOR snapshot reader which will help in doing away with some Hadoop/Hive imports in Trino-Hudi code. This PR needs to be reworked with the new reader.

shan-shanmugam · 2023-08-21T02:18:33Z

@codope, Thanks for the update. Any expected timeline for this?

codope · 2023-09-04T04:07:52Z

@shan-shanmugam Hudi 0.14.0 RC vote is ongoing and the community is working on RC2. It should be out in a couple of weeks and then the remaining work on this PR can be picked up.

electrum · 2023-10-09T11:36:20Z

@codope It looks like even the latest version of HoodieMergeOnReadSnapshotReader still depends on Hadoop FileSystem. Do you have plans to address that, or will need a custom version in Trino?

codope · 2023-10-10T02:01:14Z

@codope It looks like even the latest version of HoodieMergeOnReadSnapshotReader still depends on Hadoop FileSystem. Do you have plans to address that, or will need a custom version in Trino?

@electrum The current version of HoodieMergeOnReadSnapshotReader only depends on Configuration of Hadoop. Client code need not create FileSystem. There shouldn't be any FileSystem imports in trino-hudi code. The next step is to create HudiFileSystem and HudiPath abstraction to remove Hadoop from hudi - https://issues.apache.org/jira/browse/HUDI-6497. This is planned for 1.0.0.

zhuzhengjun01 · 2023-12-26T08:22:01Z

This pr is very helpful to me, but when we use 433 version and I tried to cherry-pick this pr found that there are many conflicts after this pr 【#18241 】，especially the RecordCursor is not used after this pr.

mosabua · 2024-01-11T23:51:12Z

👋 @willzgw - this PR has become inactive. We hope you are still interested in working on it. Please let us know, and we can try to get reviewers to help with that.

We're working on closing out old and inactive PRs, so if you're too busy or this has too many merge conflicts to be worth picking back up, we'll be making another pass to close it out in a few weeks.

Also cc @codope @brandyml

willzgw · 2024-01-19T16:08:49Z

👋 @willzgw - this PR has become inactive. We hope you are still interested in working on it. Please let us know, and we can try to get reviewers to help with that.

We're working on closing out old and inactive PRs, so if you're too busy or this has too many merge conflicts to be worth picking back up, we'll be making another pass to close it out in a few weeks.

Also cc @codope @brandyml

Hi @mosabua
I am still working on this. I found some bugs of the HudiTableFileSystemView and this PR.
I will raise an issue to follow these problems and fix them in this PR.

mosabua · 2024-01-19T16:58:27Z

Great .. feel free to reach out to us if you need help. @codope can help on the Hudi side..

…ption

cla-bot · 2024-01-19T17:20:36Z

Thank you for your pull request and welcome to our community. We could not parse the GitHub identity of the following contributors: Guowei Zhang.
This is most likely caused by a git client misconfiguration; please make sure to:

check if your git client is configured with an email to sign commits git config --list | grep email
If not, set it up using git config --global user.email email@example.com
Make sure that the git commit email is configured in your GitHub account settings, see https://github.com/settings/emails

mosabua · 2024-01-19T17:36:09Z

Looks like github still thinks you have a lot of conflicts to resolve @willzgw

willzgw · 2024-01-20T02:06:41Z

Looks like github still thinks you have a lot of conflicts to resolve @willzgw

Yeah, still in progress.

pravin1406 · 2024-05-27T11:19:43Z

@willzgw @codope Any timeline for this support ?

mosabua · 2024-05-27T15:21:54Z

I am not sure if it is worth refactoring this PR. Currently @yihua and @codope are working towards a new Hudi release that removes the Hadoop dependencies. Once that is out they will work on Hudi connector updates that will probably change the codebase quite a bit. So maybe its worth waiting to see if MOR snapshot support will come out of these PRs or just be easier to implement from scratch then.

willzgw · 2024-05-30T02:35:47Z

I am not sure if it is worth refactoring this PR. Currently @yihua and @codope are working towards a new Hudi release that removes the Hadoop dependencies. Once that is out they will work on Hudi connector updates that will probably change the codebase quite a bit. So maybe its worth waiting to see if MOR snapshot support will come out of these PRs or just be easier to implement from scratch then.

@mosabua
Thanks for the heads up. I will evaluate and follow up once HUDI is released.

github-actions · 2024-09-04T17:06:46Z

This pull request has gone a while without any activity. Tagging the Trino developer relations team: @bitsondatadev @colebow @mosabua

github-actions · 2024-09-26T17:12:21Z

Closing this pull request, as it has been stale for six weeks. Feel free to re-open at any time.

ebyhr · 2024-10-22T00:38:28Z

Currently @yihua and @codope are working towards a new Hudi release that removes the Hadoop dependencies

@yihua @codope Did Hudi 0.15.0 remove all Hadoop dependencies? I see https://issues.apache.org/jira/browse/HUDI-6497 contains 0.15.0 as a fix version.

cla-bot bot added the cla-signed label Oct 27, 2022

willzgw added the hudi Hudi connector label Oct 27, 2022

willzgw requested review from codope, findepi and 7c00 October 27, 2022 07:02

codope reviewed Oct 27, 2022

View reviewed changes

plugin/trino-hudi/src/main/java/io/trino/plugin/hudi/HudiSplit.java Outdated Show resolved Hide resolved

plugin/trino-hudi/src/main/java/io/trino/plugin/hudi/HudiPageSourceProvider.java Outdated Show resolved Hide resolved

willzgw force-pushed the hudi-mor branch from 43fdf2e to d20b819 Compare November 9, 2022 15:51

willzgw requested review from codope and fengjian428 and removed request for findepi, fengjian428 and 7c00 November 9, 2022 15:59

codope mentioned this pull request Nov 14, 2022

Add singlenode-hudi product test environment and product tests #14936

Merged

codope reviewed Nov 14, 2022

View reviewed changes

willzgw force-pushed the hudi-mor branch from a11413f to 08703af Compare December 14, 2022 05:03

martint force-pushed the master branch from 40dbb4f to 0d73d10 Compare December 19, 2022 20:15

willzgw force-pushed the hudi-mor branch from 08703af to 2d45569 Compare December 30, 2022 02:53

willzgw force-pushed the hudi-mor branch from 2d45569 to c16a824 Compare March 4, 2023 15:27

This was referenced Mar 5, 2023

[SUPPORT] How to query different queries via Hive 3 & Trino apache/hudi#8038

Closed

[SUPPORT]Whether there are some problems when using hudi-trino-bundle of 0.13.0 version to query hudi table of 0.12.2 version apache/hudi#8096

Closed

kokosing force-pushed the master branch from 3f05134 to 58d6356 Compare March 14, 2023 11:34

codope mentioned this pull request Mar 15, 2023

[SUPPORT] Getting older data using trino hudi connector, for real time MOR table .. whereas getting correct data in hive for same mor_rt table apache/hudi#8185

Closed

fujianhua168 mentioned this pull request May 18, 2023

[SUPPORT] PrestoDB encountered data quality issues while reading the Hudi Mor table. apache/hudi#8754

Open

willzgw force-pushed the hudi-mor branch 2 times, most recently from 28b7b67 to 92ec871 Compare June 22, 2023 09:51

willzgw added 4 commits January 20, 2024 01:08

Fix NoClassDefFoundError: com/github/benmanes/caffeine/cache/Caffeine

1e37889

Fix Hudi file extension set

23cf660

Fix AvroRuntimeException: Not a Specific class

03d46cc

Fix org.apache.avro.AvroRuntimeException: java.lang.IllegalAccessExce…

e14d098

…ption

cla-bot bot removed the cla-signed label Jan 19, 2024

Fix: GenericData cannot be cast to class SpecificRecordBase

488d236

willzgw force-pushed the hudi-mor branch from 397075d to 488d236 Compare January 19, 2024 17:25

cla-bot bot added the cla-signed label Jan 19, 2024

findepi removed the tests:hive label Apr 18, 2024

github-actions bot added the stale label Sep 4, 2024

github-actions bot closed this Sep 26, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support Hudi MOR snapshot #14786

Support Hudi MOR snapshot #14786

willzgw commented Oct 27, 2022 •

edited

Loading

codope left a comment

willzgw commented Oct 28, 2022

codope left a comment

codope Nov 14, 2022

codope Nov 14, 2022

ebyhr commented Dec 7, 2022

codope commented Mar 4, 2023

willzgw commented Mar 4, 2023

shan-shanmugam commented Aug 17, 2023

codope commented Aug 18, 2023 •

edited

Loading

shan-shanmugam commented Aug 21, 2023

codope commented Sep 4, 2023

electrum commented Oct 9, 2023

codope commented Oct 10, 2023

zhuzhengjun01 commented Dec 26, 2023

mosabua commented Jan 11, 2024

willzgw commented Jan 19, 2024

mosabua commented Jan 19, 2024

cla-bot bot commented Jan 19, 2024

mosabua commented Jan 19, 2024

willzgw commented Jan 20, 2024

pravin1406 commented May 27, 2024

mosabua commented May 27, 2024

willzgw commented May 30, 2024

github-actions bot commented Sep 4, 2024

github-actions bot commented Sep 26, 2024

ebyhr commented Oct 22, 2024 •

edited

Loading

Support Hudi MOR snapshot #14786

Support Hudi MOR snapshot #14786

Conversation

willzgw commented Oct 27, 2022 • edited Loading

Description

Non-technical explanation

Release notes

codope left a comment

Choose a reason for hiding this comment

willzgw commented Oct 28, 2022

codope left a comment

Choose a reason for hiding this comment

codope Nov 14, 2022

Choose a reason for hiding this comment

codope Nov 14, 2022

Choose a reason for hiding this comment

ebyhr commented Dec 7, 2022

codope commented Mar 4, 2023

willzgw commented Mar 4, 2023

shan-shanmugam commented Aug 17, 2023

codope commented Aug 18, 2023 • edited Loading

shan-shanmugam commented Aug 21, 2023

codope commented Sep 4, 2023

electrum commented Oct 9, 2023

codope commented Oct 10, 2023

zhuzhengjun01 commented Dec 26, 2023

mosabua commented Jan 11, 2024

willzgw commented Jan 19, 2024

mosabua commented Jan 19, 2024

cla-bot bot commented Jan 19, 2024

mosabua commented Jan 19, 2024

willzgw commented Jan 20, 2024

pravin1406 commented May 27, 2024

mosabua commented May 27, 2024

willzgw commented May 30, 2024

github-actions bot commented Sep 4, 2024

github-actions bot commented Sep 26, 2024

ebyhr commented Oct 22, 2024 • edited Loading

willzgw commented Oct 27, 2022 •

edited

Loading

codope commented Aug 18, 2023 •

edited

Loading

ebyhr commented Oct 22, 2024 •

edited

Loading