use default fpp for writing in Hive connector if not provided in table metadata

[Chaho12]

Description

I think that if table metadata does not have FPP value, default fpp value should be used for bloom filter for writing ORC files.
As of now, it returns error saying that FPP for bloom filter is missing, which is ironic as

1. Hive uses default bloom filter value (0.05%) if value is missing
2. Trino doc saids that default value of fpp is 0.05

Additional context and related issues

Hive has default bloom filter of 0.05 (5%)

• https://github.com/apache/hive/blob/master/storage-api/src/java/org/apache/hive/common/util/BloomFilter.java#L44

Trino doc saids that default value of fpp is 0.05

• https://trino.io/docs/current/connector/hive.html#table-properties

Release notes

( ) This is not user-visible or docs only and no release notes are required.
( ) Release notes are required, please propose a release note for me.
(x) Release notes are required, with the following suggested text:

# Hive Connector
* Use default FPP for bloom filters in ORC when not specified in table metadata.

# Iceberg Connector
* Use default FPP for bloom filters in ORC when not specified in table metadata

[Chaho12]

@Praveen2112 Hey, I was not sure who to tag so I tagged you as a reviewer because I noticed that you made changes to orc bloom filter previously. I hope you don't mind :)

[Chaho12]

Hi ✋ PTAL whenever you have time. Thx :)

[Praveen2112]

It looks good to me but can we add some test coverage here ? But even the table properties are default - doesn't hive specify them when creating a table ?

[Chaho12]

can we add some test coverage here

I will add it soon.

doesn't hive specify them when creating a table ?

Yes, it does not specify when creating a table.

when I do show create tables after creating in hive, it does not add orc.bloom.filter.fpp in tblproperties (only adds 'bucketing_version'='2', 'transient_lastDdlTime'='1693569887') and neither can I find the propertiy in metastore db (at TABLE_PARAMS#PARAM_KEY COLUMN). Despite the document from hive, those properties are not added saved in metastore unless I specify them so.

When you look up new BloomFilter code in Hive and you can find it at 3 places

• https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/HybridHashTableContainer.java#L349-L352
  • for hybrid hash table, use default fpp value hard-coded default fpp value of 0.05
• https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/HybridHashTableContainer.java#L359-L361
  • for hybrid hash table, calculate new fpp value if key count is larger than bloom filter max size
• https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java#L1655
  • fpp is set by hms with 1% (0.01) for ms sqls.

So one can wonder where does this 0.05 default value come from that is mentioned in the docs? Well it comes from map reducing job for ORC writers. The writer checks it the value is set in table properties and if it doesn't have one, default value is used which is set as 0.05 in orc conf.

• This is very similar to Trino as Trino writer checks if it has fpp value in table property, and uses default value if there isn't one (this feature is what I added)

[Chaho12]

Added test coverage when there isn't fpp value

[Chaho12]

PTAL rebased and updated some minor changes :)

[Praveen2112]

Thanks for fixing this issue.

[github-actions]

This pull request has gone a while without any activity. Tagging the Trino developer relations team: @bitsondatadev @colebow @mosabua

[mosabua]

@Praveen2112 is this still good to go and we can merge?

[Praveen2112]

@Chaho12 Thanks for working on this.