Exclude files that do not satisfy the metadata predicate

[chenjian2664]

Description

Exclude files that do not satisfy the metadata predicate while computing the table statistics in Iceberg.
While building the tableScan metadata predicates are ignored, it results scan files more than expectation.

Additional context and related issues

Release notes

(x) This is not user-visible or docs only and no release notes are required.

[findinpath]

trino> create table iceberg.default.ptest1(c1 integer, c2 integer, p integer) with (partitioning = array['p'], format='parquet');
CREATE TABLE
trino> insert into iceberg.default.ptest1 values (4,5, 10), (11,16,20);
INSERT: 2 rows

trino> insert into iceberg.default.ptest1 values (1,2, 10), (2,3,10), (12,14,20);


rino> show stats for (select * from iceberg.default.ptest1 where "$path" = 'hdfs://hadoop-master:9000/user/hive/warehouse/ptest1-5f9f47e5f84b4c66a25017e3952ecb0d/data/p=10/20230703_114010_00002_wmerg-de99aaa0-fa75-441a-ad87-174e6a86e50a.parquet');
 column_name | data_size | distinct_values_count | nulls_fraction | row_count | low_value | high_value 
-------------+-----------+-----------------------+----------------+-----------+-----------+------------
 c1          |      NULL |                   2.0 |            0.0 |      NULL | 1         | 2          
 c2          |      NULL |                   2.0 |            0.0 |      NULL | 2         | 3          
 p           |      NULL |                   1.0 |            0.0 |      NULL | 10        | 10         
 NULL        |      NULL |                  NULL |           NULL |       2.0 | NULL      | NULL       
(4 rows)

Query 20230703_121334_00025_wmerg, FINISHED, 1 node
http://localhost:8080/ui/query.html?20230703_121334_00025_wmerg
Splits: 1 total, 1 done (100.00%)
CPU Time: 0.0s total,     0 rows/s,     0B/s, 0% active
Per Node: 0.0 parallelism,     0 rows/s,     0B/s
Parallelism: 0.0
Peak Memory: 546B
25.82 [0 rows, 0B] [0 rows/s, 0B/s]

trino> show stats for (select * from iceberg.default.ptest1 where "$path" LIKE 'hdfs://hadoop-master:9000/user/hive/warehouse/ptest1-5f9f47e5f84b4c66a25017e3952ecb0d/data/p=10/%');
 column_name | data_size | distinct_values_count | nulls_fraction | row_count | low_value | high_value 
-------------+-----------+-----------------------+----------------+-----------+-----------+------------
 c1          |      NULL |                  NULL |           NULL |      NULL | NULL      | NULL       
 c2          |      NULL |                  NULL |           NULL |      NULL | NULL      | NULL       
 p           |      NULL |                  NULL |           NULL |      NULL | NULL      | NULL       
 NULL        |      NULL |                  NULL |           NULL |      NULL | NULL      | NULL       
(4 rows)

Retrieving stats for a single file works.
Retrieving stats for the entire partition doesn't seem to work.

Iceberg connector provides the stats as expected through the changes of @chenjian2664 .
However io.trino.cost.FilterStatsCalculator.FilterExpressionStatsCalculatingVisitor#process dismisses them.

@raunaqmorarka do you happen to know why this is happening?

[findepi]

Iceberg connector provides the stats as expected through the changes of @chenjian2664 . However io.trino.cost.FilterStatsCalculator.FilterExpressionStatsCalculatingVisitor#process dismisses them.

do you happen to know why this is happening?

Iceberg doesn't produce stats for $path column, co the engine cannot determine stats for a predicate involving the $path. The way to solve this would be to subsume the $path predicate into Iceberg connector, which can sometimes be done.

[github-actions]

This pull request has gone a while without any activity. Tagging the Trino developer relations team: @bitsondatadev @colebow @mosabua

[mosabua]

@findinpath and @chenjian2664 could you connect and figure out next steps here?

[findinpath]

@chenjian2664 pls rebase and I'll make another round of review. Appologies for not following up with the review on the last request.

[github-actions]

This pull request has gone a while without any activity. Tagging the Trino developer relations team: @bitsondatadev @colebow @mosabua

[chenjian2664]

@ebyhr @findinpath Need your help to review on this pr

[github-actions]

This pull request has gone a while without any activity. Tagging the Trino developer relations team: @bitsondatadev @colebow @mosabua

[github-actions]

Closing this pull request, as it has been stale for six weeks. Feel free to re-open at any time.

[mosabua]

I think this is still important to pursue. Hopefully @chenjian2664 can rebase and then @findinpath and others can follow up.

[chenjian2664]

Thanks @mosabua
I believe it is ready to be reviewed @findinpath @ebyhr

[findepi]

from PR description:

Fix #17451

i am not sure this is true. I failed to reproduce #17451 on current master.

Can you please update the PR title and description to reflect current state?

[chenjian2664]

from PR description:

Fix #17451

i am not sure this is true. I failed to reproduce #17451 on current master.

Can you please update the PR title and description to reflect current state?

Yes, just notice the issue already has been fixed. 👍🏻
Updated the description, appreciate it if could take time to look at it. thanks

[findinpath]

pls add a comment why we have different stats when working with avro

[chenjian2664]

I don’t know all the reasons, but what I understand is that Avro is a row-based storage format and does not inherently support storing column-level statistics. Therefore, it's somewhat different from columnar storage formats like Parquet and ORC
Could you help to provide a concise comment

[chenjian2664]

@findinpath @mosabua @ebyhr @findepi Thank you all for the help