feat: introduce `StaticFileSegment::BlockMeta` #13226

joshieDo · 2024-12-09T13:11:14Z

Related to History pre-merge sync#13186 as this data would need to be provided as well.

Adds StaticFileSegment::BlockMeta which would contain:

BlockBodyIndices
BlockOmmers
BlockWithdrawals

Only introduces the segment, alongside some of the future methods.

~~Side-effect: on booting up, it will create an empty block meta static file which won't be used.~~
Should have no side effects

codspeed-hq · 2024-12-27T14:10:13Z

CodSpeed Performance Report

Merging #13226 will not alter performance

_{Comparing joshie/bmeta (0a80ff5) with main (46f4d73)}

Summary

✅ 77 untouched benchmarks

mattsse

I think all of this makes sense, this doesn't integrate this yet, so this doesn't take effect, right?

Side-effect: on booting up, it will create an empty block meta static file which won't be used.

can you elaborate on this

mattsse · 2025-01-08T15:45:10Z

crates/cli/commands/src/db/get.rs

@@ -72,6 +72,7 @@ impl Command {
                    StaticFileSegment::Receipts => {
                        (table_key::<tables::Receipts>(&key)?, <ReceiptMask<ReceiptTy<N>>>::MASK)
                    }
+                    StaticFileSegment::BlockMeta => todo!(),


these should be fine for now I think

done on the follow-up (linked above)

crates/static-file/types/src/lib.rs

crates/static-file/types/src/segment.rs

mattsse · 2025-01-08T15:52:49Z

crates/static-file/types/src/segment.rs

+        if self.segment.is_block_based() {
+            if let Some(range) = &mut self.block_range {
+                if num > range.end - range.start {
+                    self.block_range = None;
+                } else {
+                    range.end = range.end.saturating_sub(num);
+                }
+            };
+        } else if let Some(range) = &mut self.tx_range {
+            if num > range.end - range.start {
+                self.tx_range = None;
+            } else {
+                range.end = range.end.saturating_sub(num);


these are functially unchanged it appears?

yeah, better to have is_block_based instead of enumerating all variants imo

mattsse · 2025-01-08T15:53:52Z

crates/storage/provider/src/providers/static_file/manager.rs

@@ -890,8 +902,11 @@ impl<N: NodePrimitives> StaticFileProvider<N> {
            );
            let mut writer = self.latest_writer(segment)?;
            if segment.is_headers() {
+                // TODO(joshie): is_block_meta


what's missing here?

done on the followup

mattsse · 2025-01-08T15:54:47Z

crates/storage/provider/src/providers/static_file/manager.rs

        Err(ProviderError::UnsupportedProvider)
    }
 }

 impl<N: FullNodePrimitives<BlockHeader: Value>> OmmersProvider for StaticFileProvider<N> {
-    fn ommers(&self, _id: BlockHashOrNumber) -> ProviderResult<Option<Vec<Self::Header>>> {
-        // Required data not present in static_files
+    fn ommers(&self, id: BlockHashOrNumber) -> ProviderResult<Option<Vec<Self::Header>>> {


I see we now have to impl all of these for statics

mattsse · 2025-01-08T15:56:31Z

crates/storage/provider/src/providers/static_file/writer.rs

+        field1: &F1,
+        field2: &F2,


should this perhaps be an iter over additional comlumns?

The number of columns of a static file are statically defined. I think it's okay for now as is and improve on it if necessary

actually, why not

joshieDo · 2025-01-09T13:00:19Z

Side-effect: on booting up, it will create an empty block meta static file which won't be used.

can you elaborate on this

No longer happening, but:

When checking the consistency of a static file segment in a RW env, we fetch a StaticFileWriter of that segment, which leads to creating it (empty) on disk if it doesn't exist yet. Skipped block meta for now.

put on bottom

joshieDo added C-enhancement New feature or request A-db Related to the database A-static-files Related to static files labels Dec 9, 2024

joshieDo mentioned this pull request Dec 10, 2024

Add BlockBodies & BlockBodyIndices to StaticFiles #13264

Open

joshieDo force-pushed the joshie/bmeta branch 7 times, most recently from 67be940 to 342105c Compare December 16, 2024 16:33

joshieDo force-pushed the joshie/bmeta branch 3 times, most recently from 69a497a to 9c241b1 Compare December 20, 2024 09:46

joshieDo force-pushed the joshie/bmeta branch 3 times, most recently from 0aace8c to b7b5e83 Compare December 27, 2024 14:01

joshieDo force-pushed the joshie/bmeta branch 7 times, most recently from 0e13249 to 900b334 Compare January 6, 2025 11:08

joshieDo mentioned this pull request Jan 6, 2025

feat: add StaticFileSegment::BlockMeta to StaticFileProducer and Pruner #13666

Open

joshieDo force-pushed the joshie/bmeta branch 3 times, most recently from c2a840a to 26a65be Compare January 8, 2025 11:38

joshieDo marked this pull request as ready for review January 8, 2025 15:29

joshieDo requested review from rkrasiuk, mattsse, Rjected, onbjerg and gakonst as code owners January 8, 2025 15:29

mattsse requested changes Jan 8, 2025

View reviewed changes

joshieDo force-pushed the joshie/bmeta branch 2 times, most recently from a6347c2 to 2874270 Compare January 9, 2025 12:55

joshieDo force-pushed the joshie/bmeta branch 3 times, most recently from 7ed1564 to 674a603 Compare January 9, 2025 14:38

joshieDo requested a review from mattsse January 9, 2025 16:27

joshieDo added 7 commits January 13, 2025 10:34

add block meta sf segment

15a8230

clippy

23d7f0a

clippy

c254f48

fix test

04c22e8

update-book-cli

6986f25

add append_block_meta and prune_block_meta

5cdbfe1

plural

fe24b3f

joshieDo force-pushed the joshie/bmeta branch from 674a603 to a858853 Compare January 13, 2025 10:42

mattsse approved these changes Jan 14, 2025

View reviewed changes

joshieDo force-pushed the joshie/bmeta branch from a858853 to 2e3afa2 Compare January 14, 2025 23:21

joshieDo enabled auto-merge January 14, 2025 23:22

order bmeta segment

0a80ff5

put on bottom

joshieDo force-pushed the joshie/bmeta branch from 2e3afa2 to 0a80ff5 Compare January 14, 2025 23:24

joshieDo added this pull request to the merge queue Jan 14, 2025

Merged via the queue into main with commit 1267718 Jan 15, 2025
43 checks passed

joshieDo deleted the joshie/bmeta branch January 15, 2025 00:00

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: introduce `StaticFileSegment::BlockMeta` #13226

feat: introduce `StaticFileSegment::BlockMeta` #13226

joshieDo commented Dec 9, 2024 •

edited

Loading

codspeed-hq bot commented Dec 27, 2024 •

edited

Loading

mattsse left a comment

mattsse Jan 8, 2025

joshieDo Jan 8, 2025

mattsse Jan 8, 2025

joshieDo Jan 8, 2025

mattsse Jan 8, 2025

joshieDo Jan 8, 2025

mattsse Jan 8, 2025

mattsse Jan 8, 2025

joshieDo Jan 8, 2025

joshieDo Jan 8, 2025

joshieDo commented Jan 9, 2025 •

edited

Loading

feat: introduce StaticFileSegment::BlockMeta #13226

feat: introduce StaticFileSegment::BlockMeta #13226

Conversation

joshieDo commented Dec 9, 2024 • edited Loading

codspeed-hq bot commented Dec 27, 2024 • edited Loading

Merging #13226 will not alter performance

Summary

mattsse left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

joshieDo commented Jan 9, 2025 • edited Loading

feat: introduce `StaticFileSegment::BlockMeta` #13226

feat: introduce `StaticFileSegment::BlockMeta` #13226

joshieDo commented Dec 9, 2024 •

edited

Loading

codspeed-hq bot commented Dec 27, 2024 •

edited

Loading

joshieDo commented Jan 9, 2025 •

edited

Loading