Add Limit Trie Logs Early Access Feature Guide #1495

siladu · 2024-01-31T07:25:20Z

...based on https://wiki.hyperledger.org/display/BESU/Limit+Trie+Logs+for+Bonsai

…ie+Logs+for+Bonsai Signed-off-by: Simon Dudley <simon.dudley@consensys.net>

vercel · 2024-01-31T07:25:24Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
besu-docs	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Feb 2, 2024 6:07am

non-fungible-nelson

looks good but this reads like a blog - just clean-up needed. we need to have this long-lived and optimized for SEO/Google.

non-fungible-nelson · 2024-01-31T16:29:49Z

docs/public-networks/how-to/bonsai-limit-trie-logs.md

+---
+title: Limit Trie Logs for Bonsai
+sidebar_position: 12
+description: Enable this early access feature to reduce the size of your database


Enable this feature**

non-fungible-nelson · 2024-01-31T16:30:16Z

docs/public-networks/how-to/bonsai-limit-trie-logs.md

+
+1. Update Besu config to add --Xbonsai-limit-trie-logs-enabled but don’t restart yet
+1. Stop Besu
+1. (optional) Run: 


should we provide relative paths?

In this example, I opted to use the most widely used paths, which fortunately are the same for ethstaker docs and the two most popular step by step guides: Somer Esats and CoinCashew.

I put a warning to not copy and paste, but if ppl still do then it should work out of the box if they followed the popular guides.

non-fungible-nelson · 2024-01-31T16:30:45Z

docs/public-networks/how-to/bonsai-limit-trie-logs.md

+`sudo /usr/local/bin/besu/bin/besu --data-storage-format=BONSAI --data-path=/var/lib/besu --sync-mode=X_SNAP storage x-trie-log prune`
+1. Start Besu (remembering to run sudo systemctl daemon-reload if you use a systemd service file as per CoinCashew and Somer)
+1. Look out for `Limit trie logs enabled: retention: 512; prune window: 30000` in your Besu config printout during startup
+1. Enjoy more GBs


enjoy more free space

non-fungible-nelson · 2024-01-31T16:32:55Z

docs/public-networks/how-to/bonsai-limit-trie-logs.md

+
+:::
+
+*Targeting 24.1.2 release, but currently you must be on the Besu `main` branch for this to work*


I think we should wait on this page until the feature is released.

removed this line

non-fungible-nelson · 2024-01-31T16:33:41Z

docs/public-networks/how-to/bonsai-limit-trie-logs.md

+
+## How?
+If you want to use this feature before it is enabled by default, simply add this option to your Besu command: `--Xbonsai-limit-trie-logs-enabled`
+When you restart Besu it will begin pruning, block by block (and a cheeky bit during Besu startup).


block by block after an initial reduction in the database.

non-fungible-nelson · 2024-01-31T16:33:58Z

docs/public-networks/how-to/bonsai-limit-trie-logs.md

+If you want to use this feature before it is enabled by default, simply add this option to your Besu command: `--Xbonsai-limit-trie-logs-enabled`
+When you restart Besu it will begin pruning, block by block (and a cheeky bit during Besu startup).
+
+If you want **maximum database size reduction**, read on.


non-fungible-nelson · 2024-01-31T16:35:36Z

docs/public-networks/how-to/bonsai-limit-trie-logs.md

+
+If you have a long-running node, this will not immediately clear your backlog of trie logs in the same way that resyncing does. In order to do this we’re providing a “run once” **offline** command to immediately **prune all your old trie logs in a few minutes or less**. Note, this requires Besu to be shutdown before running, but downtime will be minimal. You will **not** need to run this command a second time if you keep `--Xbonsai-limit-trie-logs-enabled`.
+
+## I’m impatient, reduce my database size now!


Remove this header

non-fungible-nelson · 2024-01-31T16:35:44Z

docs/public-networks/how-to/bonsai-limit-trie-logs.md

+If you have a long-running node, this will not immediately clear your backlog of trie logs in the same way that resyncing does. In order to do this we’re providing a “run once” **offline** command to immediately **prune all your old trie logs in a few minutes or less**. Note, this requires Besu to be shutdown before running, but downtime will be minimal. You will **not** need to run this command a second time if you keep `--Xbonsai-limit-trie-logs-enabled`.
+
+## I’m impatient, reduce my database size now!
+Okay, okay - we got you! We built a one-off Besu command to remove this extra data (usually in seconds) and avoid having to resync. For minimal downtime, we recommend running this command **before** restarting Besu `--Xbonsai-limit-trie-logs-enabled` (easiest to do it all at the same time).


Remove okay we got you

non-fungible-nelson · 2024-01-31T16:36:48Z

docs/public-networks/how-to/bonsai-limit-trie-logs.md

+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+
+# Limit Trie Logs for Bonsai


Bonsai... to reduce database size.

change this for SEO

Are you suggesting, "Limit Trie Logs to Reduce Database Size"?

Are you happy to exclude Bonsai from the title? I wanted to make it clear it was irrelevant for FOREST, but if we plan to remove FOREST soon, maybe that doesn't matter.

joaniefromtheblock · 2024-01-31T16:42:04Z

docs/public-networks/how-to/bonsai-limit-trie-logs.md

+
+:::caution
+
+We strongly recommend reading the rest of this documentation before running these commands because your node’s configuration may require changes to these commands.


Suggested change

We strongly recommend reading the rest of this documentation before running these commands because your node’s configuration may require changes to these commands.

Limiting trie logs is an early access feature. Before executing the commands in this guide, ensure you review your node configuration.

joaniefromtheblock · 2024-01-31T16:52:10Z

docs/public-networks/how-to/bonsai-limit-trie-logs.md

+## Why?
+Some users noticed that resyncing Besu can free up disk space, especially for longer running nodes. This is despite `--data-storage-format=BONSAI` having “implicit pruning”. More on this reddit thread: https://www.reddit.com/r/ethstaker/comments/12xnxxi/clearing_up_besubonsai_confusion_on_state_growth/
+
+In reality, whilst the Merkle Patricia Trie is implicitly pruned, the BONSAI feature did introduce an extra data structure: the Trie Log (more detail about BONSAI and trie logs here: https://consensys.io/blog/bonsai-tries-guide)


I am not sure this is needed in this guide

Agree, this is better for the initial announcement blog instead

joaniefromtheblock · 2024-01-31T16:58:31Z

looks good but this reads like a blog - just clean-up needed. we need to have this long-lived and optimized for SEO/Google.

Agree with @non-fungible-nelson. This will have to be rewritten for the tech dev docs. We (docs team) can help with that. Would you prefer if we opened a PR on top of this one for suggestions, or make commits directly to this PR?

joaniefromtheblock · 2024-01-31T17:06:29Z

docs/public-networks/how-to/bonsai-limit-trie-logs.md

+1. Look out for `Limit trie logs enabled: retention: 512; prune window: 30000` in your Besu config printout during startup
+1. Enjoy more GBs
+
+## What?


I would move the explanation for why this feature exists early on in the doc... so maybe move this to the start

joaniefromtheblock · 2024-01-31T17:12:16Z

docs/public-networks/how-to/bonsai-limit-trie-logs.md

+1. Enjoy more GBs
+
+## What?
+We have a new experimental feature available: `--Xbonsai-limit-trie-logs-enabled` which aims to keep Besu’s database as small as it can be. After a brief grace period, we intend to make this enabled by default for stakers. This is only relevant if you're using `data-storage-format=BONSAI`.


For tech docs, we typically don't write with a marketing tone and usually opt for language that is a but more succinct and to the point. I think a rewrite for this would be something like:

Suggested change

We have a new experimental feature available: `--Xbonsai-limit-trie-logs-enabled` which aims to keep Besu’s database as small as it can be. After a brief grace period, we intend to make this enabled by default for stakers. This is only relevant if you're using `data-storage-format=BONSAI`.

The feature `--Xbonsai-limit-trie-logs-enabled` minimizes the database size of Besu and requires the use of `data-storage-format=BONSAI`. When enabled, this feature can reduce database growth by more than 3 GB each week.

joaniefromtheblock · 2024-01-31T17:13:00Z

docs/public-networks/how-to/bonsai-limit-trie-logs.md

+From our testing, we estimate this will **save users > 3 GB per week** in database growth. Early testing indicates Besu’s **overall database growth with this enabled is ~7 GB per week** (thanks Yorick!) which is on par with geth.
+
+## Why?
+Some users noticed that resyncing Besu can free up disk space, especially for longer running nodes. This is despite `--data-storage-format=BONSAI` having “implicit pruning”. More on this reddit thread: https://www.reddit.com/r/ethstaker/comments/12xnxxi/clearing_up_besubonsai_confusion_on_state_growth/


I think this can be removed

joaniefromtheblock · 2024-01-31T17:13:07Z

docs/public-networks/how-to/bonsai-limit-trie-logs.md

+## Why?
+Some users noticed that resyncing Besu can free up disk space, especially for longer running nodes. This is despite `--data-storage-format=BONSAI` having “implicit pruning”. More on this reddit thread: https://www.reddit.com/r/ethstaker/comments/12xnxxi/clearing_up_besubonsai_confusion_on_state_growth/
+
+In reality, whilst the Merkle Patricia Trie is implicitly pruned, the BONSAI feature did introduce an extra data structure: the Trie Log (more detail about BONSAI and trie logs here: https://consensys.io/blog/bonsai-tries-guide)


Can be removed/ rewritten or added Ito an introductory section

joaniefromtheblock · 2024-01-31T17:14:51Z

docs/public-networks/how-to/bonsai-limit-trie-logs.md

+
+`sudo /usr/local/bin/besu/bin/besu --config-file=besu-config.toml storage x-trie-log prune`
+
+## Subcommand Troubleshooting


Suggested change

## Subcommand Troubleshooting

## Troubleshoot

Signed-off-by: Simon Dudley <simon.dudley@consensys.net>

siladu · 2024-02-01T23:00:33Z

@joaniefromtheblock @non-fungible-nelson Thanks for the reviews. I've implemented basically all of your suggestions including removing the Why section. Responded to a couple of comments too.

Have moved "What" to the top as an intro instead of it's own section.

Happy for Step-by-step to be moved again, I just wanted it at the top for the initial announcement because some people don't read very far down in my experience :)

Feel free to continue editing yourselves if you prefer, I am happy to review.

Signed-off-by: Simon Dudley <simon.dudley@consensys.net>

joaniefromtheblock · 2024-02-05T18:37:35Z

@joaniefromtheblock @non-fungible-nelson Thanks for the reviews. I've implemented basically all of your suggestions including removing the Why section. Responded to a couple of comments too.

Have moved "What" to the top as an intro instead of it's own section.

Happy for Step-by-step to be moved again, I just wanted it at the top for the initial announcement because some people don't read very far down in my experience :)

Feel free to continue editing yourselves if you prefer, I am happy to review.

Thanks @siladu . I'm reviewing this PR now. I'll commit directly to the PR so it aligns more with our tech writing guidelines. Thank you so much for this content and all the work you have done ! It's really really appreciated

joaniefromtheblock · 2024-03-12T18:24:00Z

Opened another PR @siladu

#1538

joaniefromtheblock · 2024-03-14T20:28:29Z

Closing in favor of #1538

Add guide based on https://wiki.hyperledger.org/display/BESU/Limit+Tr…

19b39c8

…ie+Logs+for+Bonsai Signed-off-by: Simon Dudley <simon.dudley@consensys.net>

siladu requested a review from a team as a code owner January 31, 2024 07:25

vercel bot deployed to Preview January 31, 2024 07:29 View deployment

non-fungible-nelson suggested changes Jan 31, 2024

View reviewed changes

joaniefromtheblock reviewed Jan 31, 2024

View reviewed changes

alexandratran assigned joaniefromtheblock and alexandratran Jan 31, 2024

PR suggestions

2c928a2

Signed-off-by: Simon Dudley <simon.dudley@consensys.net>

vercel bot deployed to Preview February 1, 2024 23:02 View deployment

Tweak caution message

e7db42a

Signed-off-by: Simon Dudley <simon.dudley@consensys.net>

vercel bot deployed to Preview February 2, 2024 04:17 View deployment

update log messages

7092e19

Signed-off-by: Simon Dudley <simon.dudley@consensys.net>

vercel bot deployed to Preview February 2, 2024 06:07 View deployment

alexandratran removed their assignment Mar 4, 2024

joaniefromtheblock mentioned this pull request Mar 12, 2024

Add Limit Trie Logs Early Access Feature Guide #1538

Merged

joaniefromtheblock closed this Mar 14, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Limit Trie Logs Early Access Feature Guide #1495

Add Limit Trie Logs Early Access Feature Guide #1495

siladu commented Jan 31, 2024

vercel bot commented Jan 31, 2024 •

edited

Loading

non-fungible-nelson left a comment

non-fungible-nelson Jan 31, 2024

non-fungible-nelson Jan 31, 2024

siladu Feb 1, 2024 •

edited

Loading

non-fungible-nelson Jan 31, 2024

non-fungible-nelson Jan 31, 2024

siladu Feb 2, 2024

non-fungible-nelson Jan 31, 2024

non-fungible-nelson Jan 31, 2024

non-fungible-nelson Jan 31, 2024

non-fungible-nelson Jan 31, 2024

non-fungible-nelson Jan 31, 2024

siladu Feb 1, 2024

joaniefromtheblock Jan 31, 2024

joaniefromtheblock Jan 31, 2024

siladu Feb 1, 2024 •

edited

Loading

joaniefromtheblock commented Jan 31, 2024

joaniefromtheblock Jan 31, 2024

joaniefromtheblock Jan 31, 2024

joaniefromtheblock Jan 31, 2024

joaniefromtheblock Jan 31, 2024 •

edited

Loading

joaniefromtheblock Jan 31, 2024

siladu commented Feb 1, 2024 •

edited

Loading

joaniefromtheblock commented Feb 5, 2024

joaniefromtheblock commented Mar 12, 2024

joaniefromtheblock commented Mar 14, 2024


		:::

		Targeting 24.1.2 release, but currently you must be on the Besu `main` branch for this to work


		If you have a long-running node, this will not immediately clear your backlog of trie logs in the same way that resyncing does. In order to do this we’re providing a “run once” offline command to immediately prune all your old trie logs in a few minutes or less. Note, this requires Besu to be shutdown before running, but downtime will be minimal. You will not need to run this command a second time if you keep `--Xbonsai-limit-trie-logs-enabled`.

		## I’m impatient, reduce my database size now!


		:::caution

		We strongly recommend reading the rest of this documentation before running these commands because your node’s configuration may require changes to these commands.

	We strongly recommend reading the rest of this documentation before running these commands because your node’s configuration may require changes to these commands.
	Limiting trie logs is an early access feature. Before executing the commands in this guide, ensure you review your node configuration.

	We have a new experimental feature available: `--Xbonsai-limit-trie-logs-enabled` which aims to keep Besu’s database as small as it can be. After a brief grace period, we intend to make this enabled by default for stakers. This is only relevant if you're using `data-storage-format=BONSAI`.
	The feature `--Xbonsai-limit-trie-logs-enabled` minimizes the database size of Besu and requires the use of `data-storage-format=BONSAI`. When enabled, this feature can reduce database growth by more than 3 GB each week.


		`sudo /usr/local/bin/besu/bin/besu --config-file=besu-config.toml storage x-trie-log prune`

		## Subcommand Troubleshooting

Add Limit Trie Logs Early Access Feature Guide #1495

Add Limit Trie Logs Early Access Feature Guide #1495

Conversation

siladu commented Jan 31, 2024

vercel bot commented Jan 31, 2024 • edited Loading

non-fungible-nelson left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

siladu Feb 1, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

siladu Feb 1, 2024 • edited Loading

Choose a reason for hiding this comment

joaniefromtheblock commented Jan 31, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

joaniefromtheblock Jan 31, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

siladu commented Feb 1, 2024 • edited Loading

joaniefromtheblock commented Feb 5, 2024

joaniefromtheblock commented Mar 12, 2024

joaniefromtheblock commented Mar 14, 2024

vercel bot commented Jan 31, 2024 •

edited

Loading

siladu Feb 1, 2024 •

edited

Loading

siladu Feb 1, 2024 •

edited

Loading

joaniefromtheblock Jan 31, 2024 •

edited

Loading

siladu commented Feb 1, 2024 •

edited

Loading