From 19b39c85ac4d63e2df526e56efef3dcbbb9b6305 Mon Sep 17 00:00:00 2001 From: Simon Dudley Date: Wed, 31 Jan 2024 17:24:22 +1000 Subject: [PATCH 1/4] Add guide based on https://wiki.hyperledger.org/display/BESU/Limit+Trie+Logs+for+Bonsai Signed-off-by: Simon Dudley --- .../how-to/bonsai-limit-trie-logs.md | 187 ++++++++++++++++++ 1 file changed, 187 insertions(+) create mode 100644 docs/public-networks/how-to/bonsai-limit-trie-logs.md diff --git a/docs/public-networks/how-to/bonsai-limit-trie-logs.md b/docs/public-networks/how-to/bonsai-limit-trie-logs.md new file mode 100644 index 00000000000..c1b771b7ced --- /dev/null +++ b/docs/public-networks/how-to/bonsai-limit-trie-logs.md @@ -0,0 +1,187 @@ +--- +title: Limit Trie Logs for Bonsai +sidebar_position: 12 +description: Enable this early access feature to reduce the size of your database +tags: + - public networks +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + +# Limit Trie Logs for Bonsai +## Step by Step Guide + +:::caution + +We strongly recommend reading the rest of this documentation before running these commands because your node’s configuration may require changes to these commands. + +::: + +*Targeting 24.1.2 release, but currently you must be on the Besu `main` branch for this to work* + +1. Update Besu config to add --Xbonsai-limit-trie-logs-enabled but don’t restart yet +1. Stop Besu +1. (optional) Run: +`sudo /usr/local/bin/besu/bin/besu --data-storage-format=BONSAI --data-path=/var/lib/besu --sync-mode=X_SNAP storage x-trie-log prune` +1. Start Besu (remembering to run sudo systemctl daemon-reload if you use a systemd service file as per CoinCashew and Somer) +1. Look out for `Limit trie logs enabled: retention: 512; prune window: 30000` in your Besu config printout during startup +1. Enjoy more GBs + +## What? +We have a new experimental feature available: `--Xbonsai-limit-trie-logs-enabled` which aims to keep Besu’s database as small as it can be. After a brief grace period, we intend to make this enabled by default for stakers. This is only relevant if you're using `data-storage-format=BONSAI`. + +From our testing, we estimate this will **save users > 3 GB per week** in database growth. Early testing indicates Besu’s **overall database growth with this enabled is ~7 GB per week** (thanks Yorick!) which is on par with geth. + +## Why? +Some users noticed that resyncing Besu can free up disk space, especially for longer running nodes. This is despite `--data-storage-format=BONSAI` having “implicit pruning”. More on this reddit thread: https://www.reddit.com/r/ethstaker/comments/12xnxxi/clearing_up_besubonsai_confusion_on_state_growth/ + +In reality, whilst the Merkle Patricia Trie is implicitly pruned, the BONSAI feature did introduce an extra data structure: the Trie Log (more detail about BONSAI and trie logs here: https://consensys.io/blog/bonsai-tries-guide) + +The Trie Logs are retained in order to cope with chain reorgs. After each block is finalized, Trie Logs older than that are no longer required so it is safe to remove them from both the node's and the network's point of view. + +## How? +If you want to use this feature before it is enabled by default, simply add this option to your Besu command: `--Xbonsai-limit-trie-logs-enabled` +When you restart Besu it will begin pruning, block by block (and a cheeky bit during Besu startup). + +If you want **maximum database size reduction**, read on. + +If you have a long-running node, this will not immediately clear your backlog of trie logs in the same way that resyncing does. In order to do this we’re providing a “run once” **offline** command to immediately **prune all your old trie logs in a few minutes or less**. Note, this requires Besu to be shutdown before running, but downtime will be minimal. You will **not** need to run this command a second time if you keep `--Xbonsai-limit-trie-logs-enabled`. + +## I’m impatient, reduce my database size now! +Okay, okay - we got you! We built a one-off Besu command to remove this extra data (usually in seconds) and avoid having to resync. For minimal downtime, we recommend running this command **before** restarting Besu `--Xbonsai-limit-trie-logs-enabled` (easiest to do it all at the same time). + +If you followed Somer Esat’s (https://someresat.medium.com/guide-to-staking-on-ethereum-ubuntu-teku-f09ecd9ef2ee ) or CoinCashew’s guide (https://www.coincashew.com/coins/overview-eth/guide-or-how-to-setup-a-validator-on-eth2-mainnet/part-i-installation/step-3-installing-execution-client/besu) then you likely have these options set in your `besu.service` or `execution.service` systemd file: + +``` +... +ExecStart=/usr/local/bin/besu/bin/besu \ +... + --sync-mode=X_SNAP \ + --data-path="/var/lib/besu" \ + --data-storage-format=BONSAI \ +... +``` +So in order to prune the trie logs, your command should be something like: + +`sudo /usr/local/bin/besu/bin/besu --data-storage-format=BONSAI --data-path=/var/lib/besu --sync-mode=X_SNAP storage x-trie-log prune` + + + +The logs should look something like this: + +``` +2024-01-31 05:10:43.777+00:00 | main | INFO | KeyPairUtil | Attempting to load public key from /data/besu/key +... +2024-01-31 05:10:47.574+00:00 | main | INFO | TrieLogSubCommand | Calculating trie logs size before... +2024-01-31 05:10:51.100+00:00 | main | INFO | TrieLogSubCommand | Trie logs size before: 8 GiB +2024-01-31 05:12:45.907+00:00 | main | INFO | TrieLogHelper | Saving trie logs to retain in file /data/besu/database/trieLogsToRetain-1 (batch 1)... +2024-01-31 05:12:45.916+00:00 | main | INFO | TrieLogHelper | Obtaining trielogs from db, this may take a few minutes... +2024-01-31 05:12:46.036+00:00 | main | INFO | TrieLogHelper | Clear trie logs... +2024-01-31 05:12:46.100+00:00 | main | INFO | TrieLogHelper | Restoring trie logs retained from batch 1... +2024-01-31 05:12:46.168+00:00 | main | INFO | TrieLogHelper | Key(0): 0xc0a1f26bf828ae6d9af8fda8af3126f12a6d986afea05fdc61b25d0d0ab3b8d4 +... +2024-01-31 05:12:46.263+00:00 | main | INFO | TrieLogHelper | Key(511): 0x5eb700164f4190b8806c7ce3de2f6e5b4f4c05a61bba4058f1b0a5ef63ec73af +2024-01-31 05:12:46.359+00:00 | main | INFO | TrieLogHelper | Deleting files... +2024-01-31 05:12:46.363+00:00 | main | INFO | TrieLogSubCommand | Finished pruning - recalculating size... +2024-01-31 05:12:51.658+00:00 | main | INFO | TrieLogSubCommand | Trie logs size after: 0 B +Prune ran successfully. We estimate you freed up 8 GiB! +``` + +If you use a toml config file, then you can simply do something like: + +`sudo /usr/local/bin/besu/bin/besu --config-file=besu-config.toml storage x-trie-log prune` + +## Subcommand Troubleshooting +The prune command should look something like this for mainnet users: +`sudo /usr/local/bin/besu/bin/besu --data-path=/var/lib/besu --data-storage-format=BONSAI --sync-mode=X_SNAP storage x-trie-log prune` + +and Besu should be shutdown before running it. + +--- + +> java.lang.IllegalArgumentException: Subcommand only works with data-storage-format=BONSAI + +Are you missing --data-storage-format=BONSAI? It must be add before the storage subcommand, i.e. + +`sudo /usr/local/bin/besu/bin/besu --data-storage-format=BONSAI --data-path=/var/lib/besu --sync-mode=X_SNAP storage x-trie-log prune` + +--- + +java.lang.RuntimeException: Column handle not found for segment TRIE_BRANCH_STORAGE + +Did you specify the `data-path`? + +`sudo /usr/local/bin/besu/bin/besu --data-path=/var/lib/besu --data-storage-format=BONSAI --sync-mode=X_SNAP storage x-trie-log prune` + +--- + +> ... +> | INFO | RocksDBKeyValueStorageFactory | No existing database detected at /tmp/besu. Using version 2 +> ... +> java.lang.IllegalArgumentException: Trying to retain more trie logs than chain length (0), skipping pruning + +Did you specify the correct data-path for your node? + +`sudo /usr/local/bin/besu/bin/besu --data-path=/var/lib/besu --data-storage-format=BONSAI --sync-mode=X_SNAP storage x-trie-log prune` + +--- + +> java.lang.IllegalArgumentException: Cannot store generated private key. + +Did you specify the correct data-path for your node? + +`sudo /usr/local/bin/besu/bin/besu --data-path=/var/lib/besu --data-storage-format=BONSAI --sync-mode=X_SNAP storage x-trie-log prune` + +--- + +> ... +> | INFO | KeyPairUtil | Attempting to load public key from /data/besu/key +> java.lang.IllegalArgumentException: Supplied file does not contain valid keyPair pair. + +Check your file permission, you made need to run the command as sudo: +`sudo /usr/local/bin/besu/bin/besu --data-storage-format=BONSAI --data-path=/var/lib/besu storage --sync-mode=X_SNAP x-trie-log prune` + +--- + +> java.lang.RuntimeException: Column handle not found for segment WORLD_STATE + +Are you using data-storage-format=FOREST instead of data-storage-format=BONSAI on an existing bonsai database? + +--- + +> org.hyperledger.besu.plugin.services.exception.StorageException: org.rocksdb.RocksDBException: While lock file: /data/besu/database/LOCK: Resource temporarily unavailable + +Is Besu already running? You need to shut the Besu client down before running this subcommand. + +--- + +> java.lang.IllegalStateException: Unable to change the sync mode when snap sync is incomplete, please restart with snap sync mode + +Have you specified the sync-mode? Default is sync-mode=FAST. Most mainnet users use X_SNAP or X_CHECKPOINT + +`sudo /usr/local/bin/besu/bin/besu --sync-mode=X_SNAP --data-storage-format=BONSAI --data-path=/var/lib/besu storage x-trie-log prune` + +--- + +> java.lang.RuntimeException: No finalized block present, can't safely run trie log prune + +If your node is relatively new or recently resynced, you might not need to run this command. + +--- + +> org.hyperledger.besu.util.InvalidConfigurationException: Supplied genesis block does not match chain data stored in /data/besu. + +Are you running this command for a network other than mainnet? You need to specify the network… + +`sudo /usr/local/bin/besu/bin/besu --network=holesky --sync-mode=X_SNAP --data-storage-format=BONSAI --data-path=/var/lib/besu storage x-trie-log prune` + +--- + + +## The Details +Too much detail for users but if you’re interested: + +- https://github.com/hyperledger/besu/issues/5390 + +- https://github.com/hyperledger/besu/pull/6026 From 2c928a2d9bcffb0db090af234791f154b14a74f0 Mon Sep 17 00:00:00 2001 From: Simon Dudley Date: Fri, 2 Feb 2024 08:58:19 +1000 Subject: [PATCH 2/4] PR suggestions Signed-off-by: Simon Dudley --- .../how-to/bonsai-limit-trie-logs.md | 33 ++++++------------- 1 file changed, 10 insertions(+), 23 deletions(-) diff --git a/docs/public-networks/how-to/bonsai-limit-trie-logs.md b/docs/public-networks/how-to/bonsai-limit-trie-logs.md index c1b771b7ced..bdc365dea39 100644 --- a/docs/public-networks/how-to/bonsai-limit-trie-logs.md +++ b/docs/public-networks/how-to/bonsai-limit-trie-logs.md @@ -1,7 +1,7 @@ --- title: Limit Trie Logs for Bonsai sidebar_position: 12 -description: Enable this early access feature to reduce the size of your database +description: Enable this feature to reduce the size of your database tags: - public networks --- @@ -10,46 +10,33 @@ import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; # Limit Trie Logs for Bonsai + +The early access feature `--Xbonsai-limit-trie-logs-enabled` minimizes the database size of Besu if you are using `data-storage-format=BONSAI`. When enabled, this feature can reduce database growth by more than 3 GB each week on mainnet. + ## Step by Step Guide :::caution -We strongly recommend reading the rest of this documentation before running these commands because your node’s configuration may require changes to these commands. +Limiting trie logs is an early access feature. Before executing the commands in this guide, ensure you review your node configuration. ::: -*Targeting 24.1.2 release, but currently you must be on the Besu `main` branch for this to work* - 1. Update Besu config to add --Xbonsai-limit-trie-logs-enabled but don’t restart yet 1. Stop Besu 1. (optional) Run: `sudo /usr/local/bin/besu/bin/besu --data-storage-format=BONSAI --data-path=/var/lib/besu --sync-mode=X_SNAP storage x-trie-log prune` 1. Start Besu (remembering to run sudo systemctl daemon-reload if you use a systemd service file as per CoinCashew and Somer) 1. Look out for `Limit trie logs enabled: retention: 512; prune window: 30000` in your Besu config printout during startup -1. Enjoy more GBs - -## What? -We have a new experimental feature available: `--Xbonsai-limit-trie-logs-enabled` which aims to keep Besu’s database as small as it can be. After a brief grace period, we intend to make this enabled by default for stakers. This is only relevant if you're using `data-storage-format=BONSAI`. +1. Enjoy more free space -From our testing, we estimate this will **save users > 3 GB per week** in database growth. Early testing indicates Besu’s **overall database growth with this enabled is ~7 GB per week** (thanks Yorick!) which is on par with geth. - -## Why? -Some users noticed that resyncing Besu can free up disk space, especially for longer running nodes. This is despite `--data-storage-format=BONSAI` having “implicit pruning”. More on this reddit thread: https://www.reddit.com/r/ethstaker/comments/12xnxxi/clearing_up_besubonsai_confusion_on_state_growth/ - -In reality, whilst the Merkle Patricia Trie is implicitly pruned, the BONSAI feature did introduce an extra data structure: the Trie Log (more detail about BONSAI and trie logs here: https://consensys.io/blog/bonsai-tries-guide) - -The Trie Logs are retained in order to cope with chain reorgs. After each block is finalized, Trie Logs older than that are no longer required so it is safe to remove them from both the node's and the network's point of view. ## How? If you want to use this feature before it is enabled by default, simply add this option to your Besu command: `--Xbonsai-limit-trie-logs-enabled` -When you restart Besu it will begin pruning, block by block (and a cheeky bit during Besu startup). - -If you want **maximum database size reduction**, read on. +When you restart Besu it will begin pruning, block by block after an initial reduction in the database during each startup. -If you have a long-running node, this will not immediately clear your backlog of trie logs in the same way that resyncing does. In order to do this we’re providing a “run once” **offline** command to immediately **prune all your old trie logs in a few minutes or less**. Note, this requires Besu to be shutdown before running, but downtime will be minimal. You will **not** need to run this command a second time if you keep `--Xbonsai-limit-trie-logs-enabled`. +If you have a long-running node, this will not immediately clear your backlog of trie logs in the same way that resyncing does. Instead of resyncing, in order to do this we’re providing a “run once” **offline** command to immediately **prune all your old trie logs in a few minutes or less**. Note, this requires Besu to be shutdown before running, but downtime will be minimal. You will **not** need to run this command a second time if you keep `--Xbonsai-limit-trie-logs-enabled`. -## I’m impatient, reduce my database size now! -Okay, okay - we got you! We built a one-off Besu command to remove this extra data (usually in seconds) and avoid having to resync. For minimal downtime, we recommend running this command **before** restarting Besu `--Xbonsai-limit-trie-logs-enabled` (easiest to do it all at the same time). +For minimal downtime, we recommend running this command **before** restarting Besu with `--Xbonsai-limit-trie-logs-enabled` (easiest to do it all at the same time). If you followed Somer Esat’s (https://someresat.medium.com/guide-to-staking-on-ethereum-ubuntu-teku-f09ecd9ef2ee ) or CoinCashew’s guide (https://www.coincashew.com/coins/overview-eth/guide-or-how-to-setup-a-validator-on-eth2-mainnet/part-i-installation/step-3-installing-execution-client/besu) then you likely have these options set in your `besu.service` or `execution.service` systemd file: @@ -92,7 +79,7 @@ If you use a toml config file, then you can simply do something like: `sudo /usr/local/bin/besu/bin/besu --config-file=besu-config.toml storage x-trie-log prune` -## Subcommand Troubleshooting +## Troubleshoot The prune command should look something like this for mainnet users: `sudo /usr/local/bin/besu/bin/besu --data-path=/var/lib/besu --data-storage-format=BONSAI --sync-mode=X_SNAP storage x-trie-log prune` From e7db42abed7c86fd3dc41cf9b77ad365db3ed12d Mon Sep 17 00:00:00 2001 From: Simon Dudley Date: Fri, 2 Feb 2024 14:12:58 +1000 Subject: [PATCH 3/4] Tweak caution message Signed-off-by: Simon Dudley --- docs/public-networks/how-to/bonsai-limit-trie-logs.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/public-networks/how-to/bonsai-limit-trie-logs.md b/docs/public-networks/how-to/bonsai-limit-trie-logs.md index bdc365dea39..3df8e3aae76 100644 --- a/docs/public-networks/how-to/bonsai-limit-trie-logs.md +++ b/docs/public-networks/how-to/bonsai-limit-trie-logs.md @@ -17,7 +17,7 @@ The early access feature `--Xbonsai-limit-trie-logs-enabled` minimizes the datab :::caution -Limiting trie logs is an early access feature. Before executing the commands in this guide, ensure you review your node configuration. +The commands in this guide are an example. Before executing the commands on your node, ensure you tailor it to your own node configuration. ::: From 7092e19c38e3b6da6a940a01883b2e1ee538eca6 Mon Sep 17 00:00:00 2001 From: Simon Dudley Date: Fri, 2 Feb 2024 16:02:53 +1000 Subject: [PATCH 4/4] update log messages Signed-off-by: Simon Dudley --- .../how-to/bonsai-limit-trie-logs.md | 32 ++++++++++--------- 1 file changed, 17 insertions(+), 15 deletions(-) diff --git a/docs/public-networks/how-to/bonsai-limit-trie-logs.md b/docs/public-networks/how-to/bonsai-limit-trie-logs.md index 3df8e3aae76..b655ca5e997 100644 --- a/docs/public-networks/how-to/bonsai-limit-trie-logs.md +++ b/docs/public-networks/how-to/bonsai-limit-trie-logs.md @@ -58,21 +58,23 @@ So in order to prune the trie logs, your command should be something like: The logs should look something like this: ``` -2024-01-31 05:10:43.777+00:00 | main | INFO | KeyPairUtil | Attempting to load public key from /data/besu/key -... -2024-01-31 05:10:47.574+00:00 | main | INFO | TrieLogSubCommand | Calculating trie logs size before... -2024-01-31 05:10:51.100+00:00 | main | INFO | TrieLogSubCommand | Trie logs size before: 8 GiB -2024-01-31 05:12:45.907+00:00 | main | INFO | TrieLogHelper | Saving trie logs to retain in file /data/besu/database/trieLogsToRetain-1 (batch 1)... -2024-01-31 05:12:45.916+00:00 | main | INFO | TrieLogHelper | Obtaining trielogs from db, this may take a few minutes... -2024-01-31 05:12:46.036+00:00 | main | INFO | TrieLogHelper | Clear trie logs... -2024-01-31 05:12:46.100+00:00 | main | INFO | TrieLogHelper | Restoring trie logs retained from batch 1... -2024-01-31 05:12:46.168+00:00 | main | INFO | TrieLogHelper | Key(0): 0xc0a1f26bf828ae6d9af8fda8af3126f12a6d986afea05fdc61b25d0d0ab3b8d4 -... -2024-01-31 05:12:46.263+00:00 | main | INFO | TrieLogHelper | Key(511): 0x5eb700164f4190b8806c7ce3de2f6e5b4f4c05a61bba4058f1b0a5ef63ec73af -2024-01-31 05:12:46.359+00:00 | main | INFO | TrieLogHelper | Deleting files... -2024-01-31 05:12:46.363+00:00 | main | INFO | TrieLogSubCommand | Finished pruning - recalculating size... -2024-01-31 05:12:51.658+00:00 | main | INFO | TrieLogSubCommand | Trie logs size after: 0 B -Prune ran successfully. We estimate you freed up 8 GiB! +2024-02-02 05:45:41.162+00:00 | main | INFO | KeyPairUtil | Attempting to load public key from /data/besu/key + ... +2024-02-02 05:45:43.433+00:00 | main | INFO | TrieLogSubCommand | Estimating trie logs size before pruning... +2024-02-02 05:45:43.837+00:00 | main | INFO | TrieLogSubCommand | Estimated trie logs size before pruning: 9 GiB +2024-02-02 05:46:09.863+00:00 | main | INFO | TrieLogHelper | Starting pruning: retain 512 trie logs, processing in 1 batches... +2024-02-02 05:46:09.918+00:00 | main | INFO | TrieLogHelper | Saving trie logs to retain in file /data/besu/database/trieLogsToRetain-1 (batch 1)... +2024-02-02 05:46:09.926+00:00 | main | INFO | TrieLogHelper | Obtaining trielogs from db, this may take a few minutes... +2024-02-02 05:46:10.100+00:00 | main | INFO | TrieLogHelper | Clear trie logs... +2024-02-02 05:46:10.155+00:00 | main | INFO | TrieLogHelper | Restoring trie logs retained from batch 1... +2024-02-02 05:46:10.222+00:00 | main | INFO | TrieLogHelper | Key(0): 0xcd50706da7f6f2db7f9d54f3589122760900d9ab2508c20a4ca40b496d930368 +... +2024-02-02 05:46:10.336+00:00 | main | INFO | TrieLogHelper | Key(511): 0x238f9649b59616430ad7e43b8f3cf65bc97cac4aa54a3eddf3ad6ee666ce733e +2024-02-02 05:46:10.441+00:00 | main | INFO | TrieLogHelper | Deleting files... +2024-02-02 05:46:10.446+00:00 | main | INFO | TrieLogSubCommand | Finished pruning. Re-estimating trie logs size... +2024-02-02 05:46:11.023+00:00 | main | INFO | TrieLogSubCommand | Estimated trie logs size after pruning: 0 B (0 B estimate is normal when using default settings) +2024-02-02 05:46:11.023+00:00 | main | INFO | TrieLogSubCommand | Prune ran successfully. We estimate you freed up 9 GiB! +Prune ran successfully. We estimate you freed up 9 GiB! ``` If you use a toml config file, then you can simply do something like: