From 8bce07accc5eac88c0e9454e47095ff4b7d48522 Mon Sep 17 00:00:00 2001
From: Wen <113942165+wen-coding@users.noreply.github.com>
Date: Sun, 11 Aug 2024 18:56:28 -0700
Subject: [PATCH 01/24] Initial draft of Async Vote Execution SIMD.

---
 proposals/0154-async-vote-execution.md | 109 +++++++++++++++++++++++++
 1 file changed, 109 insertions(+)
 create mode 100644 proposals/0154-async-vote-execution.md

diff --git a/proposals/0154-async-vote-execution.md b/proposals/0154-async-vote-execution.md
new file mode 100644
index 000000000..0668f7b85
--- /dev/null
+++ b/proposals/0154-async-vote-execution.md
@@ -0,0 +1,109 @@
+---
+simd: '0154'
+title: Async Vote Execution
+authors:
+  - Wen Xu
+category: Standard
+type: Core
+status: Draft
+created: 2024-08-11T00:00:00.000Z
+feature: null
+supersedes: null
+superseded-by: null
+extends: null
+---
+## Summary
+Separate the execution of vote and non-vote transactions in each block. The
+vote transactions will be verified and executed first, then the non-vote
+transactions will be executed later asynchronously to finalize the block.
+
+## Motivation
+Currently the vote transactions and non-vote transactions are mixed together in
+a block, the vote transactions are only processed in consensus when the whole
+block has been frozen and all transactions in the block have been verified and
+executed. This is a problem because slow running non-vote transactions may affect
+how fast the votes are processed and then affect the ability of consensus to
+pick the correct fork.
+
+With different hardware and running environment, there will always be some
+difference on speed of transaction execution between validators. Generally
+speaking, because vote transactions are so simple, the variation between vote
+execution should be smaller than that between non-vote executions. Therefore,
+if we only execute vote transactions in a block before voting on the block,
+it is more likely validators can reach consensus faster.
+
+Even with async vote execution, forks can still happen because of
+various other situations, like network partitions or mis-configured validators.
+This work just reduces the chances of forks caused by variance in non-vote
+transaction executions.
+
+The non-vote transactions do need to be executed eventually. Even though it's
+hard to make sure everyone executes every block within 400ms, on average majority
+of the cluster should be able to keep up.
+
+## New Terminology
+- `VED`: Vote Execution Domain, Vote transactions and all its dependencies (e.g.
+fee payers for votes).
+- `VED Bankhash`: The hash calculated after executing only vote transactions in
+a block. If there are no votes, use default hash.
+- `UED`: User Execution Domain, currently everything other than votes. We may
+have more domains in the future.
+- `UED Bankhash`: The hash calculated after executing only non-vote transactions
+in a block. If there are no non-vote transactions, use default hash.
+
+## Detailed Design
+
+### Allow leader to skip execution of transactions (Bankless Leader)
+There is already on-going effort to totally skip execution of all transactions
+when leader pack new blocks. See SIMD 82, SIMD 83, and related trackers:
+https://github.com/anza-xyz/agave/issues/2502
+
+Theoretically we could reap some benefit without Bankless Leader, the leader
+pack as normal, while other validators only replay votes first, then later
+execute other transactions and compare with the bankhash of the leader. But in
+such a setup we gain smaller speedup without much benefits, it is a possible
+route during rollouts though.
+
+### Separating vote transactions and dependencies into a different domain
+To make sure vote transactions can be executed independently, we need to
+isolate its dependencies.
+
+#### Remove clock program's dependency on votes
+Introduce new transaction `ClockBump` to remove current clock program's
+dependency on votes.
+
+The transaction `ClockBump` is sent by a leader with at least 0.5% stake
+every 12 slots to correct the clock drift. A small script can be used to
+refund well-behaving leaders the cost of the transactions.
+
+#### Split vote accounts into two accounts in VED and UED respectively
+We need to allow users move money in and out of the vote accounts, but
+we also need the vote accounts to vote in VED. So there will be two accounts:
+- `VoteTowerAccount`: tracks tower state and vote authority, it will be
+in `VED`, it is updated by vote transactions and tracks vote credits.
+- `VoteAccount`: everything else currently in vote accounts, it will be
+in `UED`, users can move funds in and out of `VoteAccount` freely.
+
+The two accounts are synced every Epoch when the rewards are calculated.
+
+### Separate the VED and UED Domains
+- Only Vote or System program can read and write accounts in `VED`
+- Other programs can only read accounts in `VED`
+- Users can't directly access accounts in `VED` but they can move accounts
+from `VED` to `UED` and vice versa. Moving accounts from one domain to
+another takes 1 Epoch, and the migration happens at Epoch boundary
+
+### Enable Async Vote Executions
+1. The leader will no longer execute any transactions before broadcasting
+the block it packed. We do have block-id (Merkle tree root) to ensure
+everyone receives the same block.
+2. Upon receiving a new block, the validator computes the `VED bankhash`,
+then vote on this block and also gives its latest `UED bankhash` on the
+same fork. The `UED bankhash` will most likely be hash of an ancestor of
+the received block.
+3. A block is not considered Optimistically Confirmed or Finalized until
+some percentage of the validators agree on the `UED bankhash`.
+4. Add assertion that confirmed `UED bankhash` is not too far away from the
+confirmed `VED bankhash` (currently proposed at 1/2 of the Epoch)
+5. Add alerts if `UED bankhash` differs when the `VED bankhash` is the same.
+This is potentially an event worthy of cluster restart.
\ No newline at end of file

From 74f3a3326ca4a3526edf733a1d85a3b049bc40eb Mon Sep 17 00:00:00 2001
From: Wen <113942165+wen-coding@users.noreply.github.com>
Date: Sun, 11 Aug 2024 18:58:15 -0700
Subject: [PATCH 02/24] Change SIMD number.

---
 ...154-async-vote-execution.md => 0165-async-vote-execution.md} | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
 rename proposals/{0154-async-vote-execution.md => 0165-async-vote-execution.md} (99%)

diff --git a/proposals/0154-async-vote-execution.md b/proposals/0165-async-vote-execution.md
similarity index 99%
rename from proposals/0154-async-vote-execution.md
rename to proposals/0165-async-vote-execution.md
index 0668f7b85..c58a963a4 100644
--- a/proposals/0154-async-vote-execution.md
+++ b/proposals/0165-async-vote-execution.md
@@ -1,5 +1,5 @@
 ---
-simd: '0154'
+simd: '0165'
 title: Async Vote Execution
 authors:
   - Wen Xu

From 4b5015366949cddae6bb79a5be8f3c2cf51a950f Mon Sep 17 00:00:00 2001
From: Wen <113942165+wen-coding@users.noreply.github.com>
Date: Sun, 11 Aug 2024 19:12:06 -0700
Subject: [PATCH 03/24] Make linter happy.

---
 proposals/0165-async-vote-execution.md | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/proposals/0165-async-vote-execution.md b/proposals/0165-async-vote-execution.md
index c58a963a4..346cadf75 100644
--- a/proposals/0165-async-vote-execution.md
+++ b/proposals/0165-async-vote-execution.md
@@ -12,12 +12,15 @@ supersedes: null
 superseded-by: null
 extends: null
 ---
+
 ## Summary
+
 Separate the execution of vote and non-vote transactions in each block. The
 vote transactions will be verified and executed first, then the non-vote
 transactions will be executed later asynchronously to finalize the block.
 
 ## Motivation
+
 Currently the vote transactions and non-vote transactions are mixed together in
 a block, the vote transactions are only processed in consensus when the whole
 block has been frozen and all transactions in the block have been verified and
@@ -42,6 +45,7 @@ hard to make sure everyone executes every block within 400ms, on average majorit
 of the cluster should be able to keep up.
 
 ## New Terminology
+
 - `VED`: Vote Execution Domain, Vote transactions and all its dependencies (e.g.
 fee payers for votes).
 - `VED Bankhash`: The hash calculated after executing only vote transactions in
@@ -54,6 +58,7 @@ in a block. If there are no non-vote transactions, use default hash.
 ## Detailed Design
 
 ### Allow leader to skip execution of transactions (Bankless Leader)
+
 There is already on-going effort to totally skip execution of all transactions
 when leader pack new blocks. See SIMD 82, SIMD 83, and related trackers:
 https://github.com/anza-xyz/agave/issues/2502
@@ -65,10 +70,12 @@ such a setup we gain smaller speedup without much benefits, it is a possible
 route during rollouts though.
 
 ### Separating vote transactions and dependencies into a different domain
+
 To make sure vote transactions can be executed independently, we need to
 isolate its dependencies.
 
 #### Remove clock program's dependency on votes
+
 Introduce new transaction `ClockBump` to remove current clock program's
 dependency on votes.
 
@@ -77,8 +84,10 @@ every 12 slots to correct the clock drift. A small script can be used to
 refund well-behaving leaders the cost of the transactions.
 
 #### Split vote accounts into two accounts in VED and UED respectively
+
 We need to allow users move money in and out of the vote accounts, but
 we also need the vote accounts to vote in VED. So there will be two accounts:
+
 - `VoteTowerAccount`: tracks tower state and vote authority, it will be
 in `VED`, it is updated by vote transactions and tracks vote credits.
 - `VoteAccount`: everything else currently in vote accounts, it will be
@@ -87,6 +96,7 @@ in `UED`, users can move funds in and out of `VoteAccount` freely.
 The two accounts are synced every Epoch when the rewards are calculated.
 
 ### Separate the VED and UED Domains
+
 - Only Vote or System program can read and write accounts in `VED`
 - Other programs can only read accounts in `VED`
 - Users can't directly access accounts in `VED` but they can move accounts
@@ -94,6 +104,7 @@ from `VED` to `UED` and vice versa. Moving accounts from one domain to
 another takes 1 Epoch, and the migration happens at Epoch boundary
 
 ### Enable Async Vote Executions
+
 1. The leader will no longer execute any transactions before broadcasting
 the block it packed. We do have block-id (Merkle tree root) to ensure
 everyone receives the same block.

From a8aad6cdde1dd0838958f31d30cceefd9fce651e Mon Sep 17 00:00:00 2001
From: Wen <113942165+wen-coding@users.noreply.github.com>
Date: Fri, 16 Aug 2024 13:43:35 -0700
Subject: [PATCH 04/24] Make linter happy.

---
 proposals/0165-async-vote-execution.md | 22 ++++++++++++++++++++--
 1 file changed, 20 insertions(+), 2 deletions(-)

diff --git a/proposals/0165-async-vote-execution.md b/proposals/0165-async-vote-execution.md
index 346cadf75..8b78f6f07 100644
--- a/proposals/0165-async-vote-execution.md
+++ b/proposals/0165-async-vote-execution.md
@@ -6,7 +6,7 @@ authors:
 category: Standard
 type: Core
 status: Draft
-created: 2024-08-11T00:00:00.000Z
+created: 2024-08-11
 feature: null
 supersedes: null
 superseded-by: null
@@ -44,6 +44,10 @@ The non-vote transactions do need to be executed eventually. Even though it's
 hard to make sure everyone executes every block within 400ms, on average majority
 of the cluster should be able to keep up.
 
+## Alternatives Considered
+
+To be filled.
+
 ## New Terminology
 
 - `VED`: Vote Execution Domain, Vote transactions and all its dependencies (e.g.
@@ -117,4 +121,18 @@ some percentage of the validators agree on the `UED bankhash`.
 4. Add assertion that confirmed `UED bankhash` is not too far away from the
 confirmed `VED bankhash` (currently proposed at 1/2 of the Epoch)
 5. Add alerts if `UED bankhash` differs when the `VED bankhash` is the same.
-This is potentially an event worthy of cluster restart.
\ No newline at end of file
+This is potentially an event worthy of cluster restart.
+
+## Impact
+
+Since we will eliminate the impact of non-vote transaction execution speed,
+we should expect to see fewer forking and late blocks.
+
+## Security Considerations
+
+We do need to monitor and address the possibility of UED bankhash mismatches
+when VED bankhash matches.
+
+## Backward Compatibility
+
+Most of the changes would require feature gates.

From 55fae1df3a4615ab34201f794d251d110fb1462c Mon Sep 17 00:00:00 2001
From: Wen <113942165+wen-coding@users.noreply.github.com>
Date: Sun, 25 Aug 2024 18:52:55 -0700
Subject: [PATCH 05/24] Update the plan for clock calculation.

---
 proposals/0165-async-vote-execution.md | 25 ++++++++++++++++++++-----
 1 file changed, 20 insertions(+), 5 deletions(-)

diff --git a/proposals/0165-async-vote-execution.md b/proposals/0165-async-vote-execution.md
index 8b78f6f07..b3dc9e4d4 100644
--- a/proposals/0165-async-vote-execution.md
+++ b/proposals/0165-async-vote-execution.md
@@ -16,8 +16,16 @@ extends: null
 ## Summary
 
 Separate the execution of vote and non-vote transactions in each block. The
-vote transactions will be verified and executed first, then the non-vote
-transactions will be executed later asynchronously to finalize the block.
+vote transactions and non-vote transactions will be verified and executed
+independently. Because the vote transactions are normally much faster to
+execute, under most cases this means vote transactions will finish
+execution first.
+
+The eventual goal is to completely separate vote and non-vote transaction
+executions. But right now we still have some dependency that non-vote
+transactions need to read the vote state in its parent block, so currently we
+restrict that non-vote transactions cannot start execution until the vote
+transactions of its parent block has finished.
 
 ## Motivation
 
@@ -78,15 +86,22 @@ route during rollouts though.
 To make sure vote transactions can be executed independently, we need to
 isolate its dependencies.
 
-#### Remove clock program's dependency on votes
+#### Remove clock program's dependency on votes (postponed)
 
-Introduce new transaction `ClockBump` to remove current clock program's
-dependency on votes.
+The eventual goal is to introduce new transaction `ClockBump` to remove
+current clock program's dependency on votes.
 
 The transaction `ClockBump` is sent by a leader with at least 0.5% stake
 every 12 slots to correct the clock drift. A small script can be used to
 refund well-behaving leaders the cost of the transactions.
 
+But currently we will keep the clock calculation as is, since the clock
+sysvar uses the vote-state of the parent block to calculate average
+timestamp, and we have restrictions that the UED transactions in a
+block cannot start execution until the VED transactions in its
+ancestors have finished. So we can currently leave clock sysvar
+calculation as is.
+
 #### Split vote accounts into two accounts in VED and UED respectively
 
 We need to allow users move money in and out of the vote accounts, but

From ca682d78ec1bf3ccc7d113aa6eef68b3eb379f82 Mon Sep 17 00:00:00 2001
From: Wen <113942165+wen-coding@users.noreply.github.com>
Date: Thu, 19 Sep 2024 17:46:53 +0800
Subject: [PATCH 06/24] Update the proposal to reflect optimistic vote
 execution plan we discussed today.

---
 proposals/0165-async-vote-execution.md | 147 +++++++++++++------------
 1 file changed, 77 insertions(+), 70 deletions(-)

diff --git a/proposals/0165-async-vote-execution.md b/proposals/0165-async-vote-execution.md
index b3dc9e4d4..d80453fca 100644
--- a/proposals/0165-async-vote-execution.md
+++ b/proposals/0165-async-vote-execution.md
@@ -15,33 +15,34 @@ extends: null
 
 ## Summary
 
-Separate the execution of vote and non-vote transactions in each block. The
-vote transactions and non-vote transactions will be verified and executed
-independently. Because the vote transactions are normally much faster to
-execute, under most cases this means vote transactions will finish
-execution first.
-
-The eventual goal is to completely separate vote and non-vote transaction
-executions. But right now we still have some dependency that non-vote
-transactions need to read the vote state in its parent block, so currently we
-restrict that non-vote transactions cannot start execution until the vote
-transactions of its parent block has finished.
+Optimistically execute all vote transactions in a block to determine fork
+selection in consensus early on, before all the transactions in the block
+are fully executed and the actual fee payers for vote transactions are
+checked.
+
+This allows us to more quickly converge on one chain of blocks, so that
+validators don't have to execute any blocks not on selected fork. This saves
+CPU and memory resource needed in replay, it also ensures that the cluster
+will have fewer forks that are caused by slow transaction execution.
 
 ## Motivation
 
 Currently the vote transactions and non-vote transactions are mixed together in
 a block, the vote transactions are only processed in consensus when the whole
 block has been frozen and all transactions in the block have been verified and
-executed. This is a problem because slow running non-vote transactions may affect
-how fast the votes are processed and then affect the ability of consensus to
-pick the correct fork.
+executed. This is a problem because slow running non-vote transactions may
+affect how fast the votes are processed and then affect the ability of
+consensus to pick the correct fork. It may also mean that the leader will more
+often build on a minority fork so the blocks it packed will be discarded later.
 
 With different hardware and running environment, there will always be some
 difference on speed of transaction execution between validators. Generally
 speaking, because vote transactions are so simple, the variation between vote
-execution should be smaller than that between non-vote executions. Therefore,
-if we only execute vote transactions in a block before voting on the block,
-it is more likely validators can reach consensus faster.
+execution should be smaller than that between non-vote executions. Also the
+vote transactions are very simple and lock-free, so they normally execute
+faster than non-vote transactions. Therefore, if we only execute vote
+transactions in a block before voting on the block, it is more likely
+validators can reach consensus faster.
 
 Even with async vote execution, forks can still happen because of
 various other situations, like network partitions or mis-configured validators.
@@ -49,23 +50,31 @@ This work just reduces the chances of forks caused by variance in non-vote
 transaction executions.
 
 The non-vote transactions do need to be executed eventually. Even though it's
-hard to make sure everyone executes every block within 400ms, on average majority
-of the cluster should be able to keep up.
+hard to make sure everyone executes every block within 400ms, on average
+majority of the cluster should be able to keep up.
 
 ## Alternatives Considered
 
-To be filled.
+### Separating vote and non-vote transactions into different domains
+
+An earlier proposal of Async Execution proposes that we separate vote and
+non-vote transactions into different domains, so that we can execute them
+independently. The main concerns were:
+
+* We need to introduce one bit in AccountsDB for every account, this
+complicates the implementation
+
+* Topping off the vote fee payer accounts becomes difficult. We need to add a
+bounce account to move fees from user domain to vote domain, and the process
+may take one epoch
 
 ## New Terminology
 
-- `VED`: Vote Execution Domain, Vote transactions and all its dependencies (e.g.
-fee payers for votes).
 - `VED Bankhash`: The hash calculated after executing only vote transactions in
-a block. If there are no votes, use default hash.
-- `UED`: User Execution Domain, currently everything other than votes. We may
-have more domains in the future.
-- `UED Bankhash`: The hash calculated after executing only non-vote transactions
-in a block. If there are no non-vote transactions, use default hash.
+a block without checking fee payers. If there are no votes, use default hash.
+- `UED Bankhash`: The hash calculated after executing all transactions in a
+block, checking fee payers for all transactions. This is the same as the
+bankhash we use today.
 
 ## Detailed Design
 
@@ -81,61 +90,58 @@ execute other transactions and compare with the bankhash of the leader. But in
 such a setup we gain smaller speedup without much benefits, it is a possible
 route during rollouts though.
 
-### Separating vote transactions and dependencies into a different domain
-
-To make sure vote transactions can be executed independently, we need to
-isolate its dependencies.
-
-#### Remove clock program's dependency on votes (postponed)
-
-The eventual goal is to introduce new transaction `ClockBump` to remove
-current clock program's dependency on votes.
-
-The transaction `ClockBump` is sent by a leader with at least 0.5% stake
-every 12 slots to correct the clock drift. A small script can be used to
-refund well-behaving leaders the cost of the transactions.
-
-But currently we will keep the clock calculation as is, since the clock
-sysvar uses the vote-state of the parent block to calculate average
-timestamp, and we have restrictions that the UED transactions in a
-block cannot start execution until the VED transactions in its
-ancestors have finished. So we can currently leave clock sysvar
-calculation as is.
+### Optimistically execute vote transactions and vote on selected forks
 
-#### Split vote accounts into two accounts in VED and UED respectively
+This step is optimistic in the sense that validators do not check the fee
+payers when executing the vote transactions in a block. They assume vote
+transactions will not fail due to insufficient fees, apply the execution
+results to select the correct fork, then immediately vote on the bank with
+only the hash result of the vote transactions.
 
-We need to allow users move money in and out of the vote accounts, but
-we also need the vote accounts to vote in VED. So there will be two accounts:
+This is safe because the list of validators and their corresponding stake
+has already been determined at the beginning of the Epoch. The stake we used
+is correct in fork selection.
 
-- `VoteTowerAccount`: tracks tower state and vote authority, it will be
-in `VED`, it is updated by vote transactions and tracks vote credits.
-- `VoteAccount`: everything else currently in vote accounts, it will be
-in `UED`, users can move funds in and out of `VoteAccount` freely.
+### Replay the full block on selected forks later
 
-The two accounts are synced every Epoch when the rewards are calculated.
+Once a validator determined the fork it will vote on, only blocks on this fork
+will be replayed. The replay process is the same as today, all transactions
+(vote and non-vote) will be executed to determine the final bankhash. The
+computed bankhash will be attached to vote instructions. So we can still detect
+non-determinism (same of set of instructions leading to different results) as
+today, just we might find discrepancies at a later time.
 
-### Separate the VED and UED Domains
+In this step the validators will check the fee payers of vote transactions. So
+each vote transaction is executed twice: once in the optimistical voting stage
+*without* checking fee payer, and once in this stage *with* checking fee payer.
+If a staked validator does not have vote fee covered for specific votes, we
+will not accept the vote today, while in the future we accept the vote in fork
+selection, but does not actually give vote credits because the transaction
+failed.
 
-- Only Vote or System program can read and write accounts in `VED`
-- Other programs can only read accounts in `VED`
-- Users can't directly access accounts in `VED` but they can move accounts
-from `VED` to `UED` and vice versa. Moving accounts from one domain to
-another takes 1 Epoch, and the migration happens at Epoch boundary
+A side benefit of this design is that the sysvars have all been calculated and
+determined in the optimistical voting stage, also the fork of blocks are known,
+so it may be possible to release more parallelism when executing blocks in this
+stage.
 
 ### Enable Async Vote Executions
 
 1. The leader will no longer execute any transactions before broadcasting
 the block it packed. We do have block-id (Merkle tree root) to ensure
 everyone receives the same block.
-2. Upon receiving a new block, the validator computes the `VED bankhash`,
-then vote on this block and also gives its latest `UED bankhash` on the
-same fork. The `UED bankhash` will most likely be hash of an ancestor of
-the received block.
-3. A block is not considered Optimistically Confirmed or Finalized until
+2. Upon receiving a new block, the validator executes only the vote
+transactions without checking the fee payers. The result is immediately
+applied in consensus to select a fork. Then votes are sent out for the
+selected fork with the `VED bankhash` for the tip of the fork and the
+most recent `UED bankhash`.
+3. The blocks on the selected forks are scheduled to be replayed. When
+a block is replayed, all transactions are executed with fee payers checked.
+This is the same as the replay we use today.
+4. A block is not considered Optimistically Confirmed or Finalized until
 some percentage of the validators agree on the `UED bankhash`.
-4. Add assertion that confirmed `UED bankhash` is not too far away from the
+5. Add assertion that confirmed `UED bankhash` is not too far away from the
 confirmed `VED bankhash` (currently proposed at 1/2 of the Epoch)
-5. Add alerts if `UED bankhash` differs when the `VED bankhash` is the same.
+6. Add alerts if `UED bankhash` differs when the `VED bankhash` is the same.
 This is potentially an event worthy of cluster restart.
 
 ## Impact
@@ -145,8 +151,9 @@ we should expect to see fewer forking and late blocks.
 
 ## Security Considerations
 
-We do need to monitor and address the possibility of UED bankhash mismatches
-when VED bankhash matches.
+We do need to monitor and address the possibility of bankhash mismatches
+when the tip of the fork is far away from the slot where bankhash mismatch
+happened.
 
 ## Backward Compatibility
 

From 93b2506547e9d436b921a6465c1b855d55f8772e Mon Sep 17 00:00:00 2001
From: Wen <113942165+wen-coding@users.noreply.github.com>
Date: Thu, 19 Sep 2024 17:48:33 +0800
Subject: [PATCH 07/24] Update dash to asterisk.

---
 proposals/0165-async-vote-execution.md | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/proposals/0165-async-vote-execution.md b/proposals/0165-async-vote-execution.md
index d80453fca..49a33ebcb 100644
--- a/proposals/0165-async-vote-execution.md
+++ b/proposals/0165-async-vote-execution.md
@@ -70,9 +70,9 @@ may take one epoch
 
 ## New Terminology
 
-- `VED Bankhash`: The hash calculated after executing only vote transactions in
+* `VED Bankhash`: The hash calculated after executing only vote transactions in
 a block without checking fee payers. If there are no votes, use default hash.
-- `UED Bankhash`: The hash calculated after executing all transactions in a
+* `UED Bankhash`: The hash calculated after executing all transactions in a
 block, checking fee payers for all transactions. This is the same as the
 bankhash we use today.
 

From a78f256da96bdff8d8ef9e4f61539010bb927cf7 Mon Sep 17 00:00:00 2001
From: Wen <113942165+wen-coding@users.noreply.github.com>
Date: Fri, 20 Sep 2024 08:24:53 +0800
Subject: [PATCH 08/24] Rename VED and UED hash to Ephemeral and Final hash.

---
 proposals/0165-async-vote-execution.md | 24 ++++++++++++------------
 1 file changed, 12 insertions(+), 12 deletions(-)

diff --git a/proposals/0165-async-vote-execution.md b/proposals/0165-async-vote-execution.md
index 49a33ebcb..437a7d4f5 100644
--- a/proposals/0165-async-vote-execution.md
+++ b/proposals/0165-async-vote-execution.md
@@ -70,11 +70,11 @@ may take one epoch
 
 ## New Terminology
 
-* `VED Bankhash`: The hash calculated after executing only vote transactions in
-a block without checking fee payers. If there are no votes, use default hash.
-* `UED Bankhash`: The hash calculated after executing all transactions in a
-block, checking fee payers for all transactions. This is the same as the
-bankhash we use today.
+* `Ephemeral Bankhash`: The hash calculated after executing only vote
+transactions in a block without checking fee payers. If there are no votes,
+use default hash.
+* `Final Bankhash`: The bankhash as we know it today. This is a hash calculated
+after executing all transactions in a block, checking fee payers for all.
 
 ## Detailed Design
 
@@ -132,17 +132,17 @@ everyone receives the same block.
 2. Upon receiving a new block, the validator executes only the vote
 transactions without checking the fee payers. The result is immediately
 applied in consensus to select a fork. Then votes are sent out for the
-selected fork with the `VED bankhash` for the tip of the fork and the
-most recent `UED bankhash`.
+selected fork with the `Ephemeral bankhash` for the tip of the fork and the
+most recent `Final bankhash`.
 3. The blocks on the selected forks are scheduled to be replayed. When
 a block is replayed, all transactions are executed with fee payers checked.
 This is the same as the replay we use today.
 4. A block is not considered Optimistically Confirmed or Finalized until
-some percentage of the validators agree on the `UED bankhash`.
-5. Add assertion that confirmed `UED bankhash` is not too far away from the
-confirmed `VED bankhash` (currently proposed at 1/2 of the Epoch)
-6. Add alerts if `UED bankhash` differs when the `VED bankhash` is the same.
-This is potentially an event worthy of cluster restart.
+some percentage of the validators agree on the `Ephemeral bankhash`.
+5. Add assertion that confirmed `Final bankhash` is not too far away from the
+confirmed `Ephemeral bankhash` (currently proposed at 1/2 of the Epoch)
+6. Add alerts if `Final bankhash` differs when the `Ephemeral bankhash` is the
+same. This is potentially an event worthy of cluster restart.
 
 ## Impact
 

From 67a1f4520d76d890e35d2a8ad8fdaa6a78ff8a16 Mon Sep 17 00:00:00 2001
From: Wen <113942165+wen-coding@users.noreply.github.com>
Date: Fri, 20 Sep 2024 08:26:47 +0800
Subject: [PATCH 09/24] Change title to reflect that we calculate Ephemeral
 hash.

---
 proposals/0165-async-vote-execution.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/proposals/0165-async-vote-execution.md b/proposals/0165-async-vote-execution.md
index 437a7d4f5..14b7ab373 100644
--- a/proposals/0165-async-vote-execution.md
+++ b/proposals/0165-async-vote-execution.md
@@ -90,7 +90,7 @@ execute other transactions and compare with the bankhash of the leader. But in
 such a setup we gain smaller speedup without much benefits, it is a possible
 route during rollouts though.
 
-### Optimistically execute vote transactions and vote on selected forks
+### Calculate ephemeral hash executing votes only and vote on selected forks
 
 This step is optimistic in the sense that validators do not check the fee
 payers when executing the vote transactions in a block. They assume vote

From e38b645864c227b4abb4011872381510f1c9ea1d Mon Sep 17 00:00:00 2001
From: Wen <113942165+wen-coding@users.noreply.github.com>
Date: Fri, 27 Sep 2024 11:56:12 -0700
Subject: [PATCH 10/24] Explain the checks we will perform during ephemeral
 hash computation.

---
 proposals/0165-async-vote-execution.md | 20 ++++++++++++++++++++
 1 file changed, 20 insertions(+)

diff --git a/proposals/0165-async-vote-execution.md b/proposals/0165-async-vote-execution.md
index 14b7ab373..ab0fa515f 100644
--- a/proposals/0165-async-vote-execution.md
+++ b/proposals/0165-async-vote-execution.md
@@ -102,6 +102,26 @@ This is safe because the list of validators and their corresponding stake
 has already been determined at the beginning of the Epoch. The stake we used
 is correct in fork selection.
 
+To make sure the vote casted would be the same as that after replaying the
+whole block, we need to be consistent on whether we mark the block dead, so
+that the ephemeral hash vote doesn't vote on a block which will be marked
+dead later. Currently a block can be dead for the following reasons:
+
+1. Unable to load data from blockstore
+2. Invalid block (wrong number of ticks, duplicate block, bad last fec, etc)
+3. Error while set root
+4. Invalid transaction
+
+For the first two, the same check can be performed computing ephemeral hash.
+We will set root on a bank only when it has full hash computed later, so the
+behavior will be the same as now.
+
+The only operation we can't check is invalid transaction, since we will skip
+all non-vote transaction execution, there is no way we can check for validity
+of those. The intention of this check was to prevent spams. We will remove
+this check and rely on economic incentives so that the leader can perform
+appropriate checks.
+
 ### Replay the full block on selected forks later
 
 Once a validator determined the fork it will vote on, only blocks on this fork

From 7dd98012a8eec3f4a3416a35a3d49e2240cb717d Mon Sep 17 00:00:00 2001
From: Wen <113942165+wen-coding@users.noreply.github.com>
Date: Fri, 27 Sep 2024 14:58:52 -0700
Subject: [PATCH 11/24] Add new fields in TowerSync.

---
 proposals/0165-async-vote-execution.md | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/proposals/0165-async-vote-execution.md b/proposals/0165-async-vote-execution.md
index ab0fa515f..554a33be8 100644
--- a/proposals/0165-async-vote-execution.md
+++ b/proposals/0165-async-vote-execution.md
@@ -92,6 +92,12 @@ route during rollouts though.
 
 ### Calculate ephemeral hash executing votes only and vote on selected forks
 
+Two new fields will be added to `TowerSync` vote transaction:
+
+* `ephemeral_hash`: This is the hash obtained by hashing blockid and all
+vote transactions in the block together. Used primarily in consensus.
+* `ephemeral_slot`: This is the slot where the ephemeral_hash is calculated.
+
 This step is optimistic in the sense that validators do not check the fee
 payers when executing the vote transactions in a block. They assume vote
 transactions will not fail due to insufficient fees, apply the execution

From 60a66afd367b9d3fca84440a9110aaae881a6668 Mon Sep 17 00:00:00 2001
From: Wen <113942165+wen-coding@users.noreply.github.com>
Date: Fri, 27 Sep 2024 15:29:18 -0700
Subject: [PATCH 12/24] Update that which block (slot, hash) on the vote
 transaction is.

---
 proposals/0165-async-vote-execution.md | 16 ++++++++++------
 1 file changed, 10 insertions(+), 6 deletions(-)

diff --git a/proposals/0165-async-vote-execution.md b/proposals/0165-async-vote-execution.md
index 554a33be8..e3656dcd4 100644
--- a/proposals/0165-async-vote-execution.md
+++ b/proposals/0165-async-vote-execution.md
@@ -130,12 +130,16 @@ appropriate checks.
 
 ### Replay the full block on selected forks later
 
-Once a validator determined the fork it will vote on, only blocks on this fork
-will be replayed. The replay process is the same as today, all transactions
-(vote and non-vote) will be executed to determine the final bankhash. The
-computed bankhash will be attached to vote instructions. So we can still detect
-non-determinism (same of set of instructions leading to different results) as
-today, just we might find discrepancies at a later time.
+There is no protocol enforced order of block replay for various validator
+implemenations, but when vote instructions are sent out, the `(slot, hash)`
+is the hash of latest replayed block on the selected fork.
+
+Once a validator determined the fork it will vote on, it can prioritize
+replaying blocks on the selected fork. The replay process is the same as today,
+all transactions (vote and non-vote) will be executed to determine the final
+bankhash. The computed bankhash will be attached to vote instructions. So we
+can still detect non-determinism (same of set of instructions leading to
+different results) as today, just we might find discrepancies at a later time.
 
 In this step the validators will check the fee payers of vote transactions. So
 each vote transaction is executed twice: once in the optimistical voting stage

From 5432353e5e78b8065b1887543310cf92cd9f6ee0 Mon Sep 17 00:00:00 2001
From: Wen <113942165+wen-coding@users.noreply.github.com>
Date: Fri, 27 Sep 2024 15:33:53 -0700
Subject: [PATCH 13/24] Clarify that new votes should be sent out when either
 hash changes.

---
 proposals/0165-async-vote-execution.md | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/proposals/0165-async-vote-execution.md b/proposals/0165-async-vote-execution.md
index e3656dcd4..d7791fcd0 100644
--- a/proposals/0165-async-vote-execution.md
+++ b/proposals/0165-async-vote-execution.md
@@ -132,7 +132,8 @@ appropriate checks.
 
 There is no protocol enforced order of block replay for various validator
 implemenations, but when vote instructions are sent out, the `(slot, hash)`
-is the hash of latest replayed block on the selected fork.
+is the hash of latest replayed block on the selected fork. New vote
+transactions should be sent when the ephemeral hash or final hash changes.
 
 Once a validator determined the fork it will vote on, it can prioritize
 replaying blocks on the selected fork. The replay process is the same as today,

From 40d4be66b816ca16934274d81ab05e7f360e67e1 Mon Sep 17 00:00:00 2001
From: Wen <113942165+wen-coding@users.noreply.github.com>
Date: Fri, 25 Oct 2024 13:13:53 -0700
Subject: [PATCH 14/24] Propose to halt and exit if >1/3 disagrees on final
 bankhash.

---
 proposals/0165-async-vote-execution.md | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/proposals/0165-async-vote-execution.md b/proposals/0165-async-vote-execution.md
index d7791fcd0..cfc7017b0 100644
--- a/proposals/0165-async-vote-execution.md
+++ b/proposals/0165-async-vote-execution.md
@@ -173,7 +173,8 @@ some percentage of the validators agree on the `Ephemeral bankhash`.
 5. Add assertion that confirmed `Final bankhash` is not too far away from the
 confirmed `Ephemeral bankhash` (currently proposed at 1/2 of the Epoch)
 6. Add alerts if `Final bankhash` differs when the `Ephemeral bankhash` is the
-same. This is potentially an event worthy of cluster restart.
+same. This is potentially an event worthy of cluster restart. If more than
+1/3 of the validators claim a different `Final bankhash`, halt and exit.
 
 ## Impact
 

From 1b3ed81acc2253118413c0c0b96586680b1749f1 Mon Sep 17 00:00:00 2001
From: Wen <113942165+wen-coding@users.noreply.github.com>
Date: Fri, 25 Oct 2024 13:16:03 -0700
Subject: [PATCH 15/24] Change status to Idea.

---
 proposals/0165-async-vote-execution.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/proposals/0165-async-vote-execution.md b/proposals/0165-async-vote-execution.md
index cfc7017b0..6249477af 100644
--- a/proposals/0165-async-vote-execution.md
+++ b/proposals/0165-async-vote-execution.md
@@ -5,7 +5,7 @@ authors:
   - Wen Xu
 category: Standard
 type: Core
-status: Draft
+status: Idea
 created: 2024-08-11
 feature: null
 supersedes: null

From 4e83392ebc694e57f5bc88e986b953a2d41633e8 Mon Sep 17 00:00:00 2001
From: Wen <113942165+wen-coding@users.noreply.github.com>
Date: Thu, 28 Nov 2024 12:16:37 -0800
Subject: [PATCH 16/24] Update calculation of vote only hash and others.

---
 proposals/0165-async-vote-execution.md | 90 +++++++++++++++++---------
 1 file changed, 60 insertions(+), 30 deletions(-)

diff --git a/proposals/0165-async-vote-execution.md b/proposals/0165-async-vote-execution.md
index 6249477af..b3acfc1bf 100644
--- a/proposals/0165-async-vote-execution.md
+++ b/proposals/0165-async-vote-execution.md
@@ -70,10 +70,10 @@ may take one epoch
 
 ## New Terminology
 
-* `Ephemeral Bankhash`: The hash calculated after executing only vote
-transactions in a block without checking fee payers. If there are no votes,
-use default hash.
-* `Final Bankhash`: The bankhash as we know it today. This is a hash calculated
+* `Vote Only Bankhash`: The hash calculated after executing only vote
+transactions in a block without checking fee payers. The exact calculation
+algorithm is listed in next section.
+* `Replay Tip Bankhash`: The bankhash as we know it today. It is calculated
 after executing all transactions in a block, checking fee payers for all.
 
 ## Detailed Design
@@ -85,18 +85,32 @@ when leader pack new blocks. See SIMD 82, SIMD 83, and related trackers:
 https://github.com/anza-xyz/agave/issues/2502
 
 Theoretically we could reap some benefit without Bankless Leader, the leader
-pack as normal, while other validators only replay votes first, then later
+packs as normal, while other validators only replay votes first, then later
 execute other transactions and compare with the bankhash of the leader. But in
 such a setup we gain smaller speedup without much benefits, it is a possible
 route during rollouts though.
 
-### Calculate ephemeral hash executing votes only and vote on selected forks
+### Calculate vote only hash executing votes only and vote on selected forks
 
 Two new fields will be added to `TowerSync` vote transaction:
 
-* `ephemeral_hash`: This is the hash obtained by hashing blockid and all
-vote transactions in the block together. Used primarily in consensus.
-* `ephemeral_slot`: This is the slot where the ephemeral_hash is calculated.
+* `replay_tip_hash`: This is the hash as we know it today.
+* `replay_tip_slot`: This is the slot where the replay tip hash is calculated.
+
+The `hash` and `slot` in the `TowerSync` transaction will be updated to
+the vote only hash. The vote only hash is calculated as follows:
+
+1. Sort all vote accounts with non-zero stake in the current epoch by
+vote account pubkey.
+
+2. Calculate vote account hash by hashing (vote account pubkey,
+serialized vote state) in the order given.
+
+3. Calculate vote only hash by hashing:
+
+* vote only hash of the parent bank
+* vote account hash calculated aboveß
+* block-id of the current bank
 
 This step is optimistic in the sense that validators do not check the fee
 payers when executing the vote transactions in a block. They assume vote
@@ -105,8 +119,9 @@ results to select the correct fork, then immediately vote on the bank with
 only the hash result of the vote transactions.
 
 This is safe because the list of validators and their corresponding stake
-has already been determined at the beginning of the Epoch. The stake we used
-is correct in fork selection.
+uses the leader scheduler epoch stakes, which is calculated at the beginning
+of last Epoch. Because full execution is never behind the optimistic execution
+by more than one Epoch, the epoch stakes used is stable and correct.
 
 To make sure the vote casted would be the same as that after replaying the
 whole block, we need to be consistent on whether we mark the block dead, so
@@ -128,22 +143,38 @@ of those. The intention of this check was to prevent spams. We will remove
 this check and rely on economic incentives so that the leader can perform
 appropriate checks.
 
-### Replay the full block on selected forks later
+The vote only execution will operate exclusively on replicated vote states
+stored outside the accounts DB, so vote only execution and full execution can
+happen asynchronously in any order. The vote authority of each vote account
+will be copied from accounts DB at the beginning of each epoch, this means
+in the future vote authority change will take two epochs instead of one
+epoch.
+
+### Replay the full block on selected forks asynchronously
 
 There is no protocol enforced order of block replay for various validator
-implemenations, but when vote instructions are sent out, the `(slot, hash)`
-is the hash of latest replayed block on the selected fork. New vote
-transactions should be sent when the ephemeral hash or final hash changes.
+implemenations, new vote transactions could be sent when the vote only hash
+or final hash changes.
 
-Once a validator determined the fork it will vote on, it can prioritize
+Once a validator has determined the fork it will vote on, it can prioritize
 replaying blocks on the selected fork. The replay process is the same as today,
 all transactions (vote and non-vote) will be executed to determine the final
 bankhash. The computed bankhash will be attached to vote instructions. So we
 can still detect non-determinism (same of set of instructions leading to
-different results) as today, just we might find discrepancies at a later time.
+different results) like today, only that maybe at a later time.
+
+To guarantee the blockchain will halt when full replay is having problems, we
+propose:
+
+1. If full replay is behind vote only replay by more than 1/2 epoch or vice
+versa, stop producing new blocks until the lagging replay catches up. Also set
+up monitoring if the distance between two replays are growing larger.
+
+2. If more than 1/3 of the validators send a different final hash for a block
+with the same vote only hash, panic and prompt for further debugging.
 
 In this step the validators will check the fee payers of vote transactions. So
-each vote transaction is executed twice: once in the optimistical voting stage
+each vote transaction is executed twice: once in the optimistic voting stage
 *without* checking fee payer, and once in this stage *with* checking fee payer.
 If a staked validator does not have vote fee covered for specific votes, we
 will not accept the vote today, while in the future we accept the vote in fork
@@ -151,9 +182,8 @@ selection, but does not actually give vote credits because the transaction
 failed.
 
 A side benefit of this design is that the sysvars have all been calculated and
-determined in the optimistical voting stage, also the fork of blocks are known,
-so it may be possible to release more parallelism when executing blocks in this
-stage.
+determined in the vote only execution, also the fork of blocks are known, so it
+may be possible to release more parallelism during full executions.
 
 ### Enable Async Vote Executions
 
@@ -163,18 +193,18 @@ everyone receives the same block.
 2. Upon receiving a new block, the validator executes only the vote
 transactions without checking the fee payers. The result is immediately
 applied in consensus to select a fork. Then votes are sent out for the
-selected fork with the `Ephemeral bankhash` for the tip of the fork and the
-most recent `Final bankhash`.
+selected fork with the `Vote only bankhash` for the tip of the fork and the
+most recent `Replay tip bankhash`.
 3. The blocks on the selected forks are scheduled to be replayed. When
 a block is replayed, all transactions are executed with fee payers checked.
 This is the same as the replay we use today.
-4. A block is not considered Optimistically Confirmed or Finalized until
-some percentage of the validators agree on the `Ephemeral bankhash`.
-5. Add assertion that confirmed `Final bankhash` is not too far away from the
-confirmed `Ephemeral bankhash` (currently proposed at 1/2 of the Epoch)
-6. Add alerts if `Final bankhash` differs when the `Ephemeral bankhash` is the
-same. This is potentially an event worthy of cluster restart. If more than
-1/3 of the validators claim a different `Final bankhash`, halt and exit.
+4. Optimisticly confirmed or finalized on `Vote only bankhash` and `Replay tip
+bankhash` will be both exposed through RPC so users can select accordingly.
+5. Add assertion that confirmed `Replay tip bankhash` is not too far away from
+the confirmed `Vote only bankhash` (currently proposed at 1/2 of the Epoch)
+6. Add alerts if `Replay tip bankhash` differs when the `Vote only bankhash` is
+the same. This is potentially an event worthy of cluster restart. If more than
+1/3 of the validators claim a different `Replay tip bankhash`, halt and exit.
 
 ## Impact
 

From 35ec6178f22d16489474426bfedd6ca430cf9080 Mon Sep 17 00:00:00 2001
From: Wen <113942165+wen-coding@users.noreply.github.com>
Date: Wed, 11 Dec 2024 12:22:20 -0800
Subject: [PATCH 17/24] Fix some small problems.

---
 proposals/0165-async-vote-execution.md | 8 ++------
 1 file changed, 2 insertions(+), 6 deletions(-)

diff --git a/proposals/0165-async-vote-execution.md b/proposals/0165-async-vote-execution.md
index b3acfc1bf..1674ccf96 100644
--- a/proposals/0165-async-vote-execution.md
+++ b/proposals/0165-async-vote-execution.md
@@ -109,7 +109,7 @@ serialized vote state) in the order given.
 3. Calculate vote only hash by hashing:
 
 * vote only hash of the parent bank
-* vote account hash calculated aboveß
+* vote account hash calculated above
 * block-id of the current bank
 
 This step is optimistic in the sense that validators do not check the fee
@@ -154,7 +154,7 @@ epoch.
 
 There is no protocol enforced order of block replay for various validator
 implemenations, new vote transactions could be sent when the vote only hash
-or final hash changes.
+or replay tip hash changes.
 
 Once a validator has determined the fork it will vote on, it can prioritize
 replaying blocks on the selected fork. The replay process is the same as today,
@@ -181,10 +181,6 @@ will not accept the vote today, while in the future we accept the vote in fork
 selection, but does not actually give vote credits because the transaction
 failed.
 
-A side benefit of this design is that the sysvars have all been calculated and
-determined in the vote only execution, also the fork of blocks are known, so it
-may be possible to release more parallelism during full executions.
-
 ### Enable Async Vote Executions
 
 1. The leader will no longer execute any transactions before broadcasting

From 93372ea556d56cdbed3e2fedd46ae3dee44301a1 Mon Sep 17 00:00:00 2001
From: Wen <113942165+wen-coding@users.noreply.github.com>
Date: Wed, 11 Dec 2024 14:13:53 -0800
Subject: [PATCH 18/24] Address review comments.

---
 proposals/0165-async-vote-execution.md | 16 ++++++++--------
 1 file changed, 8 insertions(+), 8 deletions(-)

diff --git a/proposals/0165-async-vote-execution.md b/proposals/0165-async-vote-execution.md
index 1674ccf96..83fa1db41 100644
--- a/proposals/0165-async-vote-execution.md
+++ b/proposals/0165-async-vote-execution.md
@@ -28,12 +28,12 @@ will have fewer forks that are caused by slow transaction execution.
 ## Motivation
 
 Currently the vote transactions and non-vote transactions are mixed together in
-a block, the vote transactions are only processed in consensus when the whole
-block has been frozen and all transactions in the block have been verified and
-executed. This is a problem because slow running non-vote transactions may
-affect how fast the votes are processed and then affect the ability of
-consensus to pick the correct fork. It may also mean that the leader will more
-often build on a minority fork so the blocks it packed will be discarded later.
+a block, a block is considered in consensus only after the whole block has been
+frozen and all transactions in the block have been verified and executed. This
+is a problem because slow running non-vote transactions may affect affect the
+ability of consensus to pick the correct fork. It may also mean that the leader
+will more often build on a minority fork so the blocks it packed will be
+discarded later.
 
 With different hardware and running environment, there will always be some
 difference on speed of transaction execution between validators. Generally
@@ -100,8 +100,8 @@ Two new fields will be added to `TowerSync` vote transaction:
 The `hash` and `slot` in the `TowerSync` transaction will be updated to
 the vote only hash. The vote only hash is calculated as follows:
 
-1. Sort all vote accounts with non-zero stake in the current epoch by
-vote account pubkey.
+1. Sort all vote accounts with non-zero stake in the current or previous
+epoch by vote account pubkey.
 
 2. Calculate vote account hash by hashing (vote account pubkey,
 serialized vote state) in the order given.

From 7daaf7acc144b6cf1db49139d26f887c8c402046 Mon Sep 17 00:00:00 2001
From: Wen <113942165+wen-coding@users.noreply.github.com>
Date: Wed, 11 Dec 2024 19:09:04 -0800
Subject: [PATCH 19/24] Specify the hash function used.

---
 proposals/0165-async-vote-execution.md | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)

diff --git a/proposals/0165-async-vote-execution.md b/proposals/0165-async-vote-execution.md
index 83fa1db41..0e5608ec5 100644
--- a/proposals/0165-async-vote-execution.md
+++ b/proposals/0165-async-vote-execution.md
@@ -103,10 +103,11 @@ the vote only hash. The vote only hash is calculated as follows:
 1. Sort all vote accounts with non-zero stake in the current or previous
 epoch by vote account pubkey.
 
-2. Calculate vote account hash by hashing (vote account pubkey,
-serialized vote state) in the order given.
+2. Calculate vote account hash by calculating sha256 hash of (vote account
+pubkey, serialized vote state) in the order given.
 
-3. Calculate vote only hash by hashing:
+3. Calculate vote only hash by calculating sha256 hash of the following in
+the given order:
 
 * vote only hash of the parent bank
 * vote account hash calculated above

From b7fc4039f73cb802eab5c1607d0168da6cefe61f Mon Sep 17 00:00:00 2001
From: Wen <113942165+wen-coding@users.noreply.github.com>
Date: Thu, 12 Dec 2024 11:14:41 -0800
Subject: [PATCH 20/24] Clarify the set root error.

---
 proposals/0165-async-vote-execution.md | 9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)

diff --git a/proposals/0165-async-vote-execution.md b/proposals/0165-async-vote-execution.md
index 0e5608ec5..42614d144 100644
--- a/proposals/0165-async-vote-execution.md
+++ b/proposals/0165-async-vote-execution.md
@@ -131,12 +131,13 @@ dead later. Currently a block can be dead for the following reasons:
 
 1. Unable to load data from blockstore
 2. Invalid block (wrong number of ticks, duplicate block, bad last fec, etc)
-3. Error while set root
+3. Error while performing set root on the new root
 4. Invalid transaction
 
-For the first two, the same check can be performed computing ephemeral hash.
-We will set root on a bank only when it has full hash computed later, so the
-behavior will be the same as now.
+For the first three, the same check can be performed computing vote only
+hash. We will add a new check that the new root must be vote only replayed
+and fully replayed, this may mean the tower has more than 32 slots
+occasionally.
 
 The only operation we can't check is invalid transaction, since we will skip
 all non-vote transaction execution, there is no way we can check for validity

From f4278f43999488e1e73ede6c52c06bf5127f5581 Mon Sep 17 00:00:00 2001
From: Wen <113942165+wen-coding@users.noreply.github.com>
Date: Thu, 12 Dec 2024 12:33:40 -0800
Subject: [PATCH 21/24] Set root will not cause the block to be marked dead.

---
 proposals/0165-async-vote-execution.md | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)

diff --git a/proposals/0165-async-vote-execution.md b/proposals/0165-async-vote-execution.md
index 42614d144..4451fc02b 100644
--- a/proposals/0165-async-vote-execution.md
+++ b/proposals/0165-async-vote-execution.md
@@ -131,10 +131,9 @@ dead later. Currently a block can be dead for the following reasons:
 
 1. Unable to load data from blockstore
 2. Invalid block (wrong number of ticks, duplicate block, bad last fec, etc)
-3. Error while performing set root on the new root
-4. Invalid transaction
+3. Invalid transaction
 
-For the first three, the same check can be performed computing vote only
+For the first two, the same check can be performed computing vote only
 hash. We will add a new check that the new root must be vote only replayed
 and fully replayed, this may mean the tower has more than 32 slots
 occasionally.

From 7b3de131516fc57908cd1eaba20cde9b7aacbe6c Mon Sep 17 00:00:00 2001
From: Wen <113942165+wen-coding@users.noreply.github.com>
Date: Thu, 12 Dec 2024 13:19:18 -0800
Subject: [PATCH 22/24] Add the user visible changes section.

---
 proposals/0165-async-vote-execution.md | 28 ++++++++++++++++++++++++++
 1 file changed, 28 insertions(+)

diff --git a/proposals/0165-async-vote-execution.md b/proposals/0165-async-vote-execution.md
index 4451fc02b..c86b5bc37 100644
--- a/proposals/0165-async-vote-execution.md
+++ b/proposals/0165-async-vote-execution.md
@@ -203,6 +203,34 @@ the confirmed `Vote only bankhash` (currently proposed at 1/2 of the Epoch)
 the same. This is potentially an event worthy of cluster restart. If more than
 1/3 of the validators claim a different `Replay tip bankhash`, halt and exit.
 
+### User visible changes
+
+Because we confirm or finalize blocks based on `Vote only bankhash`, the
+following changes will be visible to users:
+
+1. New RPC Commitmment Levels:
+
+Right now we have 3 commitmment levels users can specify in RPC:
+Processed/Confirmed/Finalized. These commitmment will be calculated based on
+`Vote only bankhash`. There will be two additional commitmment levels:
+
+* ReplayTipConfirmed: The highest slot where supermajority of the cluster
+voted on with the same `Replay Tip Bankhash`. Votes with invalid fee payers
+still count toward this confirmation level.
+* ReplayTipFinalized: The highest slot where the block is Finalized and
+ReplayTipConfirmed, recognized by a supermajority of the cluster.
+
+2. Feature activation:
+
+Feature activations where the vote program isn't affected still work as
+before. Feature activations where vote program is affected will require
+two epochs to activate. When a feature affecting vote program is activated
+across block boundary, we can be sure the feature is activated only when
+the first block in the epoch is fully replayed and confirmed. Because the
+`Replay tip` block is never more than one Epoch away from `Vote only tip`,
+it's safe to assume vote program related feature is activated after one
+full epoch.
+
 ## Impact
 
 Since we will eliminate the impact of non-vote transaction execution speed,

From 8ac71802605deecc45f38d26e2575289605f2b0d Mon Sep 17 00:00:00 2001
From: Wen <113942165+wen-coding@users.noreply.github.com>
Date: Thu, 12 Dec 2024 14:03:57 -0800
Subject: [PATCH 23/24] Clarify we only use vote only hash in fork selection.

---
 proposals/0165-async-vote-execution.md | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/proposals/0165-async-vote-execution.md b/proposals/0165-async-vote-execution.md
index c86b5bc37..570f139bb 100644
--- a/proposals/0165-async-vote-execution.md
+++ b/proposals/0165-async-vote-execution.md
@@ -191,7 +191,10 @@ everyone receives the same block.
 transactions without checking the fee payers. The result is immediately
 applied in consensus to select a fork. Then votes are sent out for the
 selected fork with the `Vote only bankhash` for the tip of the fork and the
-most recent `Replay tip bankhash`.
+most recent `Replay tip bankhash`. Note that the fork selection will only
+be picked based on most recent `Vote only bankhash` and associated slot.
+`Replay tip bankhash` is used mostly for commitmment aggregation and security
+checks described below.
 3. The blocks on the selected forks are scheduled to be replayed. When
 a block is replayed, all transactions are executed with fee payers checked.
 This is the same as the replay we use today.

From 437d66db20413a8de3a087bcacb69447756840e0 Mon Sep 17 00:00:00 2001
From: Wen <113942165+wen-coding@users.noreply.github.com>
Date: Thu, 12 Dec 2024 14:43:14 -0800
Subject: [PATCH 24/24] EpochSlots should be updated when banks are vote only
 frozen.

---
 proposals/0165-async-vote-execution.md | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/proposals/0165-async-vote-execution.md b/proposals/0165-async-vote-execution.md
index 570f139bb..cccaeda74 100644
--- a/proposals/0165-async-vote-execution.md
+++ b/proposals/0165-async-vote-execution.md
@@ -193,8 +193,9 @@ applied in consensus to select a fork. Then votes are sent out for the
 selected fork with the `Vote only bankhash` for the tip of the fork and the
 most recent `Replay tip bankhash`. Note that the fork selection will only
 be picked based on most recent `Vote only bankhash` and associated slot.
-`Replay tip bankhash` is used mostly for commitmment aggregation and security
-checks described below.
+The EpochSlots will also be updated when the banks have completed vote
+only replay. `Replay tip bankhash` is used mostly for commitmment aggregation
+and security checks described below.
 3. The blocks on the selected forks are scheduled to be replayed. When
 a block is replayed, all transactions are executed with fee payers checked.
 This is the same as the replay we use today.