You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
As we can see from the logs, the event is "unvoiding" the transaction 0000407a55615a5dc909d534365d36c226d98f3ba2e0210d96f2d606480fde81
The way we're handling "unvoided" transaction events is to basically remove all traces (and subtracting or adding balances from all involved addresses) and then re-adding the transaction by using the data that came in the event
As we can see on line 7, the service removed the transaction successfully and then transitioned to handlingVertexAccepted to re-add it to database
When it was trying to add it (on line 10), we got a balance error because the resulting unlocked_balance was lower than 0
Querying the database, we can see that indeed the balance is indeed zero after removing all traces from the transaction ***:
So when 0000407a55615a5dc909d534365d36c226d98f3ba2e0210d96f2d606480fde81 attempts to spend 5 HTR from it, it will yield exactly the error from our logs
Why?
I queried the address_tx_history for this address to check the history of transactions for this address:
As we can see, there are only two transactions for this address, it receives the utxo spent on 0000407a55615a5dc909d534365d36c226d98f3ba2e0210d96f2d606480fde81 adding 5 HTR to the balance and spends it later on on 0000076da38a8a9077b84e472dbaf9cc2c1d1da6903916b57ad3b142f3653c52, removing 5 HTR from it
I checked this second transaction on the fullnode, and found out that it spends exactly the same utxo that our unspent tx spends, causing a double-spend, but has a smaller acc weight (63.89 vs over 65.12)
So our unspent transaction should be the one on our database and this second one should have been voided
Why?
I checked the event history manually by using the HTTP API starting at one event before our event (5760219) and apparently what happened was an event ordering issue
I still need to download the entire event database to understand the complete chronology, but apparently what happened here was this, in a chronological way:
Transaction 0000407a55615a5dc909d534365d36c226d98f3ba2e0210d96f2d606480fde81 gets inserted into database, spending funds from HBtTjEvTeL5QN5JfvDTShC6SxaoKsvZRZh on event XXX
Transaction 0000407a55615a5dc909d534365d36c226d98f3ba2e0210d96f2d606480fde81 gets voided, re-adding funds to HBtTjEvTeL5QN5JfvDTShC6SxaoKsvZRZh on event XXX
Transaction 0000076da38a8a9077b84e472dbaf9cc2c1d1da6903916b57ad3b142f3653c52 gets inserted into database, spending funds from HBtTjEvTeL5QN5JfvDTShC6SxaoKsvZRZh
This is the state of the database before our event (5760219) arrives, now that's what happens:
Event 5760219 arrives, "unvoiding" transaction 0000407a55615a5dc909d534365d36c226d98f3ba2e0210d96f2d606480fde81, causing it to spend funds from HBtTjEvTeL5QN5JfvDTShC6SxaoKsvZRZh, but since 0000076da38a8a9077b84e472dbaf9cc2c1d1da6903916b57ad3b142f3653c52 was not yet voided, the address has a zero balance and thus causing our invalid balance error
Following on the events using the HTTP API, we can see that the very next event (5760220) voids the transaction 0000076da38a8a9077b84e472dbaf9cc2c1d1da6903916b57ad3b142f3653c52
So this is an ordering issue, the sync daemon by itself apparently is unable to handle this event order, we need to think on a solution in the fullnode
***: This is an issue, we should have rolled back the entire operation, this should also be fixed
Solution
This was fixed by @glevco in this PR by changing the event ordering to send the voided transaction first
The text was updated successfully, but these errors were encountered:
Summary
Sync stopped again while handling event
5760219
with the following logs (numbered by me):Five-whys
Why?
As we can see from the logs, the event is "unvoiding" the transaction
0000407a55615a5dc909d534365d36c226d98f3ba2e0210d96f2d606480fde81
The way we're handling "unvoided" transaction events is to basically remove all traces (and subtracting or adding balances from all involved addresses) and then re-adding the transaction by using the data that came in the event
As we can see on line 7, the service removed the transaction successfully and then transitioned to
handlingVertexAccepted
to re-add it to databaseWhen it was trying to add it (on line 10), we got a balance error because the resulting
unlocked_balance
was lower than 0Querying the database, we can see that indeed the balance is indeed zero after removing all traces from the transaction ***:
So when
0000407a55615a5dc909d534365d36c226d98f3ba2e0210d96f2d606480fde81
attempts to spend 5 HTR from it, it will yield exactly the error from our logsWhy?
I queried the
address_tx_history
for this address to check the history of transactions for this address:As we can see, there are only two transactions for this address, it receives the utxo spent on
0000407a55615a5dc909d534365d36c226d98f3ba2e0210d96f2d606480fde81
adding 5 HTR to the balance and spends it later on on0000076da38a8a9077b84e472dbaf9cc2c1d1da6903916b57ad3b142f3653c52
, removing 5 HTR from itI checked this second transaction on the fullnode, and found out that it spends exactly the same utxo that our unspent tx spends, causing a double-spend, but has a smaller acc weight (63.89 vs over 65.12)
So our unspent transaction should be the one on our database and this second one should have been voided
Why?
I checked the event history manually by using the HTTP API starting at one event before our event (
5760219
) and apparently what happened was an event ordering issueI still need to download the entire event database to understand the complete chronology, but apparently what happened here was this, in a chronological way:
0000407a55615a5dc909d534365d36c226d98f3ba2e0210d96f2d606480fde81
gets inserted into database, spending funds fromHBtTjEvTeL5QN5JfvDTShC6SxaoKsvZRZh
on event XXX0000407a55615a5dc909d534365d36c226d98f3ba2e0210d96f2d606480fde81
gets voided, re-adding funds toHBtTjEvTeL5QN5JfvDTShC6SxaoKsvZRZh
on event XXX0000076da38a8a9077b84e472dbaf9cc2c1d1da6903916b57ad3b142f3653c52
gets inserted into database, spending funds fromHBtTjEvTeL5QN5JfvDTShC6SxaoKsvZRZh
This is the state of the database before our event (
5760219
) arrives, now that's what happens:5760219
arrives, "unvoiding" transaction0000407a55615a5dc909d534365d36c226d98f3ba2e0210d96f2d606480fde81
, causing it to spend funds fromHBtTjEvTeL5QN5JfvDTShC6SxaoKsvZRZh
, but since0000076da38a8a9077b84e472dbaf9cc2c1d1da6903916b57ad3b142f3653c52
was not yet voided, the address has a zero balance and thus causing our invalid balance errorFollowing on the events using the HTTP API, we can see that the very next event (
5760220
) voids the transaction0000076da38a8a9077b84e472dbaf9cc2c1d1da6903916b57ad3b142f3653c52
So this is an ordering issue, the sync daemon by itself apparently is unable to handle this event order, we need to think on a solution in the fullnode
***: This is an issue, we should have rolled back the entire operation, this should also be fixed
Solution
This was fixed by @glevco in this PR by changing the event ordering to send the voided transaction first
The text was updated successfully, but these errors were encountered: