Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(validator): add alert for reorg #5070

Draft
wants to merge 2 commits into
base: main
Choose a base branch
from
Draft

feat(validator): add alert for reorg #5070

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
d2dc566
add merkle_root_mismatch
aroralanuk Dec 23, 2024
2f13348
fmt
aroralanuk Dec 23, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 14 additions & 3 deletions rust/main/agents/validator/src/submit.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -213,14 +213,17 @@ impl ValidatorSubmitter {
chrono::Utc::now().timestamp() as u64,
self.reorg_period.clone(),
);

self.metrics.merkle_root_mismatch.set(1);

error!(
?checkpoint,
?correctness_checkpoint,
?reorg_event,
"Incorrect tree root, something went wrong"
"CRITICAL SAFETY ISSUE: Merkle root mismatch detected - potential chain fork or invalid message"
);

let mut panic_message = "Incorrect tree root, something went wrong.".to_owned();
let mut panic_message = "CRITICAL SAFETY ISSUE: Merkle root mismatch detected. This could indicate a chain fork or invalid message. Immediate investigation required.".to_owned();
if let Err(e) = self
.checkpoint_syncer
.write_reorg_status(&reorg_event)
Expand Down Expand Up @@ -318,6 +321,7 @@ fn tree_exceeds_checkpoint(checkpoint: &Checkpoint, tree: &IncrementalMerkle) ->
pub(crate) struct ValidatorSubmitterMetrics {
latest_checkpoint_observed: IntGauge,
latest_checkpoint_processed: IntGauge,
merkle_root_mismatch: IntGauge,
}

impl ValidatorSubmitterMetrics {
Expand All @@ -330,6 +334,11 @@ impl ValidatorSubmitterMetrics {
latest_checkpoint_processed: metrics
.latest_checkpoint()
.with_label_values(&["validator_processed", chain_name]),
merkle_root_mismatch: IntGauge::new(
"merkle_root_mismatch",
"Number of times a merkle root mismatch has been detected",
)
.unwrap(),
}
}
}
Expand Down Expand Up @@ -554,7 +563,9 @@ mod test {
}

#[tokio::test]
#[should_panic(expected = "Incorrect tree root, something went wrong.")]
#[should_panic(
expected = "CRITICAL SAFETY ISSUE: Merkle root mismatch detected. This could indicate a chain fork or invalid message. Immediate investigation required."
)]
async fn reorg_is_detected_and_persisted_to_checkpoint_storage() {
let unix_timestamp = chrono::Utc::now().timestamp() as u64;
let expected_reorg_period = 12;
Expand Down
Loading