Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(code): Experimental node CLI #163

Closed
wants to merge 50 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
50 commits
Select commit Hold shift + click to select a range
2ffb3fe
Implement TCP-based broadcast network
romac Feb 6, 2024
b1287cc
Identify which peer sent a message
romac Feb 19, 2024
8d05ecb
Fix warning
romac Feb 19, 2024
abb62cc
Implement `Network` for `broadcast::Handle`
romac Feb 20, 2024
45e2f17
Remove manual impls of common trait and use `derive-where` instead
romac Feb 20, 2024
31c0ca9
First draft of Protobuf definitions for Malachite types
romac Feb 20, 2024
198709d
Cleanup
romac Feb 20, 2024
86d9390
Make network messages agnostic of `Context` by using proto definition…
romac Feb 20, 2024
d6c85ae
Refactor `Protobuf` trait
romac Feb 20, 2024
67babff
Add `Protobuf` impl for `SignedVote`
romac Feb 20, 2024
ac5f202
Cleanup
romac Feb 20, 2024
967587b
Remove Protobuf related code from `malachite-common` and into `malach…
romac Feb 20, 2024
5f06576
Refactor `Protobuf` trait once more
romac Feb 20, 2024
959e8aa
Implement skeleton for basic node
romac Feb 21, 2024
6436ff6
Simpler design for Protobuf encoding
romac Feb 21, 2024
9cacb53
Automatically clean elapsed timeouts
romac Feb 21, 2024
47581ba
Add basic node CLI with support for 3 peers
romac Feb 21, 2024
7c284aa
Allow node to propose a single fixed value
romac Feb 21, 2024
0099fd6
Cleanup
romac Feb 21, 2024
647eb67
Feed back proposal and vote into the driver
romac Feb 21, 2024
a7a6715
Cleanup
romac Feb 21, 2024
c0d3a93
Add a commit timeout
romac Feb 21, 2024
b1ad611
Only process timeouts for the current round
romac Feb 21, 2024
097aa42
Allow consensus to move across multiple heights
romac Feb 21, 2024
859e00c
Verify signature of votes and proposals
romac Feb 21, 2024
2ad08bc
Add `ValueBuilder` trait for building values
romac Feb 21, 2024
277c3d0
Install `protoc` on CI
romac Feb 21, 2024
5418db1
Remove redundant import
romac Feb 21, 2024
4915de4
Add integration for 3 non-faulty nodes
romac Feb 22, 2024
ff4359b
Fix node binary, and print each decisions
romac Feb 22, 2024
b4173d6
Exclude binary and Debug impls from coverage
romac Feb 22, 2024
378c152
Fix coverage
romac Feb 22, 2024
363b6f5
Add height to `ProposerSelector`
romac Feb 22, 2024
70a55cb
Clear prevote/precommit timeout when we get 2/3+ votes for a value
romac Feb 23, 2024
f2617c1
Track current round in span
romac Feb 23, 2024
1ee0e70
Remove `peers.toml` config, generate it on the fly
romac Feb 23, 2024
673796a
Use `turmoil` for writing broadcast test
romac Feb 22, 2024
862d140
Better error handling in broadcast network
romac Feb 26, 2024
d2a1881
Fix integration test and use 10 validators
romac Feb 26, 2024
cb1a5f9
Basic libp2p-based gossip as a standalone crate
romac Feb 26, 2024
37bc4d6
Use mDNS-based peer discovery for now
romac Feb 26, 2024
d12bf3c
Add gossip-based integration test with 10 nodes
romac Feb 26, 2024
59c80ed
Change CLI to use gossip layer
romac Feb 26, 2024
d67d73e
Fix tests on CI
romac Feb 27, 2024
c732936
Add validator address to proposals
romac Feb 27, 2024
d77f9cf
Re-add vote and proposal signature verification
romac Feb 27, 2024
6c47d84
Rotate the proposer each height and round
romac Feb 27, 2024
b442080
Less noisy logging
romac Feb 27, 2024
367f412
Use QUIC instead of TCP+Noise+Yamux
romac Mar 4, 2024
4c71de7
Add support for multiple gossip channels
romac Mar 20, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/codespell/codespell.ini
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
[codespell]
skip = ./code/target
skip = ./code/target,./code/Cargo.lock
ignore-words = .github/codespell/words.txt
6 changes: 3 additions & 3 deletions .github/codespell/words.txt
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
crate
shs
ser
numer
manuel
numer
ser
shs
26 changes: 21 additions & 5 deletions .github/workflows/coverage.yml
Original file line number Diff line number Diff line change
Expand Up @@ -25,10 +25,14 @@ jobs:
steps:
- name: Checkout
uses: actions/checkout@v4
- uses: actions/setup-node@v3
- name: Install Protoc
uses: arduino/setup-protoc@v3
- name: Setup Node
uses: actions/setup-node@v3
with:
node-version: "18"
- run: npm install -g @informalsystems/quint
- name: Install Quint
run: npm install -g @informalsystems/quint
- name: Setup Rust toolchain
uses: actions-rust-lang/setup-rust-toolchain@v1
with:
Expand All @@ -39,7 +43,15 @@ jobs:
- name: Install cargo-llvm-cov
uses: taiki-e/install-action@cargo-llvm-cov
- name: Generate code coverage
run: cargo llvm-cov nextest --workspace --exclude malachite-itf --all-features --ignore-run-fail --lcov --output-path lcov.info
run: |
cargo llvm-cov nextest \
--workspace \
--exclude malachite-itf \
--ignore-filename-regex node/bin \
--all-features \
--ignore-run-fail \
--lcov \
--output-path lcov.info
- name: Generate text report
run: cargo llvm-cov report
- name: Upload coverage to Codecov
Expand All @@ -61,10 +73,14 @@ jobs:
steps:
- name: Checkout
uses: actions/checkout@v4
- uses: actions/setup-node@v3
- name: Install Protoc
uses: arduino/setup-protoc@v3
- name: Setup Node
uses: actions/setup-node@v3
with:
node-version: "18"
- run: npm install -g @informalsystems/quint
- name: Install Quint
run: npm install -g @informalsystems/quint
- name: Setup Rust toolchain
uses: actions-rust-lang/setup-rust-toolchain@v1
with:
Expand Down
8 changes: 6 additions & 2 deletions .github/workflows/mbt.yml
Original file line number Diff line number Diff line change
Expand Up @@ -27,10 +27,14 @@ jobs:
steps:
- name: Checkout
uses: actions/checkout@v4
- uses: actions/setup-node@v3
- name: Install Protoc
uses: arduino/setup-protoc@v3
- name: Setup Node
uses: actions/setup-node@v3
with:
node-version: "18"
- run: npm install -g @informalsystems/quint
- name: Install Quint
run: npm install -g @informalsystems/quint
- name: Setup Rust toolchain
uses: actions-rust-lang/setup-rust-toolchain@v1
- name: Install cargo-nextest
Expand Down
12 changes: 9 additions & 3 deletions .github/workflows/rust.yml
Original file line number Diff line number Diff line change
Expand Up @@ -31,25 +31,31 @@ jobs:
steps:
- name: Checkout
uses: actions/checkout@v4
- uses: actions/setup-node@v3
- name: Install Protoc
uses: arduino/setup-protoc@v3
- name: Setup Node
uses: actions/setup-node@v3
with:
node-version: "18"
- run: npm install -g @informalsystems/quint
- name: Install Quint
run: npm install -g @informalsystems/quint
- name: Setup Rust toolchain
uses: actions-rust-lang/setup-rust-toolchain@v1
- name: Install cargo-nextest
uses: taiki-e/install-action@cargo-nextest
- name: Build code
run: cargo nextest run --workspace --all-features --no-run
- name: Run tests
run: cargo nextest run --workspace --all-features
run: cargo nextest run --workspace --all-features --no-capture

clippy:
name: Clippy
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Install Protoc
uses: arduino/setup-protoc@v3
- name: Setup Rust toolchain
uses: actions-rust-lang/setup-rust-toolchain@v1
with:
Expand Down
53 changes: 38 additions & 15 deletions code/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,11 @@ resolver = "2"

members = [
"common",
"driver",
"driver",
"gossip",
"itf",
"node",
"proto",
"round",
"test",
"vote",
Expand All @@ -18,17 +21,37 @@ license = "Apache-2.0"
publish = false

[workspace.dependencies]
derive-where = "1.2.7"
ed25519-consensus = "2.1.0"
futures = "0.3"
glob = "0.3.0"
itf = "0.2.2"
num-bigint = "0.4.4"
num-traits = "0.2.17"
pretty_assertions = "1.4"
rand = { version = "0.8.5", features = ["std_rng"] }
serde = "1.0"
serde_json = "1.0"
serde_with = "3.4"
sha2 = "0.10.8"
signature = "2.1.0"
malachite-common = { version = "0.1.0", path = "common" }
malachite-driver = { version = "0.1.0", path = "driver" }
malachite-gossip = { version = "0.1.0", path = "gossip" }
malachite-itf = { version = "0.1.0", path = "itf" }
malachite-node = { version = "0.1.0", path = "node" }
malachite-proto = { version = "0.1.0", path = "proto" }
malachite-round = { version = "0.1.0", path = "round" }
malachite-test = { version = "0.1.0", path = "test" }
malachite-vote = { version = "0.1.0", path = "vote" }

async-trait = "0.1.77"
derive-where = "1.2.7"
ed25519-consensus = "2.1.0"
futures = "0.3"
glob = "0.3.0"
itf = "0.2.2"
num-bigint = "0.4.4"
num-traits = "0.2.17"
pretty_assertions = "1.4"
prost = "0.12.3"
prost-types = "0.12.3"
prost-build = "0.12.3"
rand = "0.8.5"
serde = "1.0"
serde_json = "1.0"
serde_with = "3.4"
sha2 = "0.10.8"
signature = "2.1.0"
thiserror = "1.0"
tokio = "1.35.1"
tokio-stream = "0.1"
toml = "0.8.10"
tracing = "0.1.40"
tracing-subscriber = "0.3.18"
4 changes: 2 additions & 2 deletions code/common/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -9,5 +9,5 @@ license.workspace = true
publish.workspace = true

[dependencies]
derive-where.workspace = true
signature.workspace = true
derive-where.workspace = true
signature.workspace = true
19 changes: 15 additions & 4 deletions code/common/src/context.rs
Original file line number Diff line number Diff line change
@@ -1,13 +1,13 @@
use crate::{
Address, Height, NilOrVal, Proposal, PublicKey, Round, SignedVote, SigningScheme, Validator,
ValidatorSet, Value, ValueId, Vote,
Address, Height, NilOrVal, Proposal, PublicKey, Round, SignedProposal, SignedVote,
SigningScheme, Validator, ValidatorSet, Value, ValueId, Vote,
};

/// This trait allows to abstract over the various datatypes
/// that are used in the consensus engine.
pub trait Context
where
Self: Sized,
Self: Sized + Clone + Send + Sync + 'static,
{
/// The type of address of a validator.
type Address: Address;
Expand All @@ -33,7 +33,7 @@ where
/// The signing scheme used to sign votes.
type SigningScheme: SigningScheme;

/// Sign the given vote our private key.
/// Sign the given vote with our private key.
fn sign_vote(&self, vote: Self::Vote) -> SignedVote<Self>;

/// Verify the given vote's signature using the given public key.
Expand All @@ -43,12 +43,23 @@ where
public_key: &PublicKey<Self>,
) -> bool;

/// Sign the given proposal with our private key.
fn sign_proposal(&self, proposal: Self::Proposal) -> SignedProposal<Self>;

/// Verify the given proposal's signature using the given public key.
fn verify_signed_proposal(
&self,
signed_proposal: &SignedProposal<Self>,
public_key: &PublicKey<Self>,
) -> bool;

/// Build a new proposal for the given value at the given height, round and POL round.
fn new_proposal(
height: Self::Height,
round: Round,
value: Self::Value,
pol_round: Round,
address: Self::Address,
) -> Self::Proposal;

/// Build a new prevote vote by the validator with the given address,
Expand Down
4 changes: 3 additions & 1 deletion code/common/src/height.rs
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,8 @@ use core::fmt::Debug;
/// A height of 0 represents a chain which has not yet produced a block.
pub trait Height
where
Self: Default + Copy + Clone + Debug + PartialEq + Eq + PartialOrd + Ord,
Self: Default + Copy + Clone + Debug + PartialEq + Eq + PartialOrd + Ord + Send + Sync,
{
/// Increment the height by one.
fn increment(&self) -> Self;
}
4 changes: 4 additions & 0 deletions code/common/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -12,10 +12,13 @@
#![cfg_attr(not(test), deny(clippy::unwrap_used, clippy::panic))]
#![cfg_attr(coverage_nightly, feature(coverage_attribute))]

extern crate alloc;

mod context;
mod height;
mod proposal;
mod round;
mod signed_proposal;
mod signed_vote;
mod signing;
mod timeout;
Expand All @@ -42,6 +45,7 @@ pub use context::Context;
pub use height::Height;
pub use proposal::Proposal;
pub use round::Round;
pub use signed_proposal::SignedProposal;
pub use signed_vote::SignedVote;
pub use signing::SigningScheme;
pub use timeout::{Timeout, TimeoutStep};
Expand Down
5 changes: 4 additions & 1 deletion code/common/src/proposal.rs
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ use crate::{Context, Round};
/// Defines the requirements for a proposal type.
pub trait Proposal<Ctx>
where
Self: Clone + Debug + PartialEq + Eq,
Self: Clone + Debug + Eq + Send + Sync + 'static,
Ctx: Context,
{
/// The height for which the proposal is for.
Expand All @@ -19,4 +19,7 @@ where

/// The POL round for which the proposal is for.
fn pol_round(&self) -> Round;

/// Address of the validator who issued this proposal
fn validator_address(&self) -> &Ctx::Address;
}
8 changes: 7 additions & 1 deletion code/common/src/round.rs
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
use core::cmp;
use core::{cmp, fmt};

/// A round number.
///
Expand Down Expand Up @@ -72,6 +72,12 @@ impl Ord for Round {
}
}

impl fmt::Display for Round {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
self.as_i64().fmt(f)
}
}

#[cfg(test)]
mod tests {
use super::*;
Expand Down
29 changes: 29 additions & 0 deletions code/common/src/signed_proposal.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
use derive_where::derive_where;

use crate::{Context, Signature};

/// A signed proposal, ie. a proposal emitted by a validator and signed by its private key.
#[derive_where(Clone, Debug, PartialEq, Eq)]
pub struct SignedProposal<Ctx>
where
Ctx: Context,
{
/// The proposal.
pub proposal: Ctx::Proposal,

/// The signature of the proposal.
pub signature: Signature<Ctx>,
}

impl<Ctx> SignedProposal<Ctx>
where
Ctx: Context,
{
/// Create a new signed proposal from the given proposal and signature.
pub fn new(proposal: Ctx::Proposal, signature: Signature<Ctx>) -> Self {
Self {
proposal,
signature,
}
}
}
22 changes: 18 additions & 4 deletions code/common/src/signing.rs
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
use core::fmt::Debug;
use alloc::vec::Vec;
use core::fmt::{Debug, Display};

use signature::{Keypair, Signer, Verifier};

Expand All @@ -14,12 +15,25 @@ pub trait SigningScheme
where
Self: Clone + Debug + Eq,
{
/// Errors that can occur when decoding a signature from a byte array.
type DecodingError: Display;

/// The type of signatures produced by this signing scheme.
type Signature: Clone + Debug + Eq;
type Signature: Clone + Debug + Eq + Send + Sync;

/// The type of public keys produced by this signing scheme.
type PublicKey: Clone + Debug + Eq + Verifier<Self::Signature>;
type PublicKey: Clone + Debug + Eq + Send + Sync + Verifier<Self::Signature>;

/// The type of private keys produced by this signing scheme.
type PrivateKey: Clone + Signer<Self::Signature> + Keypair<VerifyingKey = Self::PublicKey>;
type PrivateKey: Clone
+ Send
+ Sync
+ Signer<Self::Signature>
+ Keypair<VerifyingKey = Self::PublicKey>;

/// Decode a signature from a byte array.
fn decode_signature(bytes: &[u8]) -> Result<Self::Signature, Self::DecodingError>;

/// Encode a signature to a byte array.
fn encode_signature(signature: &Self::Signature) -> Vec<u8>;
}
Loading
Loading