Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

style!: issue15 - cleaner polynomial interface #21

Merged
merged 22 commits into from
Jan 8, 2025
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
5d3ec00
style!: issue15 - cleaner polynomial interface
samlaf Jan 4, 2025
b07e0a0
refactor!: change kzg commit functions to take a specific form
samlaf Jan 6, 2025
339d309
test: fix tests after new breaking api changes
samlaf Jan 6, 2025
e8ad64e
style!: rename commit_coef_form -> commit_coeff_form
samlaf Jan 6, 2025
98139b4
test: fix benchmarks following api changes
samlaf Jan 6, 2025
d88faf5
style: fix lint issues
samlaf Jan 6, 2025
f61a433
fix: removed polymorphism on commit function
samlaf Jan 6, 2025
965fea8
docs: add README section about kzg commitment
samlaf Jan 6, 2025
5933b01
style: cargo fmt
samlaf Jan 6, 2025
0ed4496
docs: fix commitment explanation in readme
samlaf Jan 7, 2025
09444ff
docs: add doc comment for fn convert_by_padding_empty_byte
samlaf Jan 7, 2025
464ac34
docs: make read_file_chunks comment more precise
samlaf Jan 7, 2025
88eb5ff
docs: add "monomial" prefix to srs points in comment
samlaf Jan 7, 2025
c4c2e91
style: rename function to_coef_form -> to_coeff_form for consistency
samlaf Jan 7, 2025
123578b
docs: clarify some comments
samlaf Jan 7, 2025
5ee952e
docs: clean up and make much more precise data type documentation + r…
samlaf Jan 7, 2025
c3653a8
style: switch poly eval and coeff forms to have eval at top of file
samlaf Jan 7, 2025
e61ddcf
test: add test to check length of polynomials during conversions
samlaf Jan 7, 2025
194b32b
docs: move readme docs to lib.rs rust docs
samlaf Jan 7, 2025
8ca7620
docs: fix doc code that was failing
samlaf Jan 7, 2025
54626b1
style: cargo fmt
samlaf Jan 8, 2025
5c8e0ac
test: fix cargo doc test
samlaf Jan 8, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
42 changes: 24 additions & 18 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,9 +19,6 @@ This code is unaudited and under construction. This is experimental software and
2. Specify the files in `kzg.setup()` function, leave the `g2_points` empty, and specify the `srs_order` per the guide.
3. Note that this is process will take a few minutes to load since it is a bit intensive.

## Clippy
Linting can be triggered via running `cargo clippy --all --manifest-path Cargo.toml -- -D warnings`.

## Quick Start

See the `test_compute_kzg_proof` function in [./tests/kzg_test.rs](./tests/kzg_test.rs#) for an end to end usage of the library.
Expand All @@ -32,30 +29,39 @@ See the `test_compute_kzg_proof` function in [./tests/kzg_test.rs](./tests/kzg_t
2. Committing is performed in Lagrange format. The required IFFT is done within the function and is not required to be performed separately.
3. For proof generation, the data is treated as evaluation of polynomial. The required (I)FFT is performed by the compute function and is not required to be performed separately.

## Function Reference

### `to_polynomial_coeff_form()` / `to_polynomial_eval_form()`

From the `Blob`, a polynomial can be obtained via calling the `to_polynomial()` function. This converts the Blob to Field elements, then calculates the next power of 2 from this length of field elements and appends `zero` value elements for the remaining length.
## Library Design / Architecture

### `data_setup_custom` and `data_setup_mins` parameters
The main purpose of this library is to allow taking a piece of data, committing to it, and then generating and verifying proofs against that commitment.

The `data_setup_custom` (for testing) or `data_setup_mins` should be used to specify the number of chunks and chunk length. These parameters are used to calculate the FFT params required for FFT operations.
### Data Types

### `commit()`
The main data pipeline goes:
> user data -> Blob -> Polynomial -> KZG Commitment / Proof

The `commit` function takes in a `polynomial`. It is computed over `lagrange` basis by performing the (i)FFT depending on the `polynomial` form specified.
- User Data: bytes array
- meaningful to users (typically will be a rollup batch)
- Blob: bn254 field elements array
- meaningful to EigenDA network
- Obtained from User Data by inserting zeroes every 31 bytes to make every 32 byte an element of bn254.
- Polynomial: bn254 field elements array, interpreted as coefficients or evaluations of a polynomial
- meaningful when committing and generating/verifying proofs
- Obtained from Blob by appending zeroes to make the length a power of 2, and then interpreting the array as coefficients or evaluations of a polynomial.
- Kzg: struct storing the SRS points used to generate commitments and proofs
- SRS points: bn254 group elements
bxue-l2 marked this conversation as resolved.
Show resolved Hide resolved
- inner producted with the polynomial to generate commitments

The [Blob](./src/blob.rs) and [Polynomial](./src/polynomial.rs) structs are mostly [Plain Old Data](https://en.wikipedia.org/wiki/Passive_data_structure) with constructor and few helper methods. The interesting stuff happens in the [KZG](./src/kzg.rs) struct, which has methods for committing to a blob, polynomial in coeff or eval form, and generating and verifying proofs.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

make sure to note that this blob is different from the blob defintion from the eigenda_client. I would get super confused by it
https://github.com/Layr-Labs/eigenda/blob/master/api/clients/eigenda_client.go#L167

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is too deep in the weeds. Nobody will know about this (for now). Let's just fix that name and not mention it here. Let's try to keep the docs here self-contained.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just say not to confuse with the others. Then once the interface update, we ca update here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But we will just forget to update here and then confuse people even more. Let's not confuse them with problems they don't have haha.


### `compute_proof_with_roots_of_unity()`
Our current codebase has the types PolynomialEvalForm and PolynomialCoeffForm to represent the polynomial in evaluation and coefficient form respectively. However, we do not have types to represent the two forms of srs points. They are implicitly assumed to be in monomial form when loaded, and an IFFT is performed before taking the inner product with the polynomial in evaluation form.

The `compute_proof_with_roots_of_unity` takes in a `Polynomial` and an `index` at which it needs to be computed.
### KZG Commitments

## KZG Commitments

Below diagram explains the difference types involved between polynomials, SRS points, and kzg commitments.
Below diagram explains the different types involved between polynomials, SRS points, and KZG commitments.
A KZG commitment can be taken by an inner product between (poly_coeff, srs_monomial) or (poly_eval, srs_lagrange). FFT and IFFT operations can be performed to convert between these forms.

![KZG Commitments](./kzg_commitment_diagram.png)

Our current codebase has the types PolynomialEvalForm and PolynomialCoeffForm to represent the polynomial in evaluation and coefficient form respectively. However, we do not have types to represent the two forms of srs points. They are implicitly assumed to be in monomial form when loaded, and an IFFT is performed before taking the inner product with the polynomial in evaluation form.

### KZG Proofs

TODO: Add diagram for KZG Proofs
4 changes: 2 additions & 2 deletions src/blob.rs
Original file line number Diff line number Diff line change
Expand Up @@ -58,13 +58,13 @@ impl Blob {
self.blob_data.is_empty()
}

/// Convert the blob data to a `PolynomialEvalForm`.
/// Convert the blob data to a [PolynomialEvalForm].
pub fn to_polynomial_eval_form(&self) -> PolynomialEvalForm {
let fr_vec = helpers::to_fr_array(&self.blob_data);
PolynomialEvalForm::new(fr_vec)
}

/// Convert the blob data to a `PolynomialCoefForm`.
/// Convert the blob data to a [PolynomialCoeffForm].
pub fn to_polynomial_coeff_form(&self) -> PolynomialCoeffForm {
let fr_vec = helpers::to_fr_array(&self.blob_data);
PolynomialCoeffForm::new(fr_vec)
Expand Down
18 changes: 12 additions & 6 deletions src/kzg.rs
Original file line number Diff line number Diff line change
Expand Up @@ -104,7 +104,9 @@ impl Kzg {
Ok(chunks)
}

/// data_setup_custom is a helper function
/// Similar to [Kzg::data_setup_mins], but mainly used for setting up Kzg for testing purposes.
/// Used to specify the number of chunks and chunk length.
/// These parameters are then used to calculate the FFT params required for FFT operations.
pub fn data_setup_custom(
&mut self,
num_of_nodes: u64,
Expand All @@ -117,7 +119,8 @@ impl Kzg {
self.data_setup_mins(min_num_chunks, num_of_nodes)
}

/// data_setup_mins sets up the environment per the blob data
/// Used to specify the number of chunks and chunk length.
/// These parameters are then used to calculate the FFT params required for FFT operations.
pub fn data_setup_mins(
&mut self,
min_chunk_length: u64,
Expand Down Expand Up @@ -470,7 +473,7 @@ impl Kzg {
self.g2.to_vec()
}

/// commit the actual polynomial with the values setup
/// Commit the polynomial with the srs values loaded into [Kzg].
pub fn commit_eval_form(&self, polynomial: &PolynomialEvalForm) -> Result<G1Affine, KzgError> {
if polynomial.len() > self.g1.len() {
return Err(KzgError::SerializationError(
Expand All @@ -488,7 +491,7 @@ impl Kzg {
}
}

/// commit the actual polynomial with the values setup
/// Commit the polynomial with the srs values loaded into [Kzg].
pub fn commit_coeff_form(
&self,
polynomial: &PolynomialCoeffForm,
Expand All @@ -508,12 +511,15 @@ impl Kzg {
}
}

pub fn blob_to_kzg_commitment(&self, blob: &Blob) -> Result<G1Affine, KzgError> {
/// commit to a [Blob], by transforming it into a [PolynomialEvalForm] and then
/// calling [Kzg::commit_eval_form].
pub fn commit_blob(&self, blob: &Blob) -> Result<G1Affine, KzgError> {
let polynomial = blob.to_polynomial_eval_form();
self.commit_eval_form(&polynomial)
}

/// helper function to work with the library and the env of the kzg instance
/// Wrapper around [Kzg::compute_proof] that uses the roots of unity
/// that were expanded and stored in [Kzg::setup].
pub fn compute_proof_with_roots_of_unity(
&self,
polynomial: &PolynomialEvalForm,
Expand Down
19 changes: 16 additions & 3 deletions src/polynomial.rs
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,10 @@ pub struct PolynomialCoeffForm {
}

impl PolynomialCoeffForm {
/// Creates a new [PolynomialCoeffForm] from the given coefficients, passed as a vector of `Fr`.
/// The coefficients are padded to the next power of two by appending zeros.
/// This typically wouldn't be used directly, but instead a [crate::blob::Blob] would be
/// converted to a [PolynomialCoeffForm] using [crate::blob::Blob::to_polynomial_coeff_form].
pub fn new(coeffs: Vec<Fr>) -> Self {
let underlying_blob_len_in_bytes = coeffs.len() * BYTES_PER_FIELD_ELEMENT;
let next_power_of_two = coeffs.len().next_power_of_two();
Expand Down Expand Up @@ -71,6 +75,8 @@ impl PolynomialCoeffForm {
helpers::to_byte_array(&self.coeffs, self.len_underlying_blob_bytes)
}

/// Converts the polynomial to evaluation form. This is done by performing
/// an FFT on the coefficients.
pub fn to_eval_form(&self) -> Result<PolynomialEvalForm, PolynomialError> {
let evals = GeneralEvaluationDomain::<Fr>::new(self.len())
.ok_or(PolynomialError::FFTError(
Expand All @@ -86,7 +92,10 @@ pub struct PolynomialEvalForm {
/// evaluations contains the evaluations of the polynomial, padded with 0s
/// to the next power of two. Hence if the polynomial is created with
/// coefficients [1, 2, 3], the internal representation will be [1, 2,
/// 3, 0].
/// 3, 0]. Note that this changes the polynomial! This is an inconsistency in our
/// current representations. Polynomials are the objects that get committed, not the
/// underlying Blobs.
/// TODO: do we also want to force blobs to be of powers-of-two length?
evaluations: Vec<Fr>,
/// Number of bytes in the underlying blob, which was used to create the
/// polynomial. This is passed as is when converting between Coefficient
Expand All @@ -101,12 +110,14 @@ pub struct PolynomialEvalForm {
}

impl PolynomialEvalForm {
/// Creates a new [PolynomialEvalForm] from the given coefficients, passed as a vector of `Fr`.
/// The coefficients are padded to the next power of two by appending zeros.
/// This typically wouldn't be used directly, but instead a [crate::blob::Blob] would be
/// converted to a [PolynomialEvalForm] using [crate::blob::Blob::to_polynomial_eval_form].
pub fn new(evals: Vec<Fr>) -> Self {
let underlying_blob_len_in_bytes = evals.len() * BYTES_PER_FIELD_ELEMENT;
let next_power_of_two = evals.len().next_power_of_two();
let mut padded_evals = evals;
// TODO: does it make sense to pad evaluations with zeros? This changes the
// polynomial...
padded_evals.resize(next_power_of_two, Fr::zero());
Copy link
Collaborator

@anupsv anupsv Jan 7, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If im understanding this right, when we call to_eval_form(), the Fr field size is changed to the next power of 2 and then padded with the "0" field element right ?

Thinking through, if someone calls to_eval or to_coeff this might happen ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not the Fr field size, but the size of the polynomial (number of evaluation points) is extended to the next power of 2 usings 0s yes. See the comment here

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So if i have [1, 2, 3] in coeff, it gets padded to [1,2,3,0] and when to_eval() is called, does it do [1, 2, 3, 0, 0, 0, 0, 0] ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

correct because we use https://doc.rust-lang.org/std/primitive.usize.html#method.next_power_of_two

Returns the smallest power of two greater than or equal to self.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added test: e61ddcf

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see. I'm thinking we shouldn't pad extra 0's when calling to_eval() or to_coeff() right ? The fft or ifft will return the same amount of elements and we should be using only those after conversion ?
Or is it the understanding that since it's 0's the commitment will remain the same and the caller will only parse the number of bytes as specified by the header byte, which contains the length, attached by the proxy ?

Copy link
Collaborator

@bxue-l2 bxue-l2 Jan 8, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, that is what we are doing in hokulea, we get the entire blob and just read the data based on the header that encodes how much is the user data
https://github.com/Layr-Labs/hokulea/blob/master/crates/eigenda/src/eigenda_data.rs#L41

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah I see. Then let's just add a comment in the code to capture this behaviour and be done ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see. I'm thinking we shouldn't pad extra 0's when calling to_eval() or to_coeff() right ? The fft or ifft will return the same amount of elements and we should be using only those after conversion ?

it won't pad extra 0s, because the evals/coeffs are already a power of 2, so that won't change.

Or is it the understanding that since it's 0's the commitment will remain the same and the caller will only parse the number of bytes as specified by the header byte, which contains the length, attached by the proxy ?

not sure I understand what you mean here


Self {
Expand Down Expand Up @@ -151,6 +162,8 @@ impl PolynomialEvalForm {
helpers::to_byte_array(&self.evaluations, self.len_underlying_blob_bytes)
}

/// Converts the polynomial to coefficient form. This is done by performing
/// an IFFT on the evaluations.
pub fn to_coeff_form(&self) -> Result<PolynomialCoeffForm, PolynomialError> {
let coeffs = GeneralEvaluationDomain::<Fr>::new(self.len())
.ok_or(PolynomialError::FFTError(
Expand Down
2 changes: 1 addition & 1 deletion tests/kzg_test.rs
Original file line number Diff line number Diff line change
Expand Up @@ -193,7 +193,7 @@ mod tests {
use ark_bn254::Fq;

let blob = Blob::from_raw_data(GETTYSBURG_ADDRESS_BYTES);
let fn_output = KZG_3000.blob_to_kzg_commitment(&blob).unwrap();
let fn_output = KZG_3000.commit_blob(&blob).unwrap();
let commitment_from_da = G1Affine::new_unchecked(
Fq::from_str(
"2961155957874067312593973807786254905069537311739090798303675273531563528369",
Expand Down
Loading