From 11bd20bbff5cdd80ab9f88480a2fff359a2f19db Mon Sep 17 00:00:00 2001 From: "Dr. Ernie Prabhakar" Date: Fri, 23 Feb 2024 03:20:06 -0800 Subject: [PATCH] Add Amazon S3's parallel SHA2-256 (#343) When data is uploaded to S3 in can be chunked. Each chunk is hashed, those hashes are then concatenated and hashes again into a single hash. Quilt is using 8MiB chunks, but increases the size in case the input is large, as the maximum size of chunks is 10k. For all details about the hash algorithm see: https://github.com/quiltdata/quilt/blob/fb36269def0f8b811e0c1293fcf3e853732a7b71/docs/sha2-256-chunked.md --- table.csv | 1 + 1 file changed, 1 insertion(+) diff --git a/table.csv b/table.csv index 1eee53f7..1c580c1e 100644 --- a/table.csv +++ b/table.csv @@ -510,6 +510,7 @@ poseidon-bls12_381-a2-fc1-sc, multihash, 0xb402, draft, Pose rdfc-1, ipld, 0xb403, draft, The result of canonicalizing an input according to RDFC-1.0 and then expressing its hash value as a multihash value. ssz, serialization, 0xb501, draft, SimpleSerialize (SSZ) serialization ssz-sha2-256-bmt, multihash, 0xb502, draft, SSZ Merkle tree root using SHA2-256 as the hashing function and SSZ serialization for the block binary +sha2-256-chunked, multihash, 0xb510, draft, Hash of concatenated SHA2-256 digests of 8*2^n MiB source chunks; n = ceil(log2(source_size/(10^4 * 8MiB))) json-jcs, ipld, 0xb601, draft, The result of canonicalizing an input according to JCS - JSON Canonicalisation Scheme (RFC 8785) iscc, softhash, 0xcc01, draft, ISCC (International Standard Content Code) - similarity preserving hash zeroxcert-imprint-256, zeroxcert, 0xce11, draft, 0xcert Asset Imprint (root hash)