Documentation of a decentralized sequencer

warp-contracts · Dec 15, 2023 · b9cf921 · b9cf921
1 parent 671a928
commit b9cf921
Show file tree

Hide file tree

Showing 19 changed files with 937 additions and 12 deletions.
diff --git a/warp-academy-docs/docs/sdk/advanced/bundled-interaction.md b/warp-academy-docs/docs/sdk/advanced/bundled-interaction.md
@@ -200,8 +200,3 @@ query {
 ## Interaction retrieval via Warp gateway
 
 The Warp `/gateway/interactions/:id` endpoint allows to retrieve the interaction info based on its original id.
-
-## Further development
-
-1. Blind/trustless sequencing - https://github.com/warp-contracts/gateway/issues/48
-2. Sequencer decentralization - https://github.com/warp-contracts/gateway/issues/93
diff --git a/warp-academy-docs/docs/sdk/advanced/vrf.md b/warp-academy-docs/docs/sdk/advanced/vrf.md
@@ -1,5 +1,11 @@
 # Verifiable randomness
 
+:::caution
+Random values will be handled in a new way once the decentralized sequencer is launched.
+More details can be found [here](/docs/sequencer/lifecycle/ordering#random-values).
+For backward compatibility, the function `SmartWeave.vrf.randomInt(maxValue)` will still work, but it will use the new function `SmartWeave.randomNumber(maxValue)` under the hood.
+:::
+
 You have an option to use random values in contracts via Verifiable Random Functions.  
 This may be a very useful feature in gaming, dynamic NFT generation, etc.
 

diff --git a/warp-academy-docs/docs/sdk/legacy/bundled-interaction-legacy.md b/warp-academy-docs/docs/sdk/legacy/bundled-interaction-legacy.md
@@ -1,7 +1,7 @@
 # Warp Sequencer - legacy
 
 :::caution
-Please note, that you are viewing old Warp Sequencer docs. Since `warp-contracts@1.4.13` nested bundles have been introduced instead of the format described in this file. In order to read about current interaction format, please head to [Warp Sequencer](../advanced/bundled-interaction.md).
+Please note, that you are viewing old Warp Sequencer docs. Since `warp-contracts@1.4.13` nested bundles have been introduced instead of the format described in this file. In order to read about current interaction format, please head to [Warp Sequencer](../../sequencer/introduction.md).
 :::
 
 Warp Sequencer is a module that is responsible for submitting users' transactions into Arweave (via [Bundlr Network](https://bundlr.network/)).

diff --git a/warp-academy-docs/docs/sequencer/centralized.md b/warp-academy-docs/docs/sequencer/centralized.md
@@ -0,0 +1,158 @@
+# Warp Centralized Sequencer
+
+:::caution
+This section describes the operation of the current centralized sequencer, which will soon be replaced by the new [decentralized solution](/docs/sequencer/decentralized).
+:::
+
+## How it works
+
+Instead of posting the interaction transactions directly to Arweave mainnet, Warp SDK creates a data item with interaction input and tags which is then signed with user's wallet. Data item is then sent to Warp Sequencer (`/gateway/v2/sequencer/register` endpoint) (this is the default behaviour of Warp's SDK `contract.writeInteraction`
+function, when `forMainnet` instance is being used).
+
+The Warp Sequencer then:
+
+#### 1. Generates a sort key
+
+A sort key is generated from:
+
+1. current mainnet network height
+2. current sequence value
+3. original transaction id
+4. current mainnet block hash
+
+In the original SmartWeave protocol specification, a _sort key_ is defined
+as a `[ block_height,sha256(transactionId + blockHash) ]`, where:
+
+1. `block_height` - current network height, l-padded with `0` to 12 chars, e.g. for block height `1015556`, the
+   result is `000001015556`
+2. `sha256(transactionId + blockHash)` - a `sha256` hash of the concatenated buffers of the transaction id and block
+   hash,
+   e.g. for txId `UR_35HORbjjZ_NnUqinkZuWkcNB1-gBST3Rezt5JrDs` and block
+   hash `ixWCxRN36DjVUxQRa68xIeoZLfvLDTtX78e0ae8RAAJjOPpDBuVKVaEKYOpq7bLS`,
+   the result is `44edd70f2018924f22a878a558a8f2d5cae8bc1f718d567df43bf52b6384d260`.
+
+The complete _sort key_ for the above values would
+be: `000001015556,44edd70f2018924f22a878a558a8f2d5cae8bc1f718d567df43bf52b6384d260`.
+
+The generated sort keys are then used by the SmartWeave protocol to lexicographically sort the transactions.
+
+The Warp Sequencer extends this default mechanism by the current sequence value.  
+The formula for the _sort key_ is extended to:
+`[ block_height,sequence_value,sha256(transactionId + blockHash) ]`
+
+This sequence value can be obtained from the Sequencer's node timestamp, database or other sources.
+In its current implementation - a Sequencer node timestamp value is being used.  
+This in effect gives a fair transactions ordering - the transactions will have the sequence assigned in order in which
+they are processed by the Sequencer.
+
+Assuming transaction id `La_NpAFAWxGj-VIiLfg7NbBfox0RZ8uuEJSOOZykd48`, block
+hash `-o88tFYsMG9RXSGcNXX5sVDuSV5uHy7zuFRj6vYo91e3mXpmng6qw322Ip0-EguA`,
+block height `1015560` and current Sequencer value `1663069424541`, the generated _sort key_ would
+be `000001015560,1663069424541,a21ac8a60326ba8c2bb8caa05cff3334a22e9960ef55de0b5392caa30b484d0a`
+
+**NOTE** All the transactions sent to Arweave directly, have the sequence value assigned to `0000000000000000`.
+This effectively means that if transactions to a given contract are sent both directly to Arweave mainnet and Warp
+Sequencer -
+if two transactions happen to be at the same block height, the "direct" Arweave transactions take precedence.
+This also means that the sequencing algorithm is fully backwards compatible with the original SmartWeave protocol.
+
+#### 2. Generates tags for the Bundlr transaction
+
+| Tag Name                                    | Tag Value                                                            |
+| ------------------------------------------- | -------------------------------------------------------------------- |
+| `Sequencer`                                 | `Warp`                                                               |
+| `Sequencer-Owner`                           | The original owner/signar of the contract transaction                |
+| `Sequencer-Mills`                           | The sequence value used by the Sequencer for this transaction        |
+| `Sequencer-Sort-Key`                        | The generated sort key for this transaction                          |
+| `Sequencer-Prev-Sort-Key`                   | The sort key of the previous transaction                             |
+| `Sequencer-Tx-Id`                           | The original transaction id                                          |
+| `Sequencer-Block-Height`                    | The block height used for generating the sort key                    |
+| `Sequencer-Block-Id`                        | The block hash used for generating the sort key                      |
+| `Sequencer-Block-Timestamp`                 | The timestamp of the block that was used for generating the sort key |
+| ...all the tags of the original transaction |                                                                      |
+
+:::tip
+The `Sequencer-Prev-Sort-Key` tells what is the sort key of the 'previous' transaction in the sequencer and
+can be used to verify whether all transactions have been properly loaded (i.e. if one
+decides to load them directly from L1 nodes) and none is missing.
+:::
+
+Additional set of tags are added in case user requests generating a random value using VRF (Verifiable Random Function):
+
+| Tag Name     | Tag Value                                                               |
+| ------------ | ----------------------------------------------------------------------- |
+| `vrf-index`  | The original hash generated by the VRF (using `sort_key` as input data) |
+| `vrf-proof`  | The original proof generated by the VRF                                 |
+| `vrf-bigint` | A BigInt value evaluated from the hash generated by the VRF             |
+| `vrf-pubkey` | The public key used by the VRF                                          |
+
+Verifiable randomness can be used by contracts that require using random values - e.g. gaming contracts, nft/loot
+generating contracts, etc.
+Using the `sort_key`, `vrf-proof` and `vrf-pubkey`, the client can always verify the generated random value.
+
+#### 3. Wrap original data item in a bundle and uploads the bundle to Bundlr
+
+Apart from all the tags from point 2, some additional tags are added to the wrapping bundle:
+
+| Tag Name         | Tag Value         |
+| ---------------- | ----------------- |
+| `Bundle-Format`  | `binary`          |
+| `Bundle-Version` | `2.0.0`           |
+| `App-Name`       | `Warp`            |
+| `Action`         | `WarpInteraction` |
+
+In order to send original data item to Bundlr, we use the concept of nested bundles and set an interaction data item inside a bundle. If you're not faimiliar with this concept, here is a quick summary:
+
+:::info
+Bundling allows to write multiple data items into one top level transaction. A data item differs from a regular transaction by not allowing to transfer AR tokens and passing reward but it has most of the transaction's properties - such as owner, data, tags, owner and id.
+
+In a nutshell, the nested bundles concept means that a data item of a bundle can also be a bundle containg other data items. According to ANS-104 standard it can lead to theoretically unbounded levels of nesting.
+
+You can read the specification for nested bundles standard in [ANS-104](https://github.com/ArweaveTeam/arweave-standards/blob/master/ans/ANS-104.md#31-nested-bundle).
+:::
+
+:::info
+Obviously the data-item is signed by the Sequencer's wallet - and as such can be easily verified.
+The Sequencer wallet address is `jnioZFibZSCcV8o-HkBXYPYEYNib4tqfexP0kCBXX_M`.
+:::
+
+**NOTE** The original data item is not modified in any way - this is to preserve the original
+signature!
+
+After receiving proper response and receipt from Bundlr, the Warp gateway indexes the contract interaction
+internally - to make it instantly available.
+
+#### 4. Finally, the Warp gateway returns the response from the Bundlr to the client.
+
+## Interaction data item retrieval (generated by the Warp Sequencer) via Arweave gateway
+
+Use the GQL endpoint, with the original data item id passed in the `Contract` tag. Note that all the
+interactions will be part of a bundle (i.e. will have the `edges.node.bundledIn.id` value set).
+
+```qql
+query {
+  transactions(
+    ids: ["1UIhK4vL5lc2X4aMsJFmMpJqfdgrjznVzi2F17yLBlc"]
+  ) {
+    edges {
+      node {
+          id
+          tags {
+            name
+            value
+          }
+          block {
+            height
+          }
+          bundledIn {
+            id
+          }
+        }
+    }
+  }
+}
+```
+
+## Interaction retrieval via Warp gateway
+
+The Warp `/gateway/interactions/:id` endpoint allows to retrieve the interaction info based on its original id.
diff --git a/warp-academy-docs/docs/sequencer/data-and-api.md b/warp-academy-docs/docs/sequencer/data-and-api.md
@@ -0,0 +1,99 @@
+# Data and API Specification
+
+This chapter specifies the format of data stored by the sequencer network and the API provided by each sequencer node.
+
+## Genesis state
+
+The genesis state of the Warp Sequencer is the initial data that is used to start the blockchain. This data includes:
+- Information about the Arweave block from which the sequencer will start.
+- Data required to calculate the prev sort key for interactions handled by the sequencer.
+
+### Arweave Block Information
+
+The first type of data needed for the sequencer to start is information about the Arweave block that was last processed by the previous centralized sequencer, 
+as well as data about the Arweave block that will be added first to the sequencer blockchain, along with the interactions it contains.
+These details are extracted from the `arweave_block.json` file, and a sample representation is provided below:
+
+```json
+{"lastArweaveBlock":
+   {"height":1318268,
+   "timestamp":1702040807,
+   "hash":"VwqjV_2VMlh5DzLHUbk0n3EN_a5aRbe-H2xIFACZ9AmZrujSwwmEdxt4B_M0jYxl"},
+ "nextArweaveBlock":
+    {"blockInfo":
+       {"height":1318269,
+        "timestamp":1702040913,
+        "hash":"CQCojSIk5irYvdnsvWKm1d62Oh5z9G3X7qAbNjcu2Yh-KYzgz1IYT9GIpaYIs6cJ"},
+        "transactions":
+           [{"id":"UAie4Tyqlz5eCTQK_aqoRzWznZ0UCdckTm6E4pjA_HE",
+             "contract":"KTzTXT_ANmF84fWEKHzWURD1LWd9QaFR9yfYUwH2Lxw",
+             "sort_key":"000001318269,0000000000000,56802e09c3d3310140df996fa70be6eccc8c28e715c601e39d820a6be816f54c"},
+            {"id":"6t-OqbyGeQ-yigDbwRcChtjwpB8w5uswg_yvd9LHE5A",
+             "contract":"KTzTXT_ANmF84fWEKHzWURD1LWd9QaFR9yfYUwH2Lxw",
+             "sort_key":"000001318269,0000000000000,6849c6ab2791631ed9645dfb9a7eea6868bcb5f6aa1fea19700bc48a4f9345b9"},
+            {"id":"1iJSdydrcR2YqEsZ8DVrKIrIIdv-MG7WvRL6cCe76Is",
+             "contract":"KTzTXT_ANmF84fWEKHzWURD1LWd9QaFR9yfYUwH2Lxw",
+             "sort_key":"000001318269,0000000000000,e6472aac3115163f3a14de480744be6dc6fc29f860397e50662ea18d7a9bff2f"}]}}
+```
+Where:
+- `lastArweaveBlock` is information about the block that was last processed by the previous sequencer.
+- `nextArweaveBlock` is the Arweave block that will be added first to the sequencer.
+
+### Previous Sort Keys for Contracts
+
+Information about the sort key of the last interaction for each contract is crucial for assigning prev sort keys to interactions handled by the sequencer. 
+This data is sourced from the `prev_sort_keys.json` file, structured as follows:
+```json
+[{"contract":"007g_77MJ1eJaeRvDYwj4Po-6_2_G-5DGyN7eX3AL4o",
+  "sortKey":"000001025889,1664397554325,6dca2bac0ee65c6f3a3b0f3e5fe5d162c130bc5c39ea203524b5b799d1ad7f04"},  
+ {"contract":"008p0JdJA0XY79o139Us9OgXbcLmxiKubZ_7v31HZOI",
+  "sortKey":"000001119402,1676452495692,13468b18dd98a9c0fe9cb3e6f5b482ad14da1b744dea660a373fadb0ff0360d9"},
+ {"contract":"0094QB2iMzuy7h-W5_577li7kat2jb3EXYmDxRYm2yM",
+  "sortKey":"000001237223,1691593745386,b4c26ef252025f97b6684fb88e5a2c66ec4f8b1b9d5899e1c05859550aa3265b"},
+ {"contract":"00Bh4GLtT1kY4sAu3QbsXH3ZSuXa2iBaVXO64LJovp8",
+  "sortKey":"000000801138,0000000000000,1ed5d58b01f4deda06dcd61bd4e1d8494e82142df0d8333758d8a6d0648eb78f"},
+ ...
+]
+```
+
+## Messages format
+
+Sequencer blocks encapsulate [transactions](https://docs.cosmos.network/main/learn/advanced/transactions), which in turn contain [messages](https://docs.cosmos.network/main/build/building-modules/messages-and-queries). 
+Within the realm of Warp Sequencer, two fundamental types of messages play a pivotal role: those tailored for incoming interactions and those designed for Arweave blocks retrieved by the sequencer.
+Messages are defined and serialized in the [Protocol Buffers](https://protobuf.dev/) format.
+
+### Interaction Message
+
+The `MsgDataItem` message is used to store an interaction sent to the sequencer. The message has the following structure:
+```protobuf
+message MsgDataItem {
+  bytes data_item = 1;
+  string sort_key = 2;
+  string prev_sort_key = 3;
+  bytes random = 4;
+}
+```
+
+### Arweave Block Message
+
+The `MsgArweaveBlock` message is used to store an Arweave block that has been downloaded by the sequencer. The message has the following fields:
+
+```protobuf
+message MsgArweaveBlock {
+  ArweaveBlockInfo block_info = 1;
+  repeated ArweaveTransactionWithInfo transactions = 2;
+}
+```
+
+TODO
+
+## Data stored in sequencer nodes
+
+- last_arweave_block
+- prev_sort_key
+- next_arweave_block (?)
+
+## API methods
+- Open API link
+- list of the most important API methods
+- instructions on how to connect to notifications (?)