SEP: 0035
Title: Operation IDs
Author: Scott Fleckenstein (@nullstyle), Isaiah Turner (@Isaiah-Turner), Debnil Sur (@debnil)
Track: Standard
Status: Draft
Created: 2020-08-26
Discussion: https://groups.google.com/g/stellar-dev/c/vCgQhmox32Q
This protocol defines an ID scheme for Stellar operations.
There is no universal method to generate IDs for Stellar operations. Horizon generates IDs, but that implementation is internal and changeable. This design and implementation were created by @nullstyle, and the specification and rationale are derived from his code comments. In many ways, this is already a de facto ecosystem standard: current prominent projects like Stellar Expert already utilize this scheme internally, and recent upgrades in Horizon's ingestion system will motivate other data-intensive applications. An example is SDF's own Stellar ETL project. But these applications cannot assign IDs in a consistent manner with Horizon's implementation. An ecosystem standard will unlock this consistency.
This protocol defines an assignment method for IDs for operations in historical ledger data, representing the total order of operations on the Stellar network. The ledger sequence number, transaction application order, and operation index are packed into a single ID value.
An ID is a single 64-bit signed integer. The sign guarantees SQL compatibility, since some dialects do not support unsigned integers. In order to create an id, three pieces of information are required. All three should be expressed in network byte order, which is big-endian. They are left-padded with 0s to fit the bitrange.
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Ledger Sequence Number |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Transaction Application Order | Op Index |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
- Ledger Sequence: this takes up 32 bits, from bits 0 to 31, inclusive. The maximum supported number of ledgers is 2,147,483,647.
- Transaction Application Order: the order that the transaction was applied within the ledger where it was validated. This takes up 20 bits of the id, from bits 32 to 51, inclusive. The application order value starts at 1. The maximum supported number of transactions per operation is 1,048,575.
- Operation Index: the index of the operation within its parent transaction. This takes up 12 bits of the id, from bits 52 to 63, inclusive. The operation index value starts at 1. The maximum supported number of operations per transaction is 4095.
Here is a reference implementation.
This design reflects the Horizon database's long-standing approach to generating operation IDs approach used in the Horizon database, originally designed by S. Rather than proposing a new approach, we are converting a de facto ecosystem standard into an explicit one. This design provides a sufficiently generic approach to operation IDs that can be used to display a total order of historical operations.
Note that this scheme cannot be used to order operations in advance, since the transaction application order is unknown until the network validators agree on ledger contents, close it, and publish results. The order of transactions within the ledger is pseudorandom: it doesn't depend on the transaction hash, fees, operations count, or any other predictable factor. The transaction may also not be included in the next ledger, as it may hang in the mempool in an activity spike. Thus, this schema is only applicable for historical ledger data.
Note that failed transactions are included in the ledger and consume order numbers. This can confuse upstream systems that ignore failed transactions. (It's worth to specify this explicitly, as community members encountered a bug from this behavior.)
Note that the Horizon system includes an ID scheme for transactions and ledgers. After discussion, we excluded those from this protocol. Both already have unique IDs, in transaction hashes and ledger sequence numbers. If a resource already has a unique identifier in the protocol level, additional identifiers add minimal help and can only be confusing.
N/A