Skip to content

Commit

Permalink
doc: readme changes
Browse files Browse the repository at this point in the history
  • Loading branch information
n00m4d committed Oct 14, 2024
1 parent bb1f361 commit 9647cb0
Showing 1 changed file with 6 additions and 63 deletions.
69 changes: 6 additions & 63 deletions plerkle_snapshot/README.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,5 @@
# Solana Snapshot ETL 📸

[![crates.io](https://img.shields.io/crates/v/solana-snapshot-etl?style=flat-square&logo=rust&color=blue)](https://crates.io/crates/solana-snapshot-etl)
[![docs.rs](https://img.shields.io/badge/docs.rs-solana--snapshot--etl-blue?style=flat-square&logo=docs.rs)](https://docs.rs/solana-snapshot-etl)
[![license](https://img.shields.io/badge/license-Apache--2.0-blue?style=flat-square)](#license)

**`solana-snapshot-etl` efficiently extracts all accounts in a snapshot** to load them into an external system.
Expand All @@ -23,69 +21,14 @@ A full snapshot file contains a copy of all accounts at a specific slot state (i
Historical accounts data is relevant to blockchain analytics use-cases and event tracing.
Despite archives being readily available, the ecosystem was missing an easy-to-use tool to access snapshot data.

## Building

```shell
cargo install --git https://github.com/terorie/solana-snapshot-etl --features=standalone --bins
```

## Usage

The ETL tool can extract snapshots from a variety of streaming sources
and load them into one of the supported storage backends.

The basic command-line usage is as follows:

```
USAGE:
solana-snapshot-etl [OPTIONS] <LOAD_FLAGS> <SOURCE>
```

### Sources

Extract from a local snapshot file:

```shell
solana-snapshot-etl /path/to/snapshot-*.tar.zst ...
```

Extract from an unpacked snapshot:

```shell
# Example unarchive command
tar -I zstd -xvf snapshot-*.tar.zst ./unpacked_snapshot/

solana-snapshot-etl ./unpacked_snapshot/
```

Stream snapshot from HTTP source or S3 bucket:

```shell
solana-snapshot-etl 'https://my-solana-node.bdnodes.net/snapshot.tar.zst?auth=xxx' ...
```

### Targets

#### Geyser plugin

Much like `solana-validator`, this tool can write account updates to Geyser plugins.
Instruction of usage we can find in [this section](../README.md#snapshot-etl).

```shell
solana-snapshot-etl snapshot-139240745-*.tar.zst --geyser plugin-config.json
```

For more info, consult Solana's docs: https://docs.solana.com/developing/plugins/geyser-plugins
## Changes

#### Dump programs
The following changes were made to the original Solana Snapshot ETL tool:

The `--programs-out` flag exports all Solana programs (in ELF format).

```shell
solana-snapshot-etl snapshot-139240745-*.tar.zst --programs-out programs.tar
```

or to extract in place

```shell
solana-snapshot-etl snapshot-139240745-*.tar.zst --programs-out - | tar -xv
```
- The solana-opcode-stats binary has been removed.
- The current version of the ETL only streams data to the Geyser plugin. CSV and SQLite support have been removed.
- The ETL now assigns a slot number to the account data, extracting the slot number from the snapshot file name.

0 comments on commit 9647cb0

Please sign in to comment.