0.1.12 • Published 6 years ago
eth-indexer v0.1.12
High Performance Ethereum Indexer
Syncs events from Ethereum and indexes them for fast retrieval. This is useful and often essential for production Dapps.
- Support for 6 different persistence stores
- Includes "single-node friendly" stores such as flat-file JSON
- Comprehensive error retry logic when communicating with Ethereum node
- Maximum throughput via JSON RPC using batched parallel fetching
Todo
- Keep track of blockNumbers in persistence for fast-resume
- Benchmark documentation
- Support going forward syncing for Ethereum
- NPM package
- Improved failsafes for data consistency
- Split up stores into separate dependencies
Supported Indexing Stores and Benchmarks
The following indexing implementations have been benchmarked on an Ethereum blockchain section with a relatively large number of events (EtherDelta exchange blocks 4800000
- 5002718
) on a single 8-core 2017 Macbook Pro:
Store Name | Store ID | Events Per Second | Blocks Per Second |
---|---|---|---|
Local Memory | memory | 4072.7/s | 464.5/s |
Redis | redis | 2049.55/s | 235.25/s |
Local Flat File | file | 1798.75/s | 201.75/s |
MongoDB | mongodb | 795.5/s | 94.6/s |
Elasticsearch | elasticsearch | 452.25/s | 57.1/s |
LevelDB | level | 207.35/s | 26.95/s |
Install Dependencies
yarn install
Testing & Linting
yarn test
yarn lint
Integration tests (requires synced ethereum node at localhost:8545):
yarn test:integration
Example: Sync trading and balance events from decentralized trading contract: EtherDelta
Contract address: https://etherscan.io/address/0x8d12a197cb00d4747a1fe03395095ce2a5cc6819
See examples/etherdelta/sync.js
.
Define which events to index on which keys:
const indexing = {
events: {
Withdraw: {
keys: ['user'],
},
Trade: {
keys: ['tokenGive', 'tokenGet', 'get', 'give'],
},
},
};
Then create a store:
const store = new LevelStore(indexing, '/tmp/etherdelta.db');
Now boot up the indexer:
const indexer = new Indexer(store, EtherdeltaABI, '0x8d12a197cb00d4747a1fe03395095ce2a5cc6819');
await indexer.syncAll({
fromBlock: 4906764,
});
Once indexing is complete events can be retrieved as follows:
const events = await store.get('Withdraw', 'user', '0x13d8d38421eb02973f3f923a71a27917bd483190');
Directory Structure
package.json
- Configure dependenciesdist/*
- Files generated by babelsrc
- All source codesrc/*/__tests__
- Unit testsintegration/__tests__
- Integration tests