@chainlink/ea-bootstrap v0.2.0

Chainlink External Adapter Bootstrap

The core framework that every External Adapter uses.

Detailed here is optional configuration that can be provided to any EA through environment variables.

Table of Contents

1. Server configuration
2. Performance
   • Caching
   • Redis
   • Rate Limiting
     • Provider Limits
   • Cache Warming
   • Request Coalescing
3. Metrics
4. Websockets

Server configuration

Performance

Caching

To cache data every adapter using the bootstrap package has access to a simple LRU cache that will cache successful 200 responses using SHA1 hash of input as a key.

⚠️ Note

Please check and ensure caching is allowed and not in violation of the Terms of Service of the data provider's API. Disable caching flags if it is not supported by the specified API provider's TOS.

To configure caching these environment variables are available:

Cache key

The cache key of a stored request is derived by hashing the input object, using the SHA1 hash function, while by default ignoring keys ['id', 'maxAge', 'meta', 'rateLimitMaxAge', 'debug']. So for example these few requests will derive the same key:

• {"id": 1, "data": {"base":"LINK", "quote": "USD"}}
• {"id": 2, "data": {"base":"LINK", "quote": "USD", "maxAge": 10000}}
• {"id": 3, "data": {"base":"LINK", "quote": "USD"}}

The maxAge input argument can be used to set per item maxAge parameter. If not set, or set to 0, the cache level maxAge option will be used. Every time the maxAge input argument changes, the item will be cached with the new maxAge parameter. To avoid hitting the cache for a specific item set maxAge: -1 (any value < 0).

Ignoring keys

If you want to ignore specific input data object keys, to be excluded from key derivation, you can use the CACHE_KEY_IGNORED_PROPS environment variable.

For example, if the CACHE_KEY_IGNORED_PROPS=timestamp is set, these requests will derive the same key:

• {"id": 1, "data": {"base":"LINK", "quote": "USD", "timestamp": 1598874704}}
• {"id": 2, "data": {"base":"LINK", "quote": "USD", "timestamp": 1598874721}}

Local cache

Redis

For local development run a Redis Docker container:

docker run -p 6379:6379 --name ea-redis -d redis redis-server --requirepass SUPER_SECRET

For ElastiCache Redis deployments: if encryption in transit is used, to make a connection CACHE_REDIS_URL needs to be set with rediss://... protocol.

Rate Limiting

To avoid hitting rate limit issues with the data provider subscription, a rate limit capacity per minute can be set:

• Option 1, manual capacity setting:

  Required?	Name	Description	Options	Defaults to
  	RATE_LIMIT_CAPACITY	Maximum capacity on requests per minute		undefined

• Option 2, capacity by reference. Check your plan here and use it with the following configuration:

Provider Limits

Each provider is defined within limits.json as so:

{
  "[provider-name]": {
    "http": {
      "[plan-name]": {
        "rateLimit1s": 1,
        "rateLimit1m": 30,
        "rateLimit1h": 200
      },
      "premium": {
        "rateLimit1s": 10,
        "rateLimit1m": 300,
        "rateLimit1h": 2000
      }
    },
    "ws": {
      "[plan-name]": {
        "connections": 1,
        "subscriptions": 10
      }
    }
  }, {...}
}

Being:

• provider-name: The provider name. E.g. "amberdata" or "coinmarketcap"
• plan-name: The provider plan name. Used as a identifier for the plan. E.g. "free" or "premium"
• There are two protocols with different limit types:
  • http: With rateLimit1s, rateLimit1m, rateLimit1h, which stands for requests per second/minute/hour respectively. If only one is provided, the rest would be calculated based on it.
  • ws: Websocket limits, which accepts: connections and subscriptions. If websockets are not supported on the provider, can be left empty as ws: {}

Cache Warming

*To use this feature the CACHE_ENABLED environment variable must also be enabled.

Request Coalescing

One final consideration is the “thundering herd” situation, in which many clients make requests that need the same uncached downstream resource at approximately the same time. This can also occur when a server comes up and joins the fleet with an empty local cache. This results in a large number of requests from each server going to the downstream dependency, which can lead to throttling/brownout. To remedy this issue we use request coalescing, where the servers or external cache ensure that only one pending request is out for uncached resources. Some caching libraries provide support for request coalescing, and some external inline caches (such as Nginx or Varnish) do as well. In addition, request coalescing can be implemented on top of existing caches. -- Amazon on Caching challenges and strategies

To configure caching these environment variables are available:

Metrics

A metrics server can be exposed which returns prometheus compatible data on the metrics endpoint on the specified port. When enabled, a metrics endpoint is opened on /metrics, which can be prepended with the BASE_URL if METRICS_USE_BASE_URL is enabled.

*Please note that this feature is EXPERIMENTAL.

To run Prometheus and Grafana with development setup:

yarn dev:metrics

Websockets

Adapters who interact with data providers that support websockets will be able to use them offering a WS interface. Each adapter will have its corresponding WS documentation.

Multiple subscription channels are multiplexed over one connection.

For every type of request, the adapter will subscribe to the corresponding channel.

From the moment the subscription is confirmed, the adapter will start receiving messages with the relevant information, piping this information to the cache. On future requests, the adapter will always have fresh data saved on cache. If there is no data available in cache, the adapter will continue with its default execution.

*For the websockets to be effective, caching needs to be enabled