@exocet/pandora-elasticsearch v0.0.1-alpha.1

Pandora Elasticsearch

Addon to provide flow and setup functions to Elasticsearch persistence for Pandora.

npm install --save @exocet/pandora-elasticsearch

Concepts

Native ID

There are two approaches to index documents into Elasticsearch: using custom IDs or Elasticsearch generated IDs. Elasticsearch have some performance issues with custom IDs and how they suffer from lookup into Lucene index. To work around this issue this addon have two working modes per entity:

• Native ID off - With native ID off the data will be indexed with custom IDs, normally UUIDv4's generetad at application side (this feature is supported by this addon with zero effort).
• Native ID on - The addon operates with Elasticsearch generated IDs and is going to create an id field in the source of the documents to put the application generated custom IDs (here we can use the UUIDv4 auto generation feature too).

Setup

Available features:

• Create indexes with custom configuration and dynamic aliases
• Create mappings from entities definitions
• Provides input, output and AQS query parsers
• Flow steps: insert (index), update, delete, exists and search

To add this addon into your project, put the addon spec into your endpoint YAML:

kind: Pandora/endpoint
metadata:
  name: myEndpoint
spec:
  addons:
    - name: elasticsearch
      package: "@exocet/pandora-elasticsearch"
      flow: true
      setup: true
  configuration:
    elasticsearch: # Any configuration here will be passed to client configuration (1)
      node: http://127.0.0.1:9200
      searchOptions: # Optional options applied in all searches
        maxPageSize: 50 # Maximum allowed search page size
      indices:
        - entityName: myNamespace.myEntity # To create index and mapping for an entity, pass the entity name, the settings bellow are the defaults
          numberOfShards: 1
          numberOfReplicas: 1
          autoExpandReplicas: false
          refreshInterval: 1s
          maxResultWindow: 10000
          maxInnerResultWindow: 100
          checkOnStartup: false
        - entityName: myNamespace.myEntityWithCustomIndex
          index: my_entity_with_custom_index # Custom entity index name instead of autogenerated
        - index: my_index # You can create custom indexes
          mapping: # With custom mappings
            myField:
              type: keyword
          aliases: # With custom aliases
            my_alias: true # Normal alias
            my_filtered_alias: # Filtered alias (2)
              term:
                active: true

1. Client configuration
2. Filtered alias

After the setup the following property of the context will be available:

• .application.dbs.elasticsearch - The client connected to Elasticsearch

Hooks

The hooks created by this addon are:

• elasticsearchQueryParsers (.service.hooks.useHook('elasticsearchQueryParsers')) - Synchronous hook that parses the search AQS query into Elasticsearch search body.

  	```javascript
  	const [queryParsers] = this.service.hooks.useHook('elasticsearchQueryParsers');
  	const queryParser = queryParsers(entityName);
  	const {
  		body, // The bodybuilder instance with filters from AQS query (1)
  		includes // The included fields requested by AQS query
  	} = queryParser(
  		aqsQuery,
  		{ // Optional options
  			body: bodybuilder(), // Existent instance of bodybuilder to build the filters (1)
  			nativeId: false // Indicates if the targed entity uses native id approach
  		}
  	);
  	```

• elasticsearchOutputParsers (.service.hooks.useHook('elasticsearchOutputParsers')) - Synchronous hook that parses the search output from Elasticsearch (each hit object) into entity representation.

  	```javascript
  	const [outputParsers] = this.service.hooks.useHook('elasticsearchOutputParsers');
  	const outputParser = outputParsers(entityName);
  	const results = response.body.hits.hits.map(outputParser);
  	```

• elasticsearchInputParsers (.service.hooks.useHook('elasticsearchInputParsers')) - Synchronous hook that parses the input from entity data into Elasticsearch document source representation.

  	```javascript
  	const [inputParsers] = this.service.hooks.useHook('elasticsearchInputParsers');
  	const inputParser = inputParsers(entityName);
  	const documentToIndex = inputParser(entity);
  	```

1. bodybuilder

Flow

The provided flow type and steps are listed bellow:

• insert - This flow step index data into Elasticsearch:

  	```yaml
  	kind: Pandora/flowStep
  	metadata:
  	  name: entityPersistence
  	  labels:
  	    operation: insert
  	spec:
  	  type: elasticsearch # Flow step type
  	  options:
  	    operation: insert # Defines that is an insert operation
  	    inboundFrom: request.data # The path to the data to be indexed (acquired from execution context)
  	    idFrom: null # If you want to use a custom ID instead of UUIDv4 you can pass the path from the execution context to get the ID value
  	    indexName: null # If you're using custom index name, pass the name here
  	    nativeId: false # Enables or disables the Native ID use
  	    entity: # The entity to define the input parser hook
  	      namespace: myNamespace
  	      name: myEntity
  	```

• exists - This flow step check if an entity was indexed by ID:

  	```yaml
  	kind: Pandora/flowStep
  	metadata:
  	  name: entityPersistence
  	  labels:
  	    operation: exists
  	spec:
  	  type: elasticsearch
  	  options:
  	    operation: exists
  	    idFrom: request.data.id # The path of the ID
  	    outboundTo: null # The execution path of the context to put the found ID, is an useful flow step to put before update and delete flows to get the native ID before operating the entity
  	    nativeId: false # This is important, this option will instruct the flow step to search by _id field or the _source.id field
  	    indexName: null
  	    entity:
  	      namespace: myNamespace
  	      name: myEntity
  	```

• update - This flow step updates indexed entities:

  	```yaml
  	kind: Pandora/flowStep
  	metadata:
  	  name: entityPersistence
  	  labels:
  	    operation: update
  	spec:
  	  type: elasticsearch
  	  options:
  	    operation: update
  	    inboundFrom: request.data
  	    idFrom: request.data.id # The path of the ID to update entity
  	    indexName: null
  	    entity:
  	      namespace: myNamespace
  	      name: myEntity
  	```

• delete - This flow step deletes indexed entities:

  	```yaml
  	kind: Pandora/flowStep
  	metadata:
  	  name: entityPersistence
  	  labels:
  	    operation: delete
  	spec:
  	  type: elasticsearch
  	  options:
  	    operation: delete
  	    idFrom: request.data.id
  	    indexName: null
  	    entity:
  	      namespace: myNamespace
  	      name: myEntity
  	```

• search - This flow step searches for indexed entities using AQS:

  	```yaml
  	kind: Pandora/flowStep
  	metadata:
  	  name: entityPersistence
  	  labels:
  	    operation: search
  	spec:
  	  type: elasticsearch
  	  options:
  	    operation: search
    queryFrom: request.query # The execution context path that have the AQS querystring
    outboundTo: response # The execution context path to put the search result
    bodyFrom: null # The execution context path that have an existing bodybuilder instance to append the parsed AQS filters, pass null to indicate the flow step to generate a new instance
    indexName: null
  	    entity: # Defines the AQS parser and the output parser
  	      namespace: myNamespace
  	      name: myEntity
  	```