@voicenter/pastash NPM

paStash

"When logs give you spaghetti, make pasta"

What is paStash ?

PaStasH (pastaʃ'ʃ-utta) is a NodeJS multi I/O processor supporting ingestion, decoding, interpolation and correlation of data - be it logs, packets, events and beyond. PaStash supports the Logstash configuration format and delivers cross-functionality comparable to "Beats" with custom modules, providing a flexible and agnostig data pipelining tool.

What can I do with paStash ?

paStash is designed manage spaghetti I/O with input, processors and output modules for all seasons, and can be useful in many scenarios, such as parsing logs to objects, distributing data to multiple formats, interexchanging and correlating protocols and streams, while interpolating and manipulating data intransit. paStash is developed using NodeJS, which is an ideal language for applications with many IO and offers:

lower memory footprint
lower cpu footprint
faster startup delay
ease of extension

paStash configuration is compatible with logstash. You can easily replace a logstash node by a paStash one in most cases. The data are formatted in the same way to be compatible with logstash UIs.

How does it work ?

The architecture is identical to logstash architecture. You have to instanciates plugins with the paStash core. There are three type of modules:

inputs plugins: where datas come into paStash. Examples: file, zeromq transport layer
filter plugins: extract and manipulate fields from logs, like timestamps. Example: regex plugin
outputs plugins: where datas leave from paStash: Examples: ElasticSearch , zeromq transport layer.

A typical paStash deployement contains agents to crawl logs and a log server.

On agent, paStash is configured whith inputs plugins to get logs from your software stack, and one output plugin to send logs to log server (eg. zeromq output plugin).

On log server, logs come trough a zeromq input plugin, are processed (fields and timestamps extraction), and send to ElasticSearch.

How to use it ?

Installation

NPM install

npm install --save @voicenter/pastash

Manual install

Install NodeJS, version >= 8.xx
Install build tools
- Debian based system: apt-get install build-essential
- Centos system: yum install gcc gcc-c++ make
Install zmq dev libraries: This is required to build the node zeromq module.
- Debian based system: apt-get install libzmq1. Under recent releases, this package is present in default repositories. On ubuntu lucid, use this ppa. On debian squeeze, use backports.
- Centos 6: yum install zeromq zeromq-devel. Before, you have to add the rpm zeromq repo : curl http://download.opensuse.org/repositories/home:/fengshuo:/zeromq/CentOS_CentOS-6/home:fengshuo:zeromq.repo > /etc/yum.repos.d/zeromq.repo
Clone repository: git clone git://github.com/sipcapture/pastash.git && cd pastash
Install dependencies: npm install.

The executable is bin/pastash

Configuration formats

There are two format for configuration. The legacy format use urls. The new one is identical to the logstash config format.

Note : multiple configuration files can be used in parallel with the --config_dir switch.

Configuration by logstash config files

Example for an input file

input {
  file {
    path => '/tmp/toto.log'
  }
}

You can use if to have an event dependent configuration. See here for details. As for urls, config can be specified

directly on the command line
in a file (use the --config_file switch)
in all files in a directory (use the --config_dir switch)

Note : the implementation is young, all bugs reports are welcome. Note : both formats can be mixed.

Command lines params

--log_level to change the log level (emergency, alert, critical, error, warning, notice, info, debug)
--log_file to redirect log to a log file.
--patterns_directories to add some directories (separated by ,), for loading config for regex plugin and grok plugins. Grok patterns files must be located under a grok subdirectory for each specified directory.
--db_file to specify the file to use as database for file inputs (see below)
--http_max_sockets to specify the max sockets of http.globalAgent.maxSockets. Default to 100.
--alarm_file to specify a file which will be created if paStash goes in alarm mode.

Examples

Config file for an agent:

input {
  file {
    path => "/var/log/nginx/access.log"
  }
}

output {
  zeromq {
    address => ["tcp://log_server:5555"]
  }
}

Config file for log server:

input {
  zeromq {
    address => ["tcp://0.0.0.0:5555"]
  }
}

filter {
  regex {
    pattern => http_combined
  }
}

output {
  elasticsearch {
    host => localhost
    port => 9200
  }
}

See our wiki for many more examples

Plugins list

Inputs

Common concepts / parameters :

Filters

Common concepts / parameters :

Apps with embedded parsers :

Outputs

Common concepts / parameters :

Adding your plugins

You can add easily add your plugins :

Manually :

create a directory layout on the path of your choice : /var/my_plugins/inputs, /var/my_plugins/outputs, /var/my_plugins/filters
set the NODE_PATH variable to NODE_PATH=/var/my_plugins:/node_logstash_path/lib
add your plugins in inputs, outputs or filters directory. In the plugin code, you can reference base plugins with var base_filter = require('lib/base_filter');
reference your plugin as usual.

With native packaging

The plugins must be deployed in /var/db/pastash/custom_plugins. All subdirectories already exists. The NODE_PATH is already set.

Signals

USR1: stoping or starting all inputs plugins. Can be used to close input when output targer are failing
USR2: see below file output plugin

Misc

Elasticsearch mapping

License

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Made by Humans

This Open-Source project is made possible by actual Humans without corporate sponsors, angels or patreons. If you use this software in production, please consider supporting its development with contributions or donations

@everything-registry/sub-chunk-990 @infinitebrahmanuniverse/nolb-_voi

1.0.46

6 years ago

1.0.44

6 years ago

1.0.43

6 years ago