revibe v0.0.3
Revibe - JSON templating and preprocessing
Example
consider the following setting:
question.txt
What is best in life ?answers.txt
Raindrops on roses
Whiskers on kittens
Crushing your enemies, seeing them driven before you and hearing the lamentations of their womenanswers.json
[
"Raindrops on roses",
"Whiskers on kittens",
"Crushing your enemies, seeing them driven before you and hearing the lamentations of their women"
]demo.rvb
This is an example of a revibe template. A revibe template is itself a JSON document whose keys and values can be either carried as-is into the resulting JSON or can be interpreted as revibe expressions by one of the defined transformers allowing data to be fetched from various sources and embedded into the resulting document.
{
"//!":"This is a comment, it will not be in the result",
"Hello":"World",
"//!2":"This is also comment, it is in the same object as the first comment so it needs a unique key",
"//!3":"",
"//!4":"",
"//!5":"Let see some transforms:",
"TransformExamples":
{
"Shell":
{
"//!":"Execute a shell command and use it's result",
"!HelloWorldFromShell":"shell://echo 'Hello world'"
},
"Javascript":
{
"//!":"Evaluate a javascript expression and use it's return value",
"!GeneratedOn":"js:// new Date()"
},
"File":
{
"//!1":"Read a text a file in a shell/os agnostic way",
"!Question":"file://question.txt",
"//!2":"Read a json file as text",
"!AnswerJson":"file://answer.json",
"//!3":"Read a json file and compose into result",
"!!Answer":"file://answer.json"
}
}
}Revibe templates can be transformed via the rvb command line program
$ rvb -i demo.rvb
with the result going to standard output
{
"Hello": "World",
"TransformExamples": {
"Javascript": {
"GeneratedOn": "2016-07-01T16:06:45.585Z"
},
"Shell": {
"HelloWorldFromShell": "Hello world"
},
"File": {
"Question": "What is best in life ?",
"AnswerFromText": [
"Raindrops on roses",
"Whiskers on kittens",
"Crushing your enemies, seeing them driven before you and hearing the lamentations of their women"
],
"AnswerFromJson": [
"Raindrops on roses",
"Whiskers on kittens",
"Crushing your enemies, seeing them driven before you and hearing the lamentations of their women"
]
}
}
}or in node via the revibe module
require('revibe')
.Revibe(
JSON.parse(fs.readFileSync('demo.rvb'))
)
.then(console.log)With the result being provied as javascript object to the provided continuation, in this case console.log.
Revibe uses ES6 Promises and targets node 4.2 and above.
Installation
OSX and Linux
From npm registry
$npm install revibe
Via npm without the registry
Download the latest .tgz package from the releases section, unpack to a local directory, change to it and run
$npm install .
Without npm
Download the latest .tgz package from the releases section, unpack to a local directory and optionally add it to your $PATH
Windows
TBD
Documentation
Key prefixes
A key prefix indicates how the corresponding value will be interpreted and embedded into the resulting document, All revibe prefixes end with a ! character, property names that are not prefixed are transformed recursively as templates.
The following key prefixes are supported:
//! - Comment, both the key and the value will not be present in the output
! - The value returned by a transformer will be used as-is (string for shell, file and http transformers and according to the type returned by the javascript expression for the javascript transformer)
*! - The value returned by a transformer is a multi-line text that will be split to lines and embedded as an array of strings
!! - The value returned is a JSON text which will then be parsed and embedded in the resulting document as an object
!#! - The value returned is a JSON text representing a revibe template which will then itself be transformed by revibe and the result of the transformation embedded in the resulting document.
Transformers
File
Reads the content of a file from the filesystem.
Shorthand expression
file://<path>where path is either an absolute or a relative (to cwd) path
Object notation
{
"Type":"file",
"Expression": <path>
"TrimTrailingNewline": <boolean> // Default: true
}Shell
Executes a command in user's default shell and returns the output
Shorthand expression
shell://<cmd>where cmd is the command to execute
Object notation
{
"Type":"shell",
"Expression": <path>
"TrimTrailingNewline": <boolean> // Default: true
}Quotes and shell variable expansion
Because the shell expressions are enclosed in JSON string literals double quotes (") need to be escaped in shell commands to allow for shell variable expansion:
{"!path":"shell:// echo 'The current PATH environment variable is $PATH'"}
will transform to
{"path": "The current PATH environment variable is $PATH"}
whereas
{"!path":"shell:// echo \"The current PATH environment variable is $PATH\""}
will transform to
{
"path": "The current PATH environment variable is /Library/Frameworks/Python.framework/Versions/3.5/bin:/opt/local/bin:/opt/local/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/opt/X11/bin:/usr/local/share/dotnet/bin:/usr/local/share/dotnet:/usr/local/git/bin:/usr/local/go/bin"
}
or whatever your $PATH is.
Multiline split
Multiline split is a key prefix that is available for all transformers and is particularly useful with shell commands. e.g.:
{"*!files":"shell//ls -1"}
will transform nicely into
{"files":["file1","file2",...,"fileN"]}
Http
Fetches a web resource
Shorthand expression
URL
Object notation
{
"Type":"http", // also for https
"Expression": <url>
"Headers": {
"User-Agent","revibe"
}
}Currently only the GET method is suported
Javascript
Evaluates a javascript expression and returns the result
Shorthand expression
js:// <js code>
Object notation
{
"Type":"js",
"Expression": <js code>
}Missing/Planned
More features for the file:// transformer
- Handle groups of files sensibly, such as "file://*.json" - merge into same
object/object-per file/array ?
- Handle groups of files sensibly, such as "file://*.json" - merge into same
js transformer - detect when the returned value is a promise and use it's result.
In rvb (command line tool) run a user-provided js code to prep the node environment for stuff used by "js://" expressions prior to transform.
A key prefix to have resulting JSON fields replace the fields in the parent scope
{"<!!foo":"{'bar':'baz'}"} => {"bar":"baz"}vs
{"!!foo":"{'bar':'baz'}"} => {"foo":{"bar":"baz"}}vs
{"!foo":"{'bar':'baz'}"} => {"foo":{"{'bar':'baz'}"}}- Better JSON parsing and serialization:
- Deterministic serialization - preserve template order for emitted properties.
- Better error reporting (where is that goddamn breaking trailing coma ?)
Extensibility for pluggable third-party transformers (mongo, couch, elasticsearch etc.)
Browser use case and relevant transformers (XHR for http etc.)
Usage considerations
Revibe is meant to make the creation and management of JSON documents used in devops tools, configuration, master data etc. cleaner and easier, but at the same time the ability to interop with the shell and javascript node environments creates a possibility for the oposite to happen - if you find yourself writing non-trivial shell scripts or javascript code beyond simple expressions in your templates you are probably not using it for the purpose intended, in particular you should not use it for or rely on any side effects caused by the transformation process.
Security considerations
A revibe template is equivalent to code, transforming a template from an untrusted source is equivalent to compiling untrusted source code and running it - don't.
Using recursive revibe (!#!) key prefix implies that the trust you put in your template author is also recursive -
the trust boundary for
template.rvb
{
"!!foo":"file.json"
}should at least cover the author of template.rvb as file.json will only be JSON.parse()-d
whereas the trust boundary for:
{
"!#!foo":"file.json"
}should cover the author of file.json as well becasue the content of file.json itself will be treated as a revibe template.