@plurid/deon NPM

deon is a notation format for structured data.

deon is intended to be:

light on syntax — friendly for human read/write, should feel more like note-taking than data entry;
moderately fast — with a general use case for configuration-like files, loaded once at build/runtime;
programming-lite — although not a programming language, the in-file imports and the linking (in-file variables) give deon a programmatic feel.

The deon filename extension is .deon, and the media type is application/deon.

Why deobject? More of a play-on-words, although a case can be made considering the linking feature and the possible 'assembling' of the root, as if the object has been de-structured. As an afterthought, deon could also be rooted into the Ancient Greek word, δέον, that which is binding, such as how data binds functionality into code.

Example

The following .deon file

// deon

{
    entities [
        {
            id 01
            name One
            active true
        }
        {
            id 02
            name Two
            active false
        }
    ]
    #time
}

time 1598439736

will produce the following data

// JavaScript/TypeScript

const data = {
    entities: [
        {
            id: '01',
            name: 'One',
            active: 'true',
        },
        {
            id: '02',
            name: 'Two',
            active: 'false',
        },
    ],
    time: '1598439736',
};

// Rust

let data = deon!({
    entities [
        {
            id 01
            name One
            active true
        }
        {
            id 02
            name Two
            active false
        }
    ]
    time: 1598439736
});

# Python

data = {
    "entities": [
        {
            "id": "01",
            "name": "One",
            "active": "true",
        },
        {
            "id": "02",
            "name": "Two",
            "active": "false",
        },
    ],
    "time": "1598439736",
}

Comparisons

Consider the following commonly-used formats with an example file from performer:

# an .yaml file

---
stages:
- name: 'Setup NPM Private Access'
  directory: '/path/to/package'
  imagene: 'ubuntu'
  command:
  - '/bin/bash'
  - './configurations/.npmrc.sh'
  secretsEnvironment:
  - 'NPM_TOKEN'

- name: 'Generate the Imagene'
  directory: '/path/to/package'
  imagene: 'docker'
  command: [
    'build',
    '-f',
    './configurations/docker.development.dockerfile',
    '-t',
    'hypod.cloud/package-name:$SHORT_SHA',
    '.'
  ]

- name: 'Push Imagene to Registry'
  directory: '/path/to/package'
  imagene: 'docker'
  command: [
    'push',
    'hypod.cloud/package-name:$SHORT_SHA'
  ]

timeout: 720

// a .json file

{
  "stages": [
    {
      "name": "Setup NPM Private Access",
      "directory": "/path/to/package",
      "imagene": "ubuntu",
      "command": [
        "/bin/bash",
        "./configurations/.npmrc.sh"
      ],
      "secretsEnvironment": [
        "NPM_TOKEN"
      ]
    },
    {
      "name": "Generate the Imagene",
      "directory": "/path/to/package",
      "imagene": "docker",
      "command": [
        "build",
        "-f",
        "./configurations/docker.development.dockerfile",
        "-t",
        "hypod.cloud/package-name:$SHORT_SHA",
        "."
      ]
    },
    {
      "name": "Push Imagene to Registry",
      "directory": "/path/to/package",
      "imagene": "docker",
      "command": [
        "push",
        "hypod.cloud/package-name:$SHORT_SHA"
      ]
    }
  ],
  "timeout": 720
}

Consider the .deon version:

// a .deon file

{
    stages [
        {
            name Setup NPM Private Access
            directory /path/to/package
            imagene ubuntu
            command [
                /bin/bash
                ./configurations/.npmrc.sh
            ]
            secretsEnvironment [
                NPM_TOKEN
            ]
        }
        {
            name Generate the Imagene
            directory /path/to/package
            imagene docker
            command [
                build
                -f
                ./configurations/docker.development.dockerfile
                -t
                hypod.cloud/package-name:$SHORT_SHA
                .
            ]
        }
        {
            name Push Imagene to Registry
            directory /path/to/package
            imagene docker
            command [
                push
                hypod.cloud/package-name:$SHORT_SHA
            ]
        }
    ]
    timeout 720
}

or with nested internal linking

// a .deon file

// the root
{
    stages [
        #stage1
        #stage2
        #stage3
    ]
    timeout 720
}


// the leaflinks
stage1 {
    name Setup NPM Private Access
    #directory
    imagene ubuntu
    command #commands.stage1
    #secretsEnvironment
}

stage2 {
    name Generate the Imagene
    #directory
    imagene docker
    command #commands.stage2
}

stage3 {
    name Push Imagene to Registry
    #directory
    imagene docker
    command #commands.stage3
}

directory /path/to/package

commands {
    stage1 [
        /bin/bash
        ./configurations/.npmrc.sh
    ]
    stage2 [
        build
        -f
        ./configurations/docker.development.dockerfile
        -t
        #imageneName
        .
    ]
    stage3 [
        push
        #imageneName
    ]
}

secretsEnvironment [
    NPM_TOKEN
]

imageneName hypod.cloud/package-name:$SHORT_SHA

Implementations

deon is implemented for:

JavaScript/TypeScript
Rust - in progress

and will be implemented for:

See specifics for implementation details.

Installs

deon can be installed locally with the appropriate package manager for each implementation language, or can be installed globally as a Command-Line Interface tool.

Using the NodeJS runtime, run the command

npm install -g @plurid/deon

or download the appropriate binary

CLI

Usage: deon [options] [command] <file>

read a ".deon" file and output the parsed result

Options:
    -v, --version                                   output the version number
    -o, --output <value>                            output type: deon, json (default: "deon")
    -t, --typed <value>                             typed output (default: "false")
    -f, --filesystem <value>                        allow filesystem (default: "true")
    -n, --network <value>                           allow network (default: "true")
    -h, --help                                      display help for command

Commands:
    convert <source> [destination]                  convert a ".json" file to ".deon"
    environment [options] <source> <command...>     loads environment variables from a ".deon" file and spawns a new command
    confile [options] <files...>                    combine files into a single ".deon" file
    exfile <source>                                 extract files from a ".deon" confile

Service

The deon data parsing can be explored on deon.plurid.com and deon.plurid.com/parse can be used for programmatic POST requests.

curl \
    -X POST \
    -H 'Content-Type: application/deon' \
    -d '{ key value }' \
    https://deon.plurid.com/parse

The response is deon by default, but can be specified through the kind query parameter (json, yaml, toml, or xml).

curl \
    -X POST \
    -H 'Content-Type: application/deon' \
    -d '{ key value }' \
    https://deon.plurid.com/parse?kind=yaml

Conversely, json, yaml, toml, or xml data can be converted to deon through a deon.plurid.com/convert POST request

curl \
    -X POST \
    -H 'Content-Type: application/json' \
    -d '{ "key": "value" }' \
    https://deon.plurid.com/convert

The deon.plurid.com service can also host .deon files to be easily imported or injected into other files from a fixed URL allowing control over the private/public status, the file's revision, or access-aware content (the content of the file changes based on who/what token requests it).

General

A deon is comprised of a required root and none or more, optional leaflinks.

In deon every endleaf value is a string. It is up to the consumer to handle the required type conversions based on the problem domain interface. An advanced use case couples deon with datasign to handle type conversions.

deon supports two types of value groupings, the map and the list.

The root can be map or list-like.

The leaflinks can be strings, maps, or lists.

The maps and the lists can have strings, lists, and maps as values,

The order of the root or of any of the leaflinks is not important.

The per map key names and the leaflinks names are expected to be unique.

When parsed or imported, a .deon file will allow access only to the root. The leaflinks are private as data-details at the file level. By convention, a __leaflinks__ key can be manually added to the root to allow access to the leaflinks if absolutely needed.

Values

An endleaf value, simply called value, is a string of characters, with or without spaces:

{
    key simpleValue
}

{
    key value with spaces
}

A value can be surrounded by singlequotes ' in order to support special characters, such as trailing spaces

{
    key 'value with 4 trailing spaces    '
}

Multi-line values are surrounded by backticks `. The multi-line string is stripped of any whitespace or new lines before the first non-space character and after the last non-space character.

{
    key `
a
multi
line
string
value
        `
}

or linked

{
    #key
}

key `
a
multi
line
string
value
`

Maps

mapName {
    mapKey mapValue
}

A map is comprised of key-value pairs. The deon root is the single base-level map without a mapName.

A mapKey is an A-Za-z0-9_- string of characters. To support special characters (such as space), the mapKey must be surrounded by single quotes, such as

mapName {
    'map Key' mapValue
}

A mapValue starts after the space of the mapKey and continues until the end of the line or until a comma.

mapName {
    mapKey1 map Value 1
    mapKey2 mapValue2
}

mapName {
    mapKey1 map Value 1, mapKey2 mapValue2
}

A mapValue can be a string, a list, or a map.

A mapValue can be an empty string:

mapName {
    mapKey1
    mapKey2 mapValue2
}

mapName {
    mapKey1 '', mapKey2 mapValue2
}

Lists

listName [
    list Value 1
    listValue2
]

A list is comprised of a listName and the list items. The deon root is the single base-level list without a listName.

A list item value starts at the first non-space character after the left square bracket [, or after the previous list item, and ends at the end of the line or at the comma.

Such as

listName [
    list Value 1, listValue2
]

listName [list Value 1, listValue2]

Each list item can be a string, a list, or a map.

A list item can be an empty string:

listName [
    ''
    listValue2
]

listName [
    '', listValue2
]

Structures

A structure is used to specify structured data

{
    aStructure <
        // structure signature
        id, value
    > [
        // first data entry
        one, two
        // second data entry
        three, four
        // third data entry
        five, six
    ]
}

Comments

Single-line comments use the doubleslash //.

// comment outside root
{
    // comment inside root
    key value // comment in-line
}

Multi-line comments use the slashstar /* to start, and the starslash to end */.

/*
    multi
    line
    comment outside root
*/
{
    /*
        multi
        line
        comment inside root
    */
    key value
}

Linking

General

A leaflink is designated using the hash sign #.

The .deon file

{
    key value
}

can be linked thus

{
    key #arbitraryName
}

arbitraryName value

or with shortened linking

{
    #key
}

key value

To support linking with special characters in name, the leaflink must be surrounded by singlequotes '.

{
    #'key with spaces'
}

'key with spaces' value

Dot-access

A leaflink can be dot-accessed:

{
    entities [
        {
            name #entity1.name
        }
    ]
}

#entity1 {
    name The Entity
}

or dot-accessed with shortened link

{
    entities [
        {
            #entity1.name
        }
    ]
}

#entity1 {
    name The Entity
}

in which case, the key will be the last key of the dot-access string.

Name-access

A leaflink can be name-accessed:

{
    entities [
        {
            name #entity1[name]
        }
    ]
}

#entity1 {
    name The Entity
}

or from a list:

{
    entities [
        {
            name #names[0]
        }
    ]
}

#names [
    one
    two
    three
]

The list has a zero-based indexation.

Spreading

A leaflink can be spreaded by tripledots ...:

{
    entities [
        {
            ...#entity1
        }
    ]
}

#entity1 {
    name The Entity
    timestamp 1598425060
}

Spreading overwrites the previously defined keys, if any, with the same name as the keys in the spreaded map.

A map can be spreaded only in another map. A list can be spreaded only in another list. A string can be spreaded in a map and will result in a map where each key equals the index of the character of the string, or can be spreaded into a list and will result in a list where each list item is a character of the string.

{
    entity {
        ...#spread
    }
}

spread abc

wil result in entity having the value:

entity {
    0 a
    1 b
    2 c
}

whereas

{
    entity [
        ...#spread
    ]
}

spread abc

wil result in entity having the value:

entity [
    a
    b
    c
]

Environment variables

A leaflink can represent an environment variable using the #$ syntax. The environment variable will be injected at parse-time:

{
    one #$SOME_ENV_VARIABLE
}

two #$ANOTHER_ENV_VARIABLE

Importing

A .deon file can import another .deon file using the following syntax

import <name> from <path>

Where the name is an arbitrary string, and the path is the path of the targeted .deon file.

The path does not need to have the .deon filename extension specified.

The path can also point to a .json file, and deon will parse it appropriately.

The import imports the root from the targeted .deon file in order to be used as a regular, in-file locally-defined leaflink.

The import statement order in file is not important, although, by convention, they sit at the top of file. Imports will be resolved primarily, before any other action. The import name must be unique among all the other imports and among the in-file locally-defined leaflinks, given that there is no discernible conceptual difference between them.

// file-1.deon

{
    name The Name
}

// file-2.deon

import file1 from ./file-1

{
    name #file1.name
}

The paths of the imported files can be relative filesystem paths, and they will be automatically searched and imported if found, or absolute filesystem paths, if all the used absolute paths are passed to the parser at parse-time.

// file-2.deon

import file1 from absolute/path/file-1

{
    name #file1.name
}

and parsed giving the absolute paths, specific to a file or general using the /* glob-like matcher:

// TypeScript example

import Deon from '@plurid/deon';


const deonFilePath = '/absolute/path/to/folder/file-1.deon';
const deonFilesPath = '/absolute/path/to/folder/';

const loadData = async () => {
    const absolutePaths = {
        // specific file
        'absolute/path/file-1': deonFilePath,
        // or file lookup at runtime
        'absolute/path/*': deonFilesPath,
    };

    const deon = new Deon();
    const data = await deon.parseFile(
        '/path/to/file-1.deon',
        {
            absolutePaths,
        },
    );

    return data;
}

const main = async () => {
    const data = await loadData();

    // use data
    console.log(data);
    // { name: 'The Name' };
}

main();

A path can also be an URL such as

// file-url.deon

import urlFile from https://example.com/url-file.deon

{
    #urlFile.key
}

In order to request URL files from protected routes, an authorization map of authorization tokens can be passed at parse-time with all the domains required by the imports

authorization {
    example.com token
}

with the token being automatically passed into the Authorization: Bearer <token> header of the adequate domain at request-time.

// TypeScript example

import Deon from '@plurid/deon';


const loadData = async () => {
    const authorization = [
        'example.com': 'token', // provide token securely using environment variables
    ];

    const deon = new Deon();
    const data = await deon.parseFile(
        '/path/to/file-url.deon',
        {
            authorization,
        },
    );

    return data;
}

const main = async () => {
    const data = await loadData();

    // use data
    console.log(data);
    // { key: 'data from url file' };
}

main();

The token can be passed at import-time in the .deon file:

import urlFile from https://example.com/url-file.deon with secret-token

{
    #urlFile.key
}

In order not to leak secrets, environment variables should be used:

import urlFile from https://example.com/url-file.deon with #$SECRET_TOKEN

{
    #urlFile.key
}

Injecting

The import statement will always try to parse the filetext into structured data.

In order to get only the filetext, the keyword inject can be used:

inject leaflinkName from /path/to/file.any

{
    key #leaflinkName
}

The arbitrarily-named inject entity can be used as a regular leaflink containing a string.

Similar to the import statement, the inject can target an URL and pass an optional authentication token.

inject file from https://example.com/file
inject secretFile from https://example.com/secret-file with secret-token

{
    key1 #file
    key2 #secretFile
}

In order to keep the .deon file secret-free, the secrets can be injected from a file outside or ignored by the versioning system.

inject secret from file-with-secret.text
inject secretFile from https://example.com/secret-file with #secret

{
    key #secretFile
}

Interpolation

A string value can be interpolated in another string using the #{} syntax.

{
    key value1 #{value2Key} value3
    list [
        value1 #{value2Key}
        value3
    ]
}

value2Key value2-text

which will produce the result

{
    key value1 value2-text text3
    list [
        value1 value2-text
        value3
    ]
}

A deon entity, map, list, string, can be "called" to provision the interpolation values dynamically, on a use-case basis.

aKey {
    subKey value1 #{value2} value3
}

{
    key1 #aKey(
        value2 value2-text
    )
    key2 #aKey(
        value2 value2-different-text
    )
}

Stringifying

A stringify method is implemented in order to convert an in-memory data representation to string. A partial options object can be passed.

interface DeonStringifyOptions {
    readable: boolean;
    indentation: number;
    leaflinks: boolean;
    leaflinkLevel: number;
    leaflinkShortening: boolean;
    generatedHeader: boolean;
    generatedComments: boolean;
}

Parsing

The parse method can receive the following partial options:

interface DeonParseOptions {
    absolutePaths: Record<string, string>,
    authorization: Record<string, string>,
    datasignFiles: string[];
    datasignMap: Record<string, string>;
}

Literals

To handle deon data inside the implementation language, a language-specific literal can be used.

`Javascript`/`Typescript`

import {
    deon,
} from '@plurid/deon';


const main = async () => {
    const data = await deon`
        // handles full-fledged deon data
        // with imports, injects, leaflinks, etc.
        {
            key value
        }
    `;

    // { key: 'value' }
    console.log(data);
}


main();

Rust

use deon::{deon}


fn main() {
    let data = deon!(
        // handles full-fledged deon data
        // with imports, injects, leaflinks, etc.
        {
            key value
        }
    );

    // { key: 'value' }
    println!("{}", data.to_string());
}

Advanced Usage

Datasign Type Conversion

When handling the parsing of .deon data, a .datasign file can be passed to handle the type conversions.

For example, given a JavaScript/TypeScript use case:

// ./Entity.datasign
data Entity {
    name: string;
    age: number;
}

// ./entity.deon
{
    entities: [
        {
            name Entity One
            age 1
        }
        {
            name Entity Two
            age 1.3
        }
    ]
}

// ./index.ts
import Deon from '@plurid/deon';


const deonFile = './entity.deon';
const datasignFile = './Entity.datasign';

const data = Deon.parse(
    deonFile,
    {
        // pass an array of all the .datasign files to be considered for type handling
        datasignFiles: [
            datasignFile,
        ],
        // pass an object of the mappings between the fields from the .deon file
        // and the expected types from the .datasign file
        datasignMap: {
            entities: 'Entity[]',
        },
    },
);

In Use

deon is used in:

developer - configuration file
delog - configuration file
deserve - configuration file
joiner - configuration file
performer - configuration file
pluridoc - plurid plane configuration

Usages

deon can be plugged in into:

Idiomaticity

It appears idiomatic to have three sections in a .deon file, ordered as:

imports;
root;
leaflinks.

The imports feel well-written when written in one line.

The root feels well-written when it has only one level of indentation, and every leaf is a leaflink (for maps or lists) or a string.

For example, the following joiner file:

import otherPackages from ../../path/to/file


{
    #packages
    #package
    #commit
}


packages [
    one
    two
    ...#otherPackages
]

package {
    manager yarn
    publisher npm
}

commit {
    engine git
    combine true
    root packages
    fullFolder true
    divider ' > '
    message setup: package
}

Specifics

`JavaScript` / `TypeScript`

Parsing

The JavaScript / TypeScript can be used in the NodeJS runtime through the Deon object, or the deon template literal.

import Deon, {
    deon,
} from '@plurid/deon';

The parsing of deon data can be achieving asynchronously or synchronously

import Deon, {
    deon,
    deonSynchronous,
} from '@plurid/deon';

const main = async () => {
    const deonData = `
        {
            key value
        }
    `;

    const deonObject = new Deon();
    const parsedObjectAsynchronously = await deonObject.parse(deonData);
    const parsedObjectSynchronously = deonObject.parseSynchronous(deonData);

    const parsedTemplateAsynchronously = await deon`
        {
            key value
        }
    `;
    const parsedTemplateSynchronously = deonSynchronous`
        {
            key value
        }
    `;
}

Synchronous parsing is to be used when the deon data does not rely on import or inject features, naturally asynchronous operations. However, when the parsing operation is to be used in a blockable environment (such as the CLI), synchronous parsing can be used for deon data with imports and injects just as well as asynchronous parsing.

deon data can also be parsed in the browser, or other sandboxed environments, using the DeonPure object, or the deonPure template literal.

import {
    DeonPure,
    deonPure,
    deonPureSynchronous,
} from '@plurid/deon';


const main = async () => {
    const deonData = `
        {
            key value
        }
    `;

    const deonObject = new DeonPure();
    const parsedObjectAsynchronously = await deonObject.parse(deonData);
    const parsedObjectSynchronously = deonObject.parseSynchronous(deonData);

    const parsedTemplateAsynchronously = await deonPure`
        {
            key value
        }
    `;
    const parsedTemplateSynchronously = deonPureSynchronous`
        {
            key value
        }
    `;
}

The deon Pure implementation does not have access to the file system for import and inject features.

Typing

In order to handle the typing of the deon parsed data the typer can be used, which handles the typing in the standard JavaScript/TypeScript fashion, or the customTyper can be used, which requires an aditional, custom typing function.

import Deon, {
    customTyper,
    typer,
} from '@plurid/deon';


const main = async () => {
    const deonData = `
        {
            keyBoolean true
            keyNumber 1
        }
    `;

    const deonObject = new Deon();
    // keyBoolean and keyNumber are typeof 'string'
    const parsedData = await deonObject.parse(deonData);

    // keyBoolean is typeof 'boolean' and keyNumber is typeof 'number'
    const defaultTypedData = typer(parsedData);

    // keyBoolean is typeof 'boolean' and keyNumber is typeof 'string'
    const customTypedData = customTyper(
        parsedData,
        (value) => {
            if (value === 'true') {
                return true;
            }

            return value;
        },
    );
}

Environment Loading

deon can be used to load environment variables at runtime from a .deon file.

Run the function call as soon as possible in the program.

// env-file.deon
{
    ONE one
}

// index.ts
import Deon from '@plurid/deon';


const loadEnvironment = async () => {
    const deon = new Deon();

    // optional
    const options = {
        overwrite: false,
    };

    await deon.loadEnvironment(
        '/path/to/env-file.deon'
        options,
    );
}

const main = async () => {
    await loadEnvironment();

    console.log(process.env.ONE) // one
}

main();

The .deon file that will be used for environment variables can use all the features of deon, however the root must be comprised only of strings, or list of strings, other values will be ignored.

Rust

In order to parse deon data the Deon implementation or the deon! macro can be used.

use deon::{
    Deon,
    deon!,
}

fn main() {
    let deon_data = "
        {
            key value
        }
    ";

    let deon_object = Deon::new();
    let deon_object_parsed = Deon::parse(deon_data);

    let deon_macro_parsed = deon!(
        {
            key value
        }
    );
}