Builder v4.2.1

Current version: 4.2.1

Contents

• About Builder
• Builder Installation
  • Command Line Tool Installation
  • Library Installation
• Builder Syntax
  • Expressions
    • Types
    • Operators
      • Binary
      • Unary
    • Member Expressions
    • Conditional Expressions
    • Variables
      • Variable Definition Order
      • Environment Variables
      • Builder Variables
    • Builder Functions
  • Comments
  • Directives
    • @{...} – Inline Expressions/Macros
    • @set
    • @macro
      • Define A Macro
      • Use A Macro
    • @include
    • @include once
    • @while
    • @repeat
    • @if... @elseif... @else
    • @error
    • @warning
  • Filters
• Include Files
  • Searching The Included File
  • Remote Include Files
    • Supported Remote Resources
    • Caching Remote Files
    • Saving And Reusing Versions Of Remote Files
    • Proxy Access To Remote Files
• Advanced Builder Usage
  • Reproducible Artifacts
    • Builder Variables: Directives
    • Repository Files: Dependencies
  • Including JavaScript Libraries
    • Binding The Context Object Correctly
• Testing
• License

About Builder

Builder combines a preprocessor with an expression language and advanced imports.

There are a number of ways in which you can install Builder depending on how you plan to integrate it into your workflow. Once installed on your computer, you can use it to process your Squirrel application and factory firmware before you transfer the code to an impCentral™ Device Group.

Note The Electric Imp VS Code extension already incorporates Builder and can be used to upload code to Device Groups. If you are using the VS Code extension, there is no need install Builder separately to take advantage of its features.

You can use Builder to pull the contents of separate code files into your main source code files. These additional files might contain library code that you make use of across a number of different products, or they might contain confidential data which you don’t want to keep inside source code files that are managed through a software version control system.

You tell Builder which files to import, and where within your main source code they should be inserted, by using the @include command. Builder is able to access additional files that are stored on your computer or held remotely on an external resource (eg. HTTP/HTTPs server or a repository).

While Builder can be used to insert code this way, it can be used in far more sophisticated ways thanks to its integrated expression processor and programming logic. For example, if you need to generate multiple versions of your application firmware for versions of your product which make use of different imp modules, you can use Builder’s conditional execution features, variables and loops to pull your various code components together at build time and output files that are ready to be transferred to impCentral.

To speed up the process, files that are stored remotely which are not expected to change between builds can be cached for quick re-use. Builder's reproducible artifacts feature makes it possible to store references to all files and variables, so that builds can be re-created for future debugging.

For details on the commands that Builder offers, please see the Directives section. This is part of the Builder Syntax section, which also describes how Builder commands are structured.

Builder Installation

Builder requires Node.js 18.14.2 and above. It can be installed and used by two ways:

• As an npm command line tool
• As an npm library.

Command Line Tool Installation

Install Builder:

npm install -g Builder

Now use Builder’s pleasebuild command to configure the newly installed utility:

pleasebuild [-l] [-D<variable> <value>]
    [--github-token <token>] [--azure-user <username> --azure-token <token>]
    [--bitbucket-server-addr <address>] [--bitbucket-server-user <username> --bitbucket-server-token <token>]
    [--lib <path_to_file>] [--use-remote-relative-includes] [--suppress-duplicate-includes-warning]
    [--cache] [--clear-cache] [--cache-exclude-list <path_to_file>]
    [--save-dependencies [<path_to_file>]] [--use-dependencies [<path_to_file>]]
    [--save-directives [<path_to_file>]] [--use-directives [<path_to_file>]]
    <input_file>

where <input_file> is the path to source file which should be preprocessed and the other options are:

Library Installation

Install Builder:

npm i --save Builder

Now instantiate, configure and execute Builder from within your source code. For example:

const Builder = require('Builder');
const builder = new Builder();

// Specify whether you need line control statements. See the "-l" CLI option.
builder.machine.generateLineControlStatements = <true|false>;

// Cache all files included from remote sources. See the "--cache" CLI option.
builder.machine.useCache = <true|false>;

// Set GitHub credentials. See the "--github-token" CLI option.
builder.machine.readers.github.token = "<PERSONAL_ACCESS_TOKEN>";

// Set Azure Repos credentials. See the "--azure-user" and "--azure-token" CLI options.
builder.machine.readers.azureRepos.username = "<USERNAME>";
builder.machine.readers.azureRepos.token = "<ACCESS_TOKEN>";

// Set Bitbucket Server address and credentials. See the "--bitbucket-server-*" CLI options.
builder.machine.readers.bitbucketSrv.serverAddr = "<ADDRESS>";
builder.machine.readers.bitbucketSrv.username = "<USERNAME>";
builder.machine.readers.bitbucketSrv.token = "<PASSWORD_OR_ACCESS_TOKEN>";

// Path to the file that lists the resources which should be excluded from caching.
// See the "--cache-exclude-list" CLI option.
builder.machine.excludeList = "<PATH_TO_FILE>";

// Interpret a non-absolute path in includes as relative to the current file.
// See the "--use-remote-relative-includes" CLI option.
builder.machine.remoteRelativeIncludes = <true|false>;

// Suppress warning about duplicate includes.
// See the "--suppress-duplicate-includes-warning" CLI option.
builder.machine.suppressDupWarning = <true|false>;

// See the "--save-dependencies" CLI option.
builder.machine.dependenciesSaveFile = <false|"PATH_TO_FILE">;
// See the "--use-dependencies" CLI option.
builder.machine.dependenciesUseFile = <false|"PATH_TO_FILE">;

// See the "--save-directives" CLI option.
builder.machine.directivesSaveFile = <false|"PATH_TO_FILE">;
// See the "--use-directives" CLI option.
builder.machine.directivesUseFile = <false|"PATH_TO_FILE">;

const inputFile = "PATH_TO_YOUR_INPUT_FILE";

// Set the directory of the input file as one of search dirs.
// path.resolve() is used to get the absolute path (requires the path nodejs module)
builder.machine.readers.file.inputFileDir = path.resolve(inputFile);

const result = builder.machine.execute(`@include "${inputFile}"`);
console.log(result);

To understand Builder configuration, please review this source code.

Builder Syntax

Expressions

Types

The following value types are supported in expressions:

• numbers (eg. 1, 1E6, 1e-6, 1.567)
• strings (eg. "abc", 'abc')
• booleans (eg. true, false)
• null

Operators

Builder supports the following operators:

Binary

||  &&  ==  !=  <  >  <=  >=  +  -  *  /  %

Unary

+  -  !

Member Expressions

Membership of an object is expressed using any of the following expressions:

• somevar.member
• somevar["member"]
• ([1, 2, 3])[1]

Conditional Expressions

Builder provides the standard ternary operator (?:) for evaluating basic conditions:

<condition> ? <if_condition_true> : <if_condition_false>

Variables

Variables can be used in Builder expressions. Variable names can contain $, _, latin letters and digits, however they must not start with a digit. Variables can be defined in the following ways:

• Builder's @set directive.
• Your computer's environment variables.
• Pass the option -D<variable name> <variable value> to Builder’s pleasebuild command.

All undefined variables are evaluated as null.

Variable Definition Order

When resolving a variable’s value:

1. Builder looks for its definition among the command line parameters (as -D<variable name> <variable value>) passed to the pleasebuild command.
2. If no such variable definition is found, the code is scanned for @set directive statements preceding the variable usage.
3. If no variable definitions are found in the previous steps, Builder looks in the host environment variables.

Environment Variables

There is no special predicate required to make use of environment variables. Builder looks in the host environment variables to try and resolve the expressions if no command line or local variables have been set.

For example, on a Mac:

server.log("Host home path is @{HOME}");

will print the home directory path of the current user of the system where Builder was executed.

Environment variables differ based on OS. If you wish to use environment variables with Builder, a quick internet search will give you details on how to list the variables currently available on your system and also how to set new variables.

Builder Variables

Builder provides the following pre-defined variables:

• __LINE__ — The line number (relative to the file in which this variable appears). For example:

  Hi from line @{__LINE__}!

• __FILE__ — The name of the file in which this variable appears. For example:

  Hi from file @{__FILE__}!

• __PATH__ — The absolute path (not including file name) to the file where this variable appears. Can contain a URL for remote includes. For example:

  Hi from file @{__PATH__}!

• __REPO_PREFIX__ — The root of the repository resource which is being processed. Is an internal variable of Builder. For example:

  github:electricimp/Builder

• __REPO_REF__ — The git reference (branch name or tag) of the repository resource which is being processed. Is an internal variable of Builder.

• __URL_ROOT__ — The root of the remote weblink resource which is being processed. Is an internal variable of Builder. For example:

• __URL_PATH__ — The path to the file on the remote weblink resource which is being processed, relative to the root of resource. Is an internal variable of Builder.

Builder has two directives @while and @repeat for managing loops. Inside these loops the following variables are available:

• loop.index — 0-indexed iteration counter
• loop.iteration — 1-indexed iteration counter

Usage examples for these variables can be found in the @while and @repeat directive examples.

Builder Functions

Builder provides the following helper functions:

• defined(<variable_name>) — returns true if a variable is defined, false otherwise.
• include(<source>) — includes external source.
• escape(<value>) — escapes special characters in string (\b, \f, \n, \r, \t, \, ', ").
• base64(<value>) — encodes value as base64.
• min(<numbers>) — returns a number equal to the lowest number in the supplied list.
• max(<numbers>) — returns a number equal to the highest number in the supplied list.
• abs(<number>) — returns the absolute value of the supplied number.

Builder also comes with some string functions, based on the JavaScript methods of the same names. These functions are available under the namespace S. The first argument of each function is always the string to be operated on. For documentation on the remaining arguments, please see ‘JavaScript String Methods’.

• S.concat()
• S.endsWith()
• S.includes()
• S.repeat()
• S.split()
• S.startsWith()
• S.substr()
• S.substring()
• S.toLowerCase()
• S.toUpperCase()
• S.trim()
• S.trimLeft()
• S.trimRight()

Comments

Lines starting with @ followed by space or a line break are treated as comments and not added to the output. For example:

@ This is a Builder comment and will not appear in the output

Any text following // and extending to the end of the line will be ignored by Builder* and will not appear in the result output. For example:

@set SOME_STRING = "my string" // This is a Builder comment that will not appear in output

Directives

All of Builder’s directives start with the @ symbol. Don’t include a space or line break between the @ and the required directive’s name as this will be interpreted as a comment.

@{...} Inline Expressions/Macros

This directive evaluates the expression enclosed by braces ({ and }) and inserts the result into the output. The enclosure can be the value of a named variable, an expression or a macro.

Example

The line:

The result is: @{123 * 456}.

results in the following output:

The result is: 56088.

@set

This directive assigns a value or the value of an expression to a variable. Variables are defined in a global context. A value can be any supported type or function.

Example

In this example, we define a number of variables using @set, then use @{...} to create squirrel log messages with those variables. If the following lines are added to your source code:

@ Set Builder global variables
@set SOME_INT    = 10
@set SOME_STRING = "my string"
@set BOOL_VAL    = (12 > 4)
@
@ Use a Builder function to set a global variable
@set MIN_INT min(1, 2, 3)
@
// Use Builder global variables in squirrel log messages
server.log(@{SOME_INT});
server.log("@{SOME_STRING}");
server.log(@{BOOL_VAL});
server.log(@{MIN_INT});

then during processing, Builder will output:

// Use Builder global variables in squirrel log messages
server.log(10);
server.log("my string");
server.log(true);
server.log(1);

@macro

This directive defines a code block with its own parameters. Macros are declared in a global scope. Macro parameters are only available within the macro scope and override global variables with the same name (but do not affect them).

Define A Macro

Note @end can be used in place of @endmacro if you prefer.

Use A Macro

Macros can be used either inline with the @{...} directive, or with the @include directive. When macros are used inline no line-control statements are generated for the output inside the macro scope and trailing newlines are trimmed from the macro output.

Inline Example

Define a macro and use the @{...} directive to create a multi-line string to log in squirrel:

@ Define a macro
@macro some_macro(a, b, c)
    Hello, @{a}!
    Roses are @{b},
    And violets are @{c}
@end
@
// Use an inline Builder macro to create a multi-line string
poem <- @"@{some_macro("username", "red", "blue")}";
server.log(poem);

This results in the following output:

// Use an inline Builder macro to create a multi-line string
poem <- @"    Hello, username!
    Roses are red,
    And violets are blue";
server.log(poem);

Include Example

Define a macro and use it with @include to create a multi-line string to log in squirrel:

@ Define a macro
@macro some_macro(a, b, c)
    Hello, @{a}!
    Roses are @{b},
    And violets are @{c}
@end
@
// Use Builder include and macro directives to create a multi-line squirrel string
poem <- @"
@include some_macro("username", "red", "blue")
";
server.log(poem);

This results in the following output:

// Use Builder include and macro directives to create a multi-line squirrel string
poem <- @"
    Hello, username!
    Roses are red,
    And violets are blue
";
server.log(poem);

Optional Parameter Example

Use a function to configure and use a macro with an optional parameter.

@ Define a macro with an optional parameter
@macro some_macro(a, b, c)
    Hello, @{a}!
    Roses are @{b},
    And violets are @{defined(c) ? c : "of undefined color"}.
@end
@
// Use Builder include and macro directives to create a multi-line squirrel string
poem <- @"
@include some_macro("username", "red")
";
server.log(poem);

This results in the following output:

// Use Builder include and macro directives to create a multi-line squirrel string
poem <- @"
    Hello, username!
    Roses are red,
    And violets are of undefined color.
";
server.log(poem);

@include

This directive can be used to include local files, files from remote resources or macros.

• For a @macro:

• For a local file:

• For a file from remote resource:

The @ character must not be present in the path and name of the file which is being included.

Please see the Include Files section for information about include formats, file searching rules, supported remote resources, examples and other details.

@include once

This directive acts just like @include but has no effect if the specified source has already been included. However, macros are always included.

@while

This directive invokes a loop which only ends when specified conditions are met. You can access Builder’s loop variables within @while loops.

Note @end may be used in place of @endwhile if you prefer.

Example

The following lines, when added to your source code:

@set myvar = 12

@while myvar > 9
    @set myvar = myvar - 1
var: @{myvar}
    loop.index: @{loop.index}
    loop.iteration: @{loop.iteration}
@end

will output:

var: 11
    loop.index: 0
    loop.iteration: 1
var: 10
    loop.index: 1
    loop.iteration: 2
var: 9
    loop.index: 2
    loop.iteration: 3

@repeat

This directive invokes a loop that repeats for a certain number of iterations. You can access Builder’s loop variables within @repeat loops.

Note @end may be used in place of @endrepeat if you prefer.

Example

The following lines, when added to your source code:

@repeat 3
    loop.index: @{loop.index}
    loop.iteration: @{loop.iteration}

@end

will output:

    loop.index: 0
    loop.iteration: 1

    loop.index: 1
    loop.iteration: 2

    loop.index: 2
    loop.iteration: 3

@if... @elseif... @else

This directive provides conditional branching.

Note @end may be used in place of @endif if you prefer.

Example

@if __FILE__ == 'abc.ext'
    // include something
@elseif __FILE__ == 'def.ext'
    // include something else
@else
    // do something completely different
@endif

@error

This directive simply emits an error.

Example

@if PLATFORM == "platform1"
    // platform 1 code
@elseif PLATFORM == "platform2"
    // platform 2 code
@elseif PLATFORM == "platform3"
    // platform 3 code
@else
    @error "Platform " + PLATFORM + " is unsupported"
@endif

@warning

This directive simply emits a warning.

Example

@if PLATFORM == "platform1"
    // platform 1 code
@elseif PLATFORM == "platform2"
    // platform 2 code
@elseif PLATFORM == "platform3"
    // platform 3 code
@else
    @warning "Building for default platform"
    // default platform code
@endif

Filters

The filter operator, |, allows you to pass a value through any of the supported functions.

This is equivalent to:

Example

// Include external HTML piped through the escape processing function
a = "@{include('index.html')|escape}"

// Include an external binary piped through the base64 encoder function
b = "@{include('file.bin')|base64}"

Include Files

This section contains detailed information about usage of the @include and @include once directives to include local or remote files.

Local file is a file on the same system where Builder is running.

Remote file is a file on a remote resource (eg. a repository or a HTTP/HTTPs server). See the Remote Include Files section for the list of supported remote resources.

The file mentioned in the @include or @include once directives may contain a relative or an absolute path.

Examples:

@include "https://example.com/somefile.ext" // absolute path to remote file
@include once "github:electricimp/Promise/promise.class.nut" // absolute path to remote file
@include "c:\somefolder\somefile.ext" // absolute path to local file
@include once "./somefile.ext" // relative path
@include "somefile.ext" // relative path

The next section clarifies how Builder searches the included file.

Searching The Included File

Processed file is a file currently being processed by Builder, it can be local or remote, it contains the @include or @include once directive(s).

Path to the processed file is an absolute path to the processed file.

Path in the include is the path mentioned in the @include or @include once directive, it can be an absolute path to local file, an absolute path to remote file or a relative path.

Full path to the include file is an absolute path to the file which will be included by Builder.

The below rules define how Builder searches the included file:

Absolute Path To Remote File

If the path in the include is an absolute path to the remote file, it is considered as the final path to the include file. If the file is not found there, Builder reports an error.

Examples:

@include "https://example.com/somefile.ext" // absolute path to remote file
@include once "github:electricimp/Promise/promise.class.nut" // absolute path to remote file

Absolute Path To Local File

If the path in the include is an absolute path to the local file, it is considered as the final path to the include file. If the file is not found there, Builder reports an error.

Examples:

@include "c:\somefolder\somefile.ext" // absolute path to local file
@include "/home/user/someuser/somefolder/somefile.ext" // absolute path to local file. But see the note below.

Note: Under some conditions a path in the include which starts from the "/" symbol is processed by the special rule described below.

"/" As Root Of Remote Resource

If

• the first symbol of the path in the include is "/"
• and the processed file is a remote file (ie. the file on a remote resource)
• and the --use-remote-relative-includes option is specified

then the final path to the include file is a concatenation of the root of remote resource and the path in the include. If the file is not found there, Builder reports an error. See the Remote Include Files section for the root definition of the supported remote resources.

Examples:

The processed file: "https://example.com/folderA/folderB/somefile.nut"

It contains:
@include "/folderC/anotherfile.nut"

The final path to the include: "https://example.com/folderC/anotherfile.nut"

The processed file: "github:someuser/somerepo/folderA/folderB/somefile.nut@develop"

It contains:
@include "/folderC/anotherfile.nut"

The final path to the include: "github:someuser/somerepo/folderC/anotherfile.nut@develop"

Relative Path To Remote File

If

• the path in the include is a relative path
• and the processed file is a remote file (ie. the file on a remote resource)
• and the --use-remote-relative-includes option is specified

then the final path to the include file is a concatenation of the path to the processed file and the path in the include. If the file is not found there, Builder reports an error.

Examples:

The processed file: "https://example.com/folderA/folderB/somefile.nut"

It contains:
@include "anotherfile.nut"

The final path to the include: "https://example.com/folderA/folderB/anotherfile.nut"

The processed file: "github:someuser/somerepo/folderA/folderB/somefile.nut@v1.2.3"

It contains:
@include "folderC/anotherfile.nut"

The final path to the include: "github:someuser/somerepo/folderA/folderB/folderC/anotherfile.nut@v1.2.3"

Relative Path To Local File

If

• the path in the include is a relative path
• and the processed file is a local file (or it is a remote file but the --use-remote-relative-includes option is not specified)

then Builder makes the following steps to find the include file:

1. Only if the processed file is a local file, the final path to the include file is a concatenation of the path to the processed file and the path in the include. If the file is not found there, moves to the next step.
2. The final path to the include file is a concatenation of the path to the file specified as the <input_file> parameter of the pleasebuild command and the path in the include. If the file is not found there, moves to the next step. Note: If you use Builder as a library and want this search step to work, make sure you specify the input file path (builder.machine.readers.file.inputFileDir variable) in your source code within the Builder initialization routine. See the Library Installation section to learn more.
3. The final path to the include file is a concatenation of the path to the directory from where the pleasebuild command has been called and the path in the include. If the file is not found there, Builder reports an error.

Examples:

The processed file: "/home/user/someuser/folderA/somefile.ext"

It contains:
@include "folderX/anotherfile.nut"

The "pleasebuild" command is called from: "/home/user/someuser/folderB"
With the <input_file> parameter: "folderC/initialfile.nut"

The anotherfile.nut file will be searched sequentially in the following locations:
1. "/home/user/someuser/folderA/folderX/"
2. "/home/user/someuser/folderB/folderC/folderX/"
3. "/home/user/someuser/folderB/folderX/"

Remote Include Files

Supported Remote Resources

Builder supports file includes from the following types of remote resources:

HTTP/HTTPs Server

For includes from weblinks.

Format:

@include "http://<server>[:<port>]/<path>"
@include "https://<server>[:<port>]/<path>"
@include once "http://<server>[:<port>]/<path>"
@include once "https://<server>[:<port>]/<path>"

where:

• server is the host name or IP address.
• port is the port number.
• path is the path to file.

Examples:

@include "https://example.com/somefile.ext"
@include once "http://example.com/folderA/folderB/somefile.nut"

Root of the remote resource:

http://<server>:<port>
https://<server>:<port>

GitHub Repository

For includes from github.com.

Format:

@include "github:<user>/<repo>/<path>[@<ref>]"
@include once "github:<user>/<repo>/<path>[@<ref>]"

where:

• user is the user/organization name.
• repo is the repository name.
• path is the path to file.
• ref is the git reference (branch name or tag, defaults to master).

Examples:

@include "github:electricimp/Promise/promise.class.nut" // head of the default branch
@include "github:electricimp/Promise/promise.class.nut@develop" // head of the develop branch
@include once "github:electricimp/Promise/promise.class.nut@v3.0.1" // tag v3.0.1

Root of the remote resource:

github:<user>/<repo>

Authentication:

Authentication is optional. It is required to access private repositories only. But please note that when you use authentication, the GitHub API provides much higher rate limits.

For the authentication you need to provide:

• a GitHub username (--github-user option)
• a GitHub personal access token or a password, which is less secure and not recommended (--github-token option).

Bitbucket Server Repository

For includes from Bitbucket Server. Note, this is not the same as Bitbucket Cloud, includes from Bitbucket.org are not supported.

Builder can work with only one Bitbucket server at a time. Its address must be specified (--bitbucket-server-addr option). Server version 5.3.0 or above is supported.

Format:

@include "bitbucket-server:<project>/<repo>/<path>[@<ref>]"
@include "bitbucket-server:~<user>/<repo>/<path>[@<ref>]"
@include once "bitbucket-server:<project>/<repo>/<path>[@<ref>]"
@include once "bitbucket-server:~<user>/<repo>/<path>[@<ref>]"

where:

• project is the project name. Should be used to include files from project repositories.
• user is the user name. Should be used to include files from personal repositories. Note, the user name must be prepended with ~. Eg, for the user name John it should be ~john.
• repo is the repository name.
• path is the path to file.
• ref is the git reference (branch name or tag, defaults to master).

Examples:

@include "bitbucket-server:Tools/Promise/promise.class.nut" // head of the default branch
@include once "bitbucket-server:Tools/Promise/promise.class.nut@develop" // head of the develop branch
@include "bitbucket-server:~john/Promise/promise.class.nut@v3.0.1" // tag v3.0.1

Root of the remote resource:

bitbucket-server:<project>/<repo>
bitbucket-server:~<user>/<repo>

Authentication:

Authentication is optional. It is required to access private repositories only.

For the authentication you need to provide:

• a Bitbucket Server username (--bitbucket-server-user option)
• a Bitbucket Server personal access token or a password, which is less secure and not recommended (--bitbucket-server-token option).

Azure Repository

For includes from Azure Repos.

Format:

@include "git-azure-repos:<org>/<project>/<repo>/<path>[@<ref>]"
@include once "git-azure-repos:<org>/<project>/<repo>/<path>[@<ref>]"

where:

• org is the organization name.
• project is the project name.
• repo is the repository name.
• path is the path to file.
• ref is the git reference (branch name or tag, defaults to master).

Examples:

@include once "git-azure-repos:org/project/repo/path/some.class.nut" // head of the default branch
@include "git-azure-repos:org/project/repo/path/some.class.nut@develop" // head of the develop branch
@include "git-azure-repos:org/project/repo/path/some.class.nut@v3.0.1" // tag v3.0.1

Root of the remote resource:

git-azure-repos:<org>/<project>/<repo>

Authentication:

Authentication is optional. It is required to access private repositories only.

For the authentication you need to provide:

• an Azure Repos username (--azure-user option)
• an Azure Repos personal access token (--azure-token option).

Local Git Repository

For includes from Git repositories hosted locally. Note, even as files are local, Builder considers them as files from a remote resource when searching the file to include.

Format:

@include "git-local:<path>[@<ref>]"
@include once "git-local:<path>[@<ref>]"

where:

• path is the path to file.
• ref is the git reference (branch name, tag or commit, defaults to current branch which the local git repository is set to).

Examples:

@include once "git-local:/path/to/repo/and/file/some.class.nut" // head of the default branch
@include once "git-local:/path/to/repo/and/file/some.class.nut@develop" // head of the develop branch
@include "git-local:/path/to/repo/and/file/some.class.nut@3.0.1" // tag v3.0.1
@include "git-local:/path/to/repo/and/file/some.class.nut@c13c59e96f3f6a37f75f9e520d0fdc5591e0ba83" // concrete commit

Root of the remote resource:

Is determined by Builder.

Notes:

• Caching is not applicable to local Git files.
• If there are uncommitted changes, they will not be seen by Builder. Hence @include "git-local:<path>" (without <ref>) is not interchangeable with @include "<path>".
• There are local and remote branches in Git. If you want to create a local remote-tracking branch from a remote branch, you can use, for example, the git checkout <remote_branch_name> command. For more info, see the Git docs.
• To include a remote git branch, you should specify the name of the remote repository in the ref part of the directive. For example:

@include "git-local:/path/to/repo/and/file/some.class.nut@develop" // include local git branch
@include "git-local:/path/to/repo/and/file/some.class.nut@origin/develop" // include remote git branch from the "origin" remote repository

Caching Remote Files

To reduce compilation time, Builder can optionally cache files included from a remote resource. If the file cache is enabled, remote files are cached locally in the .builder-cache directory. Cached resources expire and are automatically invalidated 24 hours after their addition to the cache.

To turn the cache on, pass the --cache or -c option to Builder. If this option is not specified, Builder will not use the file cache even if the cached data exists and is valid — Builder will continue to query remote resources on every execution.

To reset the cache, use both the --cache and the --clear-cache options.

If a resource should never be cached, it needs to be added to the exclude-list.builder file (see the example below). You can use wildcard characters to mask file names.

Note: Non-identical paths, even if they point to absolutely the same location, are treated as different paths. Eg., if you include /a/b/1.nut, it becomes cached, and then you include /a/b/../b/1.nut, it will not be found in the cache.

Wildcard Pattern Matching

Pattern matching syntax is a similar to that of .gitignore. A string is a wildcard pattern if it contains '?' or '*' characters. Empty strings or strings that starts with '#' are ignored.

A '?' symbol matches any single character. For example, bo?t.js matches boot.js and boat.js, but doesn't match bot.js.

A '*' matches any string, that is limited by slashes, including the empty string. For example, /foo/*ar matches /foo/bar, /foo/ar and /foo/foo-bar, but doesn't match /foo/get/bar or /foo/bar/get.

Two consecutive asterisks ** in patterns matched against full pathname may have special meaning:

• A leading ** followed by a slash means match in all directories. For example, **/foo matches file or directory foo anywhere, the same as pattern foo. **/foo/bar matches file or directory bar anywhere that is directly under directory foo.

• A trailing /** matches everything inside. For example, abc/** matches all files inside directory abc.

• A slash followed by two consecutive asterisks then a slash matches zero or more directories. For example, a/**/b matches a/b, a/x/b, a/x/y/b and so on.

• Other consecutive asterisks are considered invalid.

Examples:

# Avoid caching a specific file
github:electricimp/MessageManager/MessageManager.lib.nut

# Exclude all electricimp repos
github:electicimp/**

# Exclude all tagged files or files from the specific branches from the cache
github:*/**/*@*

Saving And Reusing Versions Of Remote Files

It is possible to save the versions of all remote files which are used during the current run of Builder, and reuse exactly that versions later. This is applicable for files from repositories only.

See the Repository Files: Dependencies section for the details.

Proxy Access To Remote Files

To specify a proxy that should be used when you are including files from remote resources, set the environment variables HTTP_PROXY/http_proxy and/or HTTPS_PROXY/https_proxy for HTTP and HTTPS protocols respectively.

For example, to operate through a proxy running at IP address 192.168.10.2 on port 3128 for HTTP requests, you should set the environment variable: HTTP_PROXY='http://192.168.10.2:3128'. All of Builder’s HTTP requests will now go through the proxy.

Advanced Builder Usage

This section contains information that will help you work with Builder more effectively, but may not be needed for more basic Builder tasks.

Reproducible Artifacts

It is possible to save the build configuration data used for preprocessing a source file in order to create an identical source file again later with that saved configuration. Builder variable definitions are saved in a ‘directives.json’ file, and references to the concrete versions of repository files and libraries are stored in a ‘dependencies.json’ file.

Builder Variables: Directives

The --save-directives [<path_to_file>] and --use-directives [<path_to_file>] options are used to, respectively, save and reuse Builder variable definitions. The definitions are saved in a JSON file. If a file name is not specified, the directives.json file in the local directory is used. These options are processed the similar way as the --save-dependencies and --use-dependencies options, above.

When the --use-directives [<path_to_file>] option is used, the saved Builder variable definitions are merged with definitions specified by -D<variable> <value> options.

A typical directives.json file looks like this:

{
  "Variable0": "value0",
  "Variable1": "value1"
}

Repository Files: Dependencies

--save-dependencies [<path_to_file>] and --use-dependencies [<path_to_file>] options are used to save and to reuse, respectively, references to concrete versions of repository files and libraries. The references are saved in a JSON file. If a file name is not specified, the dependencies.json file in the local directory is used. Every reference consists of repository file URL and:

• Git Blob ID (Git Blob SHA) — for GitHub files Note It is possible to obtain the Git Blob ID of a GitHub file using the following git command: git hash-object <path_to_file>
• Git Commit ID (Git Commit SHA) — for Bitbucket Server, Azure Repos and Git Local files

For more information, please see the Git Manual and the Git API.

These options are processed the following way:

• If only --save-dependencies [<path_to_file>] is specified, the references to all source files retrieved from repositories are saved in the provided JSON file (or dependencies.json).
• If only --use-dependencies [<path_to_file>] is specified, the source files from repositories are retrieved using the references read from the provided JSON file (or dependencies.json).
• If both --save-dependencies [<path_to_file>] and --use-dependencies [<path_to_file>] are specified, then:
  1. The source files from repositories are retrieved using the references read from the JSON file passed to the --use-dependencies option (or dependencies.json).
  2. If the source code contains @includes for files from repositories which have not yet been retrieved, they are retrieved now.
  3. Builder performs the preprocessing operation.
  4. References to all source files retrieved from repositories are saved in the JSON file passed to the --save-dependencies option (or dependencies.json).

Note If --save-dependencies is specified, the --cache option is ignored. If --use-dependencies is specified, the --cache option does not affect the files referenced in the dependency file.

A typical dependencies.json file looks like this:

[
  [
    "github:ProjectA/repositoryA/fileA",
    "2ff017dc92e826ad184f9cdeadd1a2446f8d6032"
  ],
  [
    "github:ProjectB/repositoryB/fileB",
    "a01b64f9ce764f226f52c6b9364396d4a8bd550b"
  ],
  [
    "bitbucket-server:projectC/repositoryC/fileC",
    "4bc4024f1f2ad99e8bd2ade73d151912e031d1f5"
  ],
  [
    "git-azure-repos:org/projectD/repositoryD/fileD",
    "d1ccee9ed6e250c6d5e1f052107125659d3ba9d0"
  ],
  [
    "git-local:/path/to/repo/and/fileD",
    "c13c59e96f3f6a37f75f9e520d0fdc5591e0ba83"
  ]
]

Note: Non-identical paths, even if they point to absolutely the same location, are treated as different paths. Eg., if you have saved dependencies for /a/b/1.nut and then you include /a/b/../b/1.nut - the dependencies will not be applied to this file.

Including JavaScript Libraries

Builder can accept JavaScript libraries to add functionality to its global namespace. The library should export an object, the properties of which will be merged into the global namespace. For example, to include a function, upper(), to convert strings to uppercase, define your library file like so:

module.exports = {
    upper: (s) => s.toUpperCase()
};

Now include the function within directives in your input file:

@{upper("warning:")}
@{upper(include("warning.txt"))}

Finally, run Builder with the option --lib path/to/your/lib/file.

Binding The Context Object Correctly

Functions called by Builder will be called with their this argument set to a Builder context object. Within the context object, Builder variables like __FILE__, functions like max(), and other included library functions will be made available at the top level. Variables defined in your input code with @macro or @set will be available under the key globals.

Ignoring the binding of this may cause unexpected behavior, for example when calling methods on objects. Take the following example library:

class MyClass {
    constructor(str) {
        this._str = str;
    }

    getStr() {
        return this._str;
    }
}

myObject = MyClass("my text");

module.exports = {
    myObject
};

Attempting to use this library with the directive @{myObject.getStr()} will not deliver the expected behavior because this in getStr() will be set to a Builder context object and not to myObject. When calling class methods ensure they have been bound to the correct value of this:

class MyClass {
    constructor(str) {
        this._str = str;
    }

    getStr() {
        return this._str;
    }
}

myObject = MyClass("my text");

module.exports = {
    getStr: myObject.getStr.bind(myObject)
};

Testing

When running tests locally, please test on both Windows and macOS. All environment variables are optional. However, if you are working with @includes from GitHub and do not provide GitHub credentials, rate limits imposed by GitHub may cause test failures. The default for SPEC_LOGLEVEL is error.

npm install
SPEC_LOGLEVEL=<debug|info|warning|error>
SPEC_GITHUB_TOKEN=<GitHub personal access token>
npm test

Note 1: The standard set of tests doesn't include Bitbucket Server, Azure Repos and Git Local (but see the Note 2) integration testing. To run Bitbucket Server, Azure Repos or Git Local tests, please see the sections below.

Note 2: The standard set of tests uses Git Local in several tests for testing the overall behavior of Builder. They require the SPEC_GIT_LOCAL_REPO_PATH variable to be set (see the Git Local section below). These are optional tests so they will be skipped if that variable is not set.

Note 3: There are several tests that require access to the root of the filesystem (or to the root of the disk C: on Windows). They will create/remove there a directory named builder_test_g2e5r6uh, so please make sure you don't have some important data in such directory.

Bitbucket Server

Prerequisites: 1. A running instance of Bitbucket Server 1. A clone of Builder repo placed on this server 1. If the server / repo are not public, an account (username and password / token) with permissions to access the repo

npm install
SPEC_LOGLEVEL=<debug|info|warning|error>
# E.g., "https://bitbucket-srv.itd.example.com"
SPEC_BITBUCKET_SERVER_ADDRESS=<Bitbucket Server address>
# Format: "<project>/<repo>". E.g., "myProj/BuilderClone"
# If the repo belongs to a user (not to a project), the format is: "~<user>/<repo>". E.g., "~john/BuilderClone"
SPEC_BITBUCKET_SERVER_REPO_PATH=<Path to the cloned Builder repo on the server>
SPEC_BITBUCKET_SERVER_USERNAME=<Bitbucket Server username>
SPEC_BITBUCKET_SERVER_TOKEN=<Bitbucket Server password/access token>
npm run test:bitbucket-server

Azure Repos

Prerequisites: 1. A clone of Builder repo placed at Azure Repos 1. An account (username and token) with permissions to access the repo

npm install
SPEC_LOGLEVEL=<debug|info|warning|error>
# Format: "<org>/<project>/<repo>". E.g., "myOrg/myProj/BuilderClone"
SPEC_AZURE_REPOS_REPO_PATH=<Path to the cloned Builder repo at the Azure Repos>
SPEC_AZURE_REPOS_USERNAME=<Azure Repos username>
SPEC_AZURE_REPOS_TOKEN=<Azure Repos access token>
npm run test:azure-repos

Git Local

Prerequisites: 1. A clone of Builder repo placed locally

npm install
SPEC_LOGLEVEL=<debug|info|warning|error>
SPEC_GIT_LOCAL_REPO_PATH=<Path to the root of the cloned Builder repo>
npm run test:git-local

License

Builder is licensed under the MIT License.