@oscars/ftool v3.23.0

FTool

What is it?

FTool is our new global JS tooling package for the Focus repositories. It contains the tooling for our code, from testing and building to releasing, and everything else we require done in terms of frontend tooling. It is essentially the focus-tooling Node tooling rewritten as a much more easy-to-use global package, with extra features, and the latest updates.

No more tooling.js, just FTool. The same commands that existed in the Node/Make tooling are in FTool, as well as some new ones!

Installation

FTool is currently just a git repo, not it's own defined NPM package on a registry, hence installing it is a bit roundabout:

git clone https://git.nflex.io/Focus/ftool
cd ftool/
npm install
./build/ftool.js setup

To update the version of FTool you're using, simply pull the repo and rerun npm install. FTool is built in the postinstall script, so it gets rebuilt every time you npm install.

You need to refer to FTool via it's built JS file in the 1st instance to set it up. It will add an alias to your profile, then ask you to either reload your shell or execute your updated profile. Once you have done this you will have access to FTool via an alias, so you can use it anywhere on your system.

Usage

Using FTool is simple, all the commands that previously existed are the same, and there are some new ones. Simply call FTool with the name of the command you want. Use the -v flag for extra debugging info. use -h for help. As an example, say you wanted to build a release tarball, in the old tooling this would be:

make build-release

But in FTool, it would be done like this:

ftool build-release

See how these are similar? This was intentional, to enable smooth crossover. All the commands are the same, just called in this new way.

Here is a help output for a focus app; this shows the available commands, and provides some context as to what they do. Please note that this is just the tooling that a focus application has access to, and the toolchain changes depending on what type of repository you are working with.

Usage: ftool <command1> <command2> [-h] [-v] [-f ["flag1" "flag2"]]

Commands:
    build-assets                              Copies over fonts and images to the build dir
    build-latest                              Builds a .tgz of the latest dirty tag in git
    build-release                             Builds a .tgz of the latest clean tag in git
    bump-major                                Bumps to a new major version
    bump-minor                                Bumps to a new minor version
    bump-patch                                Bumps to a new patch version
    test-coverage-statements                  Gets the coverage of statements
    test-coverage-branches                    Gets the coverage of branches
    test-coverage-lines                       Gets the coverage of lines
    test-coverage-functions                   Gets the coverage of functions
    test-coverage-ignored                     Gets the coverage of ignored
    git-check-clean                           Check you have no uncommited changes
    git-check-on-develop                      Check you are on branch develop
    git-check-on-master                       Check you are on branch master
    git-check-on-release                      Check you are on a valid release branch
    git-check-develop-up-to-date-with-remote  Check you are up to date
    check-git                                 Check you have no changes, are on develop, and up to date
    do-release                                Create and push a new minor release
    do-release-major                          Create and push a new major release
    do-release-patch                          Create and push a new patch release
    sec-audit                                 Run the security audit
    sec-audit-publish                         Publish the security audit
    notify-slack                              Send details of the latest clean release to Slack
    test-jshint                               Run the linter
    test-karma                                Run the karma tests directly
    test-complete                             Run the complete karma suite
    utl-api                                   Updates your package.json to the latest version of api
    utl-common                                Updates your package.json to the latest version of common
    utl-core                                  Updates your package.json to the latest version of core
    utl-tooling                               Updates your package.json to the latest version of tooling
    utl-utils                                 Updates your package.json to the latest version of utils
    get-latest-tag                            Get the latest dirty tag
    name                                      Get the name of the repo

Flags:
  --version      Show version number               [boolean]
  -f, --flag     A flag to alter an ftool process    [array]
  Options:
        clean-tag-build  Prevents building the repo, unless the tag is clean
        skip-pypi-check  Disables checking the PyPI server for the package before build
        skip-publish     Disables publishing to PyPI
        skip-build-test  Skip testing repo when building [NOTE: WILL PUBLISH UNTESTED BUILD]
  -v, --verbose  Print out extra info              [boolean]
  -h, --help     Show help                         [boolean]

Tooling Design

Node API & Lodash FP

Node APIs are used in FTool, mainly fs and child_process. fs allows FTool to read and write files, as well as read directories. It uses child_process to execute shell commands in a synchronous and asynchronous way.

For the design and structure of FTool, there is a flavour of the Lodash we all know and tolerate. It's called lodash FP. Take a look at the docs!

It's a more functional programming oriented type of lodash, which makes heavy use of functions like flow, map, filter, and reduce. It looks rather daunting at first, but it is imperative to understand how these functions work, in order to have a full understanding of how FTool works.

Structure & Build Process

FTool is structured in an basic way, that hopefully makes sense. This is what it looks like:

ftool/
    package.json
    webpack.config.js
    src/
        ftool.js
        shell-plugin.js
        tools/
            git.js
            build.js
            test.js
            etc...

There are a couple files in there worth note, other than ftool.js and it's tools.

webpack.config.babel.js

The way FTool is built is somewhat different to how you would structure a normal Focus application or library. Firstly, it is a UMD module, which means it can be loaded via multiple module loader methods. In this case, it's mainly so that Node can run it directly. This is also why the target is set to Node.

In the externals, it transforms the node_modules list into an object with the key and value matching (and removing binary files), which allows it to resolve the externals while also allowing for the way tools are loaded in, via a require context. This allows for bringing in all of the tools in the tools directory.

The config uses 2 plugins; the BannerPlugin is used to add a shebang to the top of the built file, which indicates to the shell you are using to use Node to execute this code. This assumes your system has the env utility in the /usr/bin/ directory. If this is wrong, edit it to point to your env executable.

The config uses a module rule to load both itself (the config), and the actual ftool code via the JS compiler Babel. This is primarily to utilise new, cleaner features where possible, and make the code as readable as possible.

The Chain Operator (|>)

This code has been rewritten using the arrow (|>) chaining operator, instead of the usual flow function in lodash-fp. This is mainly for aesthetic reasons, it improves code cleanliness, and makes it much more easy to visualise the data going through this chain of functions.

The chain operator is loaded in via the babel plugin @babel/plugin-proposal-pipeline-operator tc39 propoal babel docs. This is added to Babel to allow it to understand the new syntax.

The babel plugin for the chain operator is the only plugin used at the moment. It is using the default (minimal) proposal. This means that chaining cannot be done with asynchronous functions or async/await, so anything using it must be synchronous.

shell-plugin.js

The Shell Plugin is an implementation of shell command execution that is compatible with webpack 4. It uses the new Tapable API that Webpack 4 uses, so it does not throw a DeprecationWarning as the plugins currently available for this purpose do. It takes an array of commands to run, and executes them after Webpack is done building FTool.

The functionality here could be expanded if needed, to execute shell commands at any point in the build process. This can be done by tapping into Webpack via the hooks it provides hook docs.

Because the afterEmit hook used in the Shell Plugin has the Promise Tap, we can run these commands asynchronously, by wrapping the Node child_process exec function in a wrapper, called PromiseExecutorGenerator, which then gets mapped onto all the commands in a Promise.all, that then gets passed back to the Tap.

Not sure whether to spin this off into its own separate repo, not really needed unless this concept can be built upon for other purposes?

FTool.js

FTool is designed to firstly obtain variables about the current directory, things like; git details, name, type, dependencies, etc. This is the main set of information that FTool needs. This is compiled into an object called toolEnv. Obtuse name, i know.

Functions & Control Flow

A lot of the code in Ftool can be understood by logical reasoning, they are primarily composed functions, that pipe data through a series of relatively simple statements. A lot of these functions are defined at the top of the code, their names imply what they do. Some of them are more complex than others, but they are written to transform data structures in a readable way.

In these first few functions, probably the hardest to understand is getRepoType. It takes a list of repos organised by type, and a repo name, and returns what repo it is in, or undefined. Try to visualise the data going through each function, and grasp how these come together to do something useful.

Another important function to note, that you will find is used only in the toolEnv, which means it is used in the tools, is execPromise. It's a wrapper around Node's asynchronous exec function, that executes the command, pipes the stdout to the console, and returns a promise that will resolve when done, or reject if there was an error.

A word on reducers

There are some functions defined in their own little section, called reducers. They use the reduce function in lodash-fp, which takes 3 arguments in a curried fashion. These are the reducing function, the starting value, and the data, in that order. In this case, these reducing functions have their reducing functions and starting values defined already.

Since reduce is a curried function, that only executes when all of it's arguments have been supplied, the functionality is defined and the starting value given, the functions are unary - just expecting data. The functions are given a name that gives an idea about what the reducer does, but it can be implied via reasoning about the function.

Argument Parsing

Ftool uses a npm module called yargs-parser to parse the command line options. This returns an object (called argv in FTool) that contains the command line options in an easy to use form. Parsing the arguments into what's needed is not done though.

Firstly, the debugging is disabled if the -v flag is not set. Next the given flag(s) need to be checked for validity. This is what the arrow statement assigned to cmdOptsValid does. It makes sure that the flag(s) are defined as an option in the validOptions array at the top, except if you are in the focus main repo, in which case the flag is used to give a name to a new app or lib (the focus tooling ignores any flags set other than the first).

If the command options are valid, the options are transformed from an array of keys to an object with said keys set to true. This is done by chaining a map and on of the reducers mentioned earlier, called rArrayToObject.

Tools

The tools are designed to be self-contained functions, that can be executed with no arguments, to perform some specific task, given the toolEnv. They may import other pieces of code, including the Node API, or whatever else is needed.

In the tooling that has been created, each set of tools that can be thought of as performing closely related functions, such as releasing code, building code, or writing to Slack for example. This set of tools are created as a series of functions inside of one outer unary function that takes the toolEnv, making everything that the tooling needs available (in scope), hence why the actual tooling functions do not take arguments.

The tooling functions are not simply returned by the outer unary function. There is a specific structure that needs to be returned, to ensure the help and the functions are detected, transformed, and executed properly (see below).

In theory, the tooling file could contain any JS that returns the tool structure, and performs the desired function. There are some different approaches taken in the current toolset, as some tools are very similar, differing only in 1 specific variable, i.e the bump tooling, and the update to latest (utl) tooling.

This being said, the structure of the outer unary function is the standard design used in the current tooling set.

Tool Structure

{
        // Add all supported tooling sets separated by spaces
        'supportedType': 'app lib colo ftool repo-list',
        // Add your tooling functions here, with descriptive names
        'tool_func_1': func_1,
        'tool_func_2': func_2,
        'tool_func_3': func_3,
        ...,
        'helpMessages': {
            // You need a help message for each tooling function, the names must match.
            'tool_func_1': 'help_msg_1',
            'tool_func_2': 'help_msg_2',
            'tool_func_3': 'help_msg_3',
            ...
        }
}

Notes

Tools are not using the chain operator yet, needs to be done when more time is available.

If you need a tool, try to make it! Making good use of -v and dbgLog to debug is how this tooling was written, it should be usable for writing tools.

FTool exists to provide a stable way to get a lot of useful variables about the environment, most, if not all of what you would need to write frontend tooling. Even if you're missing something, it can be added.