@ndcode/disk_build v0.1.1

Disk Build helper

An NDCODE project.

Overview

The disk_build package exports a function disk_build(pathname, build_func, diag), which given a pathname to an existing source file, will generate a pathname for a corresponding on-disk output file, check whether this exists and is newer than the original, and if not, rebuild it via a caller-provided function.

Calling API

The interface for the disk_build-provided helper function disk_build() is:

await disk_build(pathname, build_func, diag) — checks if the source file given in pathname exists, generates the corresponding output pathname and dependency-file pathname, reads the dependency file if possible, checks the output is up-to-date with respect to all dependencies, and if not, generates a temporary output pathname temp_pathname, calls the user-provided callback function build_func(temp_pathname) to build the output into the temporary, possibly writes a new dependency file, then renames everything into place.

Finally, it returns an dictionary {deps: ..., pathname: ...} where deps is a list of dependencies not including the original, and pathname is the output file; these might come from the existing disk files or newly written ones.

The interface for the user-provided callback function build_func() is:

await build_func(temp_pathname) — the user must create the file temp_pathname and then return, or alternatively an exception can be thrown. The return value can be either undefined if no dependency file should be written, or a list of pathnames referred to (not including the original).

About asynchronicity

There is nothing too complicated here, both disk_build() and build_func() are asynchronous and thus return a Promise, and when the inner Promise resolves, the outer Promise resolves in turn. Either level throws exceptions, with various causes such as disk errors, file not found, syntax error, etc.

There is no protection against multiple clients trying to access the same file at once while the asychronous building operation proceeds. This is because the accesses to disk_build() are supposed to be wrapped in a BuildCache, which will exclude multiple access. There was no point making disk_build() keep a dictionary of operations in progress when BuildCache already does this, since disk_build() does not otherwise need any state kept in RAM (it has the disk).

Usage example

Simple usage, showing use of the clean-css CSS minimizer with disk_build(), in such a way as to return a string containing the minified CSS from the given input file described by pathname, without re-converting if already converted:

let CleanCSS = require('clean-css')
let disk_build = require('@ndcode/disk_build')
let fs = require('fs')
let util = require('util')

let clean_css = new CleanCSS({returnPromise: true})
let fs_readFile = util.promisify(fs.readFile)
let fs_writeFile = util.promisify(fs.writeFile)

let get_css_min = async pathname => {
  let render = await disk_build(
    pathname,
    async temp_pathname => {
      let render = await clean_css.minify(
        await fs_readFile(pathname, {encoding: 'utf-8'})
      )
      return /*await*/ fs_writeFile(
        temp_pathname,
        render.styles,
        {encoding: 'utf-8'}
      )
    },
    true // diagnostics on
  )
  return /*await*/ fs_readFile(render.pathname)
}

Since it is rather common that various minimizers and template renderers return an dictionary containing the result of processing plus some other information, and so does disk_cache(), we've adopted the convention of putting the result of such calls in a variable called render, despite multiple use of the name.

A more complicated example using a BuildCache to exclude multiple callers:

let BuildCache = require('@ndcode/build_cache')
let CleanCSS = require('clean-css')
let disk_build = require('@ndcode/disk_build')
let fs = require('fs')
let util = require('util')

let build_cache = new BuildCache(true) // diagnostics on
let clean_css = new CleanCSS({returnPromise: true})
let fs_readFile = util.promisify(fs.readFile)
let fs_writeFile = util.promisify(fs.writeFile)

let get_css_min = pathname => /*await*/ build_cache.get(
  pathname,
  async result => {
    let render = await disk_build(
      pathname,
      async temp_pathname => {
        let render = await clean_css.minify(
          await fs_readFile(pathname, {encoding: 'utf-8'})
        )
        return /*await*/ fs_writeFile(
          temp_pathname,
          render.styles,
          {encoding: 'utf-8'}
        )
      },
      true // diagnostics on
    )
    result.value = /*await*/ fs_readFile(render.pathname)
  }
}

In the above there was no dependency tracking needed or wanted, since CSS files cannot include further CSS source files. Less files can, handled as follows:

let BuildCache = require('@ndcode/build_cache')
let disk_build = require('@ndcode/disk_build')
let fs = require('fs')
let util = require('util')
let less = require('less/lib/less-node')
let path = require('path')

let build_cache = new BuildCache(true) // diagnostics on
let fs_readFile = util.promisify(fs.readFile)
let fs_writeFile = util.promisify(fs.writeFile)

let get_css_less = pathname => /*await*/ build_cache.get(
  pathname,
  async result => {
    let render = await disk_build(
      pathname,
      async temp_pathname => {
        let render = await less.render(
          await fs_readFile(pathname, {encoding: 'utf-8'}),
          {
            compress: true,
            filename: pathname,
            pathnames: [path.posix.dirname(pathname)]
          }
        )
        await fs_writeFile(
          temp_pathname,
          render.css,
          {encoding: 'utf-8'}
        )
        return render.imports
      },
      true // diagnostics on
    )
    result.value = /*await*/ fs_readFile(render.pathname)
  }
}

We are relying on fs_readFile() or less.render() to throw exceptions if the original stylesheet or any included stylesheet is not found or contains errors.

Also, note how much simplified the handling of asynchronicity is when using the ES7 async/await syntax. We recommend to do this for all new code, and to do it consistently, even if the use of Promise directly might give shorter code.

Code which is not already promisified, can be promisified as shown, or else we can add specific conversions in places by code like: await new Promise(...). Note the comment /* await */ where a Promise is passed through from a lower level routine, an explicit await would be consistent here but less efficient.

About diagnostics

If diag is passed as true, then a diagnostic will be printed indicating whether an output file is being rebuilt, or an existing output is being reused.

GIT repository

The development version can be cloned, downloaded, or browsed with gitweb at: https://git.ndcode.org/public/disk_build.git

License

All of our NPM packages are MIT licensed, please see LICENSE in the repository.

Contributions

We would greatly welcome your feedback and contributions. The disk_build is under active development (and is part of a larger project that is also under development) and thus the API is considered tentative and subject to change. If this is undesirable, you could possibly pin the version in your package.json.

Contact: Nick Downing nick@ndcode.org