@ndcode/jst_cache v0.1.0
Build Cache wrapper for JavaScript Template system
An NDCODE project.
Overview
The jst_cache package exports a single constructor
JSTCache(root, args, diag)
which must be called with the new operator. The resulting cache object stores
compiled JavaScript templates. Each template is, in general, a function that
can be called with certain arguments, and which will generate the HTML text for
a particular page or part of a page. The arguments can vary with each template.
The JSTCache object takes care of loading template source code, compiling it
using the jst package, and then executing the resulting JavaScript code. This
initial execution is intended to allow the template to load any resources it
needs, and generally set things up so that it is ready to generate HTML. The
initial execution returns the object which is stored in the cache. As mentioned
this is normally a function, but could be something else, e.g. any JSON object.
See the build_cache, disk_build and jst packages for more information.
The JSTCache object is essentially a wrapper object which routes the request
between these packages, to ensure that the compiled template is retrieved from
either RAM or disk if available, or is compiled and stored to RAM and to disk.
As well as wrapping the build_cache, disk_build and jst functionality
into a convenient "point and shoot" interface for template loading, jst_cache
also provides a dependency resolution service for the compiled template. If the
compiled template wants to load further templates during initialization or run-
time, it calls back into jst_cache, which recursively loads the dependency.
Template examples
See the jst package for more information about what can be in a template.
Essentially, the templates are written in a dialect of JavaScript which allows
HTML constructs to be specified directly, returning a string containing HTML.
HTML templates occuring at expression level will be completely rendered to a string, and the resulting string returned as the value of the expression.
HTML templates or strings (either single or double quoted strings, or template strings) occurring at statement level, become part of an enclosing template.
For example, the JavaScript code let text = p {'Hello, world'} sets text
to the value <p>Hello, world</p>. This consists of an HTML template p { }
occurring at expression level, then the string 'Hello, world' at statement
level inside the p { } template (any valid JavaScript statement is allowed).
Here is a complete example showing template substitution and HTML expressions:
let lang = 'en'
let name = 'John'
console.log(
html(lang=lang) {
body {`Hello, ${name}`}
}
}Running the above program will generate the output:
<html lang="en"><body>Hello, John</body></html>Template output buffer
It is also possible to use statement-level HTML templates or strings to build
up an output buffer, which can be used for whatever purpose. The output buffer
must be called _out and must be a JavaScript array of string fragments. As
each output fragment (such as an opening or closing tag or an embedded string)
is generated, it will be sent to the output buffer by an _out.push(...) call.
If there is dynamically generated text in the template, then it will be esacped
by the sequence .replace("&", "&").replace("<", "<"), this is chosen
so that no external dependency such as the html-entities package is needed,
and so that unnecessary escaping of characters such as ' or > is not done.
Attributes are done similarly, except that " is replaced instead of <. We
assume a UTF-8 environment, so there is really little need for HTML entities.
For example, consider a realistic template which we use on our demo server:
html(lang=lang) {
head {
link(rel="stylesheet" type="text/css" href="css/styles.css") {}
}
body {
p {`Hello, ${name}`}
}
}This compiles to the following plain JavaScript code:
_out.push("<html lang=\"" + lang.toString().replace("&", "&").replace("\"", """) + "\">");
{
_out.push("<head>");
_out.push("<link rel=\"stylesheet\" type=\"text/css\" href=\"css/styles.css\">");
_out.push("</head>");
}
{
_out.push("<body>");
{
_out.push("<p>");
_out.push(`Hello, ${name}`.replace("&", "&").replace("<", "<"));
_out.push("</p>");
}
_out.push("</body>");
}
_out.push("</html>");If invoked as an expression, the same code will be generated, but wrapped in an
Immediately-Invoked Function Expression (IIFE) that creates the _out buffer,
executes the above, then returns the buffer concatenated into a single string.
Template dependencies
Templates may depend on other templates. Templates are imported imported using
the syntax _require(...), similarly to the syntax require(...) in CommonJS.
We use _require() for templates instead of the normal require() because:
(1) We do not want to interfere with other symbols in the namespace, you can
also use require() in your templates to import ordinary CommonJS modules;
(2) Absolute paths have a different meaning in _require() versus require(),
for _require() they will be taken relative to the webserver's document root;
(3) Using _require() says that the file is in *.jst format, and hence it
should be parsed and any HTML constructs converted into regular JavaScript; and
(4) The template importing is an asynchronous operation, so you normally use
await _require(...), whereas require() is synchronous. Because templates
which have been edited are recompiled automatically by the running webserver,
we don't want to stop the world, especially as templates are slow to compile.
A slight difference between CommonJS exports and JavaScript template exports,
is that where CommonJS modules set the dedicated symbol module.exports to
whatever is exported, JavaScript templates instead return what is exported.
Template paths and variables
Similarly to CommonJS, the variable _require is injected by jst_cache and
is available to all templates. Other variables injected by jst_cache are
_pathname which is the path to the currently executing template, and _root
which is the root directory for absolute pathname references in _require().
Thus, paths sent to _require() are resolved relative to either _root if
absolute (beginning with /) or the directory part of _pathname if relative
(not beginning with /). Each template gets its own _require() function,
which knows the _pathname and _root values of the calling template.
If further variables should be made available to the template, they can be
included as a dictionary in the args parameter to JSTCache(). For example,
let jst_cache = new JSTCache(
'/home/myname/www',
{myvar: 'some value'}
)
let page = jst_cache.get('/home/myname/www/blog/page.jst')parses and executes the template /home/myname/www/blog/page.jst, with the
variables _pathname set to '/home/myname/www/blog/page.jst', _root set to
'/home/myname/www', and myvar set to 'some value'. Suppose page.jst is:
let footer = _require('/site/footer.jst')or
let footer = _require('../site/footer.jst')then the template /home/myname/www/site/footer.jst is parsed and executed,
with _pathname set to '/home/myname/www/site/footer.jst', and the other
injected variables _root and myvar set from the original JSTCache() call.
Multi-part template examples
Templating with HTML text strings
Here is an example of a template which takes the body of a page (as text which
may contain HTML tags), and wraps it in the standard html and body tags to
create a complete page. The idea is to use this to generate every page, so that
additional commands, such as viewport commands or script tags, can be set here.
Save this as page.jst, since it's the template for all pages in the system:
return body_text => html {
head {
link(rel="stylesheet" type="text/css" href="css/styles.css") {}
}
body {
_out(body_text)
}
}Save this as index.html.jst, as it's the template to generate index.html:
let page = await _require('/page.jst')
return page(p {'Hello, world'})Note that here we have somewhat hacked into the internals of the templating
system by directly appending the body text onto the output buffer _out. It is
intentional that you can access the internals for efficiency reasons, and that
you can, for example, avoid HTML escaping of particular text where appropriate.
Templating with callbacks
Another way, which is more sophisticated, is to use a callback function to
generate the body. If doing this, we pass the output buffer _out into the
callback function, so that the entire HTML of the page can be generated in an
unbroken stream, without having to concatenate the partial page repeatedly.
The page wrapper template page.jst which takes a body callback and calls it:
return body_callback => html {
head {
link(rel="stylesheet" type="text/css" href="css/styles.css") {}
}
body {
body_callback(_out)
}
}Then the specific page template index.html.jst which provides the callback:
let page = await _require('/page.jst')
return html(
_out => {
p {'Hello, world'}
}
)Note that in the above example, the { } around the callback function body is
essential, unlike in regular JavaScript, so that p {...} occurs in statement
rather than expression context. As an expression it would return a string which
would be ignored, since the page template is expecting output to be in _out.
File management
The JSTCache object a "point and shoot" interface which makes it very easy to
manage on-disk templates. You just call JSTCache.get() stating a pathname to
a template, normally a *.jst file, and the template is parsed, compiled to
JavaScript, and evaluated, with the evaluation result cached for future reuse.
What the JSTCache.get() function returns depends on the template source code.
Template exports are similar but slightly different to CommonJS module exports,
as noted in examples above. Usually, JSTCache.get() returns a JavaScript
function, which you call to generate HTML each time the page is to be served.
The HTML-generating function is re-useable in this way for efficiency reasons, since after the initial compilation, the cached version can be reused each time the page is served, but called with different arguments, resulting in different substitutions made on the page. For example, consider the following template:
return (lang, name) => html(lang=lang) {
body {
p {`Hello, ${name}`}
}
}Suppose the above is saved as /hello/index.html.jst under your document root
/var/www/html. When you instantiate the JSTCache object giving the document
root, and then call the get(pathname) instance function, you receive a
function of two arguments lang and name, which returns the HTML string.
Note that _require() paths beginning with / are taken relative to the
document root passed into the JSTCache() constructor call, whereas paths
beginning with anything else are taken relative to the dirname extracted from
the current value of _pathname as seen by the template. Each template has its
own _pathname and thus its _require() calls are relative to its own source.
Memory vs disk caching of templates
Templates compiled using JSTCache.get() are cached in memory, as long as the
same node interpreter is running, so that they can be retrieved using either
JSTCache.get(), or equivalently _require() inside a template, and they will
not be re-executed. In the above example, if you call JSTCache.get() twice,
you get the same object twice (it is a 2-argument function returning String).
As well as this, the compiled templates are also cached on disk, which requires
the document root to be writeable. For instance if you compile index.html.jst
in a hello/ subdirectory of the document root, you get a hidden file with the
same name prefixed by . in the same directory, here hello/.index.html.jst.
The main reason for the disk caching is really to fool the node.js require()
system into using correct relative paths, if the template uses require() as
well as _require(). However, it is also handy that the templates only need to
be re-evaluated and not recompiled if the webserver is stopped and restarted.
Before using either a disk-cached or a memory-cached template, the modification times are checked and the template is recompiled if it is stale. Thus, you can edit your website while it is live, and each page will be recompiled as needed just before serving. Templates deleted on disk are not returned from the cache.
Note: If the document root is not writeable, a simple expedient is to make sure
that all .*.jst files are up to date, so no write is attempted. You could do
this by firing up a development instance of the server in a writeable directory
and then indexing the whole site, for example by a recursive wget invocation.
To be implemented
It is intended that we will shortly add a timer function (or possibly just a function that the user should call periodically) to flush built templates from the cache after a stale time, on the assumption that the template might not be accessible or wanted anymore. For example, if the templates are HTML pages, the link structure of the site may have changed to make some pages inaccessible.
GIT repository
The development version can be cloned, downloaded, or browsed with gitweb at:
https://git.ndcode.org/public/jst_cache.git
License
All of our NPM packages are MIT licensed, please see LICENSE in the repository.
Contributions
The jst system is under active development (and is part of a larger project
that is also under development) and thus the API is tentative. Please go ahead
and incorporate the system into your project, or try out our example webserver
built on the system, subject to the caution that the API could change. Please
send us your experience and feedback, and let us know of improvements you make.
Contact: Nick Downing nick@ndcode.org
5 years ago