@ndcode/jst v0.1.5
JavaScript Template system
An NDCODE project.
Overview
The jst package exports a single function
jst(text, options)
which parses the given text in a JavaScript dialect which can contain HTML-like
constructs, then generates equivalent plain JavaScript code which when run,
would generate the requested HTML. As well as the HTML generation, there can be
normal JavaScript logic to influence the generated page or substitute into it.
The system is inspired by Pug (formerly Jade) templates, but emphasizing the
JavaScript, for instance you do not need a - sign at the start of each line,
and the parser automatically follows the JavaScript block scope. It is also
inspired by JSX templates, but the embedded HTML is less verbose, there are no
closing tags since HTML is bracketed by { } rather than by <tag>...</tag>.
Template syntax
HTML tags in templates
The plain JavaScript in the file is absolutely normal and is expected to use
CommonJS conventions, for instance, require(...) is supported and so forth.
The embedded HTML uses a syntax similar to JavaScript function definitions,
function(arg1, arg2, ...) {...}
except that instead of the word "function" any valid HTML (or other) tag name
is allowed, and instead of just argument names, HTML attributes of the form
attrname
or
attrname=value
are supported. In the attrname=value syntax, value is any valid JavaScript
expression. No commas are required between attributes, so the parser has to
automatically detect where one expression ends and another begins (similarly to
how automatic semicolon insertion works in regular JavaScript).
The ( ) around the attributes is not necessary when there are no attributes.
Thus a simple HTML file could be written as the following JavaScript template:
html(lang="en") {
head {}
body {}
}and this would translate to the HTML file:
<html lang="en"><head></head><body></body></html>For certain particular HTML tags such as img, no closing tag is generated in
the output HTML. However, the empty { } must still be given in the template.
This is partly for the parser's convenience, since it depends on recognizing
the { } to distinguish between an HTML template and ordinary JavaScript code.
It is also more uniform: there is no need to remember which tags are special.
Regular text in templates
Regular text is placed in the generated HTML by simply quoting it. That is, if a statement is seen consisting of only a JavaScript string or template string, the string is converted to HTML by escaping it (helping to guard against any HTML injection attacks), and then output as part of some enclosing template.
For example:
html(lang="en") {
body {'Hello, world'}
}This generates the text:
<html lang="en"><body>Hello, world</body></html>If the text is multi-line, backquoted (template) strings are used, to allow
embedded newlines. When the entire text is held in a variable, e.g. myvar, a
template string such as ${myvar} should be used, to convert it into a
standalone statement consisting of only a string constant, and hence into HTML.
Note that ordinary single- or double-quoted string constants should be used in preference to template strings where possible, since the constant strings will be HTML-escaped at template compile time rather than at template run-time.
Note that in ordinary HTML, certain tags are more sensitive to whitespace than others, according to complicated rules about inline block elements and so on. This is never an issue with JavaScript templates, we can use as much indenting and other whitespace as we want, and only quoted whitespace will be output.
Special tags in templates
The HTML script tag is treated specially, because it contains JavaScript,
which is understood directly by the template parser. The JavaScript inside the
script tag is minified (comments stripped, unnecessary braces and parentheses
removed, semicolons added and so on), and then converted to a string constant,
which will be copied out to the HTML file whenever the template is executed.
The HTML style tag is treated specially, because it contains CSS, which we
handle by switching temporarily to another parser that understands CSS. We use
a slightly modified version of the clean-css package to do this. The result
is then minified and converted to a string constant, similar to script tags.
Note that the script and style tags cannot contain substitutions, in the
way that ordinary JST code can. This is because (i) the entire contents of the
special tag are minified and stringified at template compile time, so the
content of the special tag is the same each time the page is generated, and
(ii) it runs elsewhere, which doesn't have access to the template variables.
HTML class and id shorthand
The tag name can be followed by #name as shorthand for the HTML attribute
id="name"
or by .name as shorthand for the HTML attribute
class="name"
and these can be repeated as needed, the id attribute collects all #name
separated by spaces and the class attribute collects all .name similarly.
These must come before any ordinary attributes (before an opening parenthesis).
Parser limitations
Certain tag or attribute names present difficulty since they contain - signs
or other characters invalid in JavaScript identifiers, or they may be reserved
words in JavaScript. The parser copes with this quite well (most syntax errors
can already be re-parsed as tag or attribute names), but in difficult cases it
could be necessary to quote the tag and/or attribute names. For example,
div.class-1.class-2 {}
doesn't compile because "1." is a floating point number, for now we write it
div.'class-1'.class-2 {}
although we expect that this restriction can be lifted in a future version.
Also, the style parser for CSS code can be confused by tag, id or class
names that end with -style. So we should be careful of code like this,
div.my-style {...}
since the div.my- prefix won't be seen until parsing is complete, hence the
parser will switch to CSS parsing inside the braces. Therefore, quote it like
div.'my-style' {...}
although again, we expect that this restriction can be lifted in the future.
Another slight limitation of the current parser is that it is more permissive
than normal in parsing regular JavaScript code, for instance commas are not
required between function arguments, because an HTML template is basically
parsed as a function call until the opening { is seen to identify it as a
template. This can also likely be improved in a future version of the system.
Expression vs statement templates
HTML templates occuring at expression level will be completely rendered to a string, and the resulting string returned as the value of the expression.
HTML templates occurring at statement level are treated somewhat like quoted strings, in that the generated HTML will become part of an enclosing template.
Here is a complete example showing template substitution and HTML expressions:
let lang = 'en'
let name = 'John'
console.log(
html(lang=lang) {
body {`Hello, ${name}`}
}
)Running the above program will generate the output:
<html lang="en"><body>Hello, John</body></html>Template output buffer
It is also possible to use statement-level HTML templates or strings to build
up an output buffer, which can be used for whatever purpose. The output buffer
must be called _out and must be a JavaScript array of string fragments. As
each output fragment (such as an opening or closing tag or an embedded string)
is generated, it will be sent to the output buffer by an _out.push(...) call.
If there is dynamically generated text in the template, then it will be esacped
by the sequence .replace("&", "&").replace("<", "<"), this is chosen
so that no external dependency such as the html-entities package is needed,
and so that unnecessary escaping of characters such as ' or > is not done.
Attributes are done similarly, except that " is replaced instead of <. We
assume a UTF-8 environment, so there is really little need for HTML entities.
For example, consider a realistic template which we use on our demo server:
html(lang=lang) {
head {
link(rel="stylesheet" type="text/css" href="css/styles.css") {}
}
body {
p {`Hello, ${name}`}
}
}This compiles to the following plain JavaScript code:
_out.push("<html lang=\"" + lang.toString().replace("&", "&").replace("\"", """) + "\">");
{
_out.push("<head>");
_out.push("<link rel=\"stylesheet\" type=\"text/css\" href=\"css/styles.css\">");
_out.push("</head>");
}
{
_out.push("<body>");
{
_out.push("<p>");
_out.push(`Hello, ${name}`.replace("&", "&").replace("<", "<"));
_out.push("</p>");
}
_out.push("</body>");
}
_out.push("</html>");If invoked as an expression, the same code will be generated, but wrapped in an
Immediately-Invoked Function Expression (IIFE) that creates the _out buffer,
executes the above, then returns the buffer concatenated into a single string.
Usage example
Suppose we want to try out the JST template processor. Returning to the example
from the section "Expression vs Statement templates", we could create a file
called mytemplate.jst containing the example JST code to print a simple page:
let lang = 'en'
let name = 'John'
console.log(
html(lang=lang) {
body {`Hello, ${name}`}
}
)
```js
We could then create an example JavaScript program `example.js` to read,
compile and then execute the template, as follows:
```js
let fs = require('fs')
let jst = require('@ndcode/jst')
eval(jst(fs.readFileSync('mytemplate.jst')))To run this, we would have to run the node interpreter at the command line:
node example.jsThis produces the output described earlier:
<html lang="en"><body>Hello, John</body></html>Command line preprocessor
Using the programmatic interface is overkill in many situations, e.g. if we
just want to see the generated JavaScript code, or if we want to compile all
pages of our website to JavaScript ahead of time by a Gruntfile or similar.
Hence, we provide the command line interface jst as a convenient way to
convert JST to JavaScript. Using the command-line, the above example becomes:
jst <mytemplate.jst >mytemplate.js
node mytemplate.jsInspecting the JavaScript file mytemplate.js shows how the HTML is generated:
let lang = 'en';
let name = 'John';
console.log((() => {
let _out = [];
_out.push("<html lang=\"" + lang.toString().replace("&", "&").replace("\"", """) + "\">");
{
_out.push("<body>");
_out.push(`Hello, ${name}`.replace("&", "&").replace("<", "<"));
_out.push("</body>");
}
_out.push("</html>");
return _out.join('');
})());Various command line options are available for setting indentation and similar.
There is also a --wrap option which adds some prologue and epilogue code,
which is part of a larger system we have designed, for allowing templates to
embed each other and so on. See the jst_server and jst_cache modules for
more information. If the extra complexity isn't needed, simply omit --wrap.
JSTizer program
In development, we often want to refer to example code from the Web, e.g. if we want to incorporate the Bootstrap front-end framework or if we want to create a web-form or some other common web-development task. The example code is usually written in HTML. We can drop this straight into a JST project by JSTizing it.
See the jstize module for more information. Essentially the workflow is (i)
get some HTML in a file *.html, it does not need to be a complete page, but
must be a syntactically valid and complete HTML fragment, (ii) use the JSTizer
to convert this into an equivalent *.jst file, (iii) use the JST system, by
either the programmatic API or the command-line tool, to convert this into an
equivalent *.js file, (iv) run the *.js file to recreate the original HTML.
Of course, the *.jst template can be modified to add any needed extra logic.
GIT repository
The development version can be cloned, downloaded, or browsed with gitweb at:
https://git.ndcode.org/public/jst.git
License
All of our NPM packages are MIT licensed, please see LICENSE in the repository.
We also gratefully acknowledge the acorn team for the original MIT license of
their JavaScript parser, which we've heavily modified to create the JST parser.
Contributions
The jst system is under active development (and is part of a larger project
that is also under development) and thus the API is tentative. Please go ahead
and incorporate the system into your project, or try out our example webserver
built on the system, subject to the caution that the API could change. Please
send us your experience and feedback, and let us know of improvements you make.
Contact: Nick Downing nick@ndcode.org