@totemorg/totem NPM

TOTEM

TOTEM provides a barebones web service with customizable features:

attach agents (local, registered, and notebook) to process endpoint requests
configure https service
denial-of-service protection
secure connections
client logins and profiles
PKI encryption and authentication
fault protected run states
indexing, uploading, downloading and cacheing static files
crud interface to mysql and neo4j databases
task queuing and regulation
periodic watchdog and polling services
file streams
data fetching using various protocols
smartcard reader
site skinning

TOTEM provides the following:

/DATASET.TYPE ? QUERY
/AREA/FILE ? QUERY
/AREA/ ? QUERY
/json/STORE ? QUERY
/graph/GRAPH ? QUERY
/AGENT ? QUERY

where the optional TYPE:

db | csv | txt | html | json

returns the dataset in the specified form.

Install

npm install @totemorg/totem		# install base service
npm update

Start

npm run ?env					# Show env variables
npm run ?config					# show configuration
npm run ?						# List start options
npm run start PLUGIN PLUGIN...	# Start totem with optional plugins

npm run setprot					# Configure for protected mode
npm run setdebug				# Configure for debugging mode
npm run setoper					# Configure for operational mode
npm run setprod					# Configure for production mode
npm run	startdbs				# Start database servers

Manage

npm run verminor				# Roll minor version
npm run vermajor				# Roll major version
npm run redoc					# Regen documentation
npm run pubminor				# republish as minor version
npm run pubmajor				# republish as major version

Usage

import * as TOTEM from "@totemorg/totem";

TOTEM.config({
	key: value, 						// set key
	"key.key": value, 					// indexed set
	"key.key.": value					// indexed append
}, sql => {
	console.log( sql ? "look mom - Im running!" : "something evil is lurking" );
});

where configuration keys follow ENUMS deep copy conventions.

Program Reference

TOTEM

Provides a basic web service. This module documented IAW jsdoc.

Requires: module:enums, module:jsdb, module:securelink, module:skin, module:agent, module:fs, module:constants, module:cluster, module:child_process, module:stream, module:vm, module:crypto, module:mime, module:xml2js, module:toobusy-js, module:json2csv, module:js2xmlparser, module:cheerio, module:markdown
Author: ACMESDS
Example

// npm test T1
// Create simple service but dont start it.
Log({
	msg: "Im simply a Totem interface so Im not even running as a service", 
	default_fetcher_endpts: TOTEM.nodeRouters,
	default_protect_mode: TOTEM.guard,
	default_cores_used: TOTEM.cores
});

Example

// npm test T2
// Totem service running in fault protection mode, no database, no UI; but I am running
// with 2 workers and the default endpoint routes.

config({
	mysql: null,
	guard: true,
	cores: 2
}, sql => {

	Log( 
`I'm a Totem service running in fault protection mode, no database, no UI; but I am running
with 2 workers and the default endpoint routes` );

});

Example

// npm test T3
// A Totem service with no workers.

config({
}, sql => {
	Log( 
`I'm a Totem service with no workers. I do, however, have a mysql database from which I've derived 
my startup options (see the openv.apps table for the Nick="Totem1") with just the standard endpoints.
	`
	);
});

Example

// npm test T4
// Only 1 worker, unprotected, a mysql database, and two endpoints.

config({
	nodeRouters: {
		dothis: function dothis(req,res) {  //< named handlers are shown in trace in console
			res( "123" );

			Log("", {
				do_query: req.query
			});
		},

		dothat: function dothat(req,res) {

			if (req.query.x)
				res( [{x:req.query.x+1,y:req.query.x+2}] );
			else
				res( _Error("We have a problem huston") );

			Log("", {
				msg: `Like dothis, but needs an ?x=value query`, 
				or_query: req.query,
				or_user: req.client
			});
		}
	}
}, sql => {
	Log("", {
		msg:
`As always, if the openv.apps Encrypt is set for the Nick="Totem" app, this service is now **encrypted** [*]
and has https (vs http) endpoints, here /dothis and /dothat endpoints.  Ive only requested only 1 worker (
aka core), Im running unprotected, and have a mysql database.  
[*] If my NICK.pfx does not already exists, Totem will create its password protected NICK.pfx cert from the
associated public NICK.crt and private NICK.key certs it creates.`,
		my_endpoints: T.nodeRouters
	});
});

Example

// npm test T5
// no cores but a mysql database and an anti-bot shield

config({
	"login.challenge.extend": 20
}, sql => {
	Log("", {
		msg:
`I am Totem client, with no cores but I do have mysql database and I have an anti-bot shield!!  Anti-bot
shields require a Encrypted service, and a UI (like that provided by DEBE) to be of any use.`, 
		mysql_derived_parms: T.siteContext
	});
});

Example

// npm test T6
// Testing tasker with database, 3 cores and an additional /test endpoint.

config({
	guard: false,	// ex override default 
	cores: 3,		// ex override default

	"nodeRouters.": {  // define endpoints
		test: function (req,res) {
			res(" here we go");  // endpoint must always repond to its client 
			if (isMaster)  // setup tasking examples on on master
				switch (req.query.opt || 1) {  // test example runTask
					case 1: 
						T.runTask({  // setup tasking for loops over these keys
							keys: "i,j",
							i: [1,2,3],
							j: [4,5]
						}, 
							// define the task which returns a message msg
							($) => "hello i,j=" + [i,j] + " from worker " + _Worker + " on " + $.node, 

							// define the message msg handler
							(msg) => console.log(msg)
						);
						break;

					case 2:
						T.runTask({
							qos: 1,
							keys: "i,j",
							i: [1,2,3],
							j: [4,5]
						}, 
							($) => "hello i,j=" + [i,j] + " from worker " + _Worker + " on " + $.node, 
							(msg) => console.log(msg)
						);
						break;

					case 3:
						break;
				}

		}
	}

}, sql => {
	Log( "Testing runTask with database and 3 cores at /test endpoint" );
});

Example

// npm test T7
// Conduct db maintenance

config({
}, sql => {				
	Log( "db maintenance" );

	if (isMaster)
		switch (process.argv[3]) {
			case 1: 
				sql.query( "select voxels.id as voxelID, chips.id as chipID from openv.voxels left join openv.chips on voxels.Ring = chips.Ring", function (err,recs) {
					recs.forEach( rec => {
						sql.query("update openv.voxels set chipID=? where ID=?", [rec.chipID, rec.voxelID], err => {
							Log(err);
						});
					});
				});
				break;

			case 2:
				sql.query("select ID, Ring from openv.voxels", function (err, recs) {
					recs.forEach( rec => {
						sql.query(
							"update openv.voxels set Point=geomFromText(?) where ?", 
							[ `POINT(${rec.Ring[0][0].x} ${rec.Ring[0][0].y})` , {ID: rec.ID} ], 
							err => {
								Log(err);
						});
					});
				});
				break;

			case 3:
				sql.query( "select voxels.id as voxelID, cache.id as chipID from openv.voxels left join openv.cache on voxels.Ring = fileCache.geo1", function (err,recs) {
					Log(err);
					recs.forEach( rec => {
						sql.query("update openv.voxels set chipID=? where ID=?", [rec.chipID, rec.voxelID], err => {
							Log(err);
						});
					});
				});
				break;

			case 4:
				sql.query("select ID, geo1 from openv.cache where bank='chip'", function (err, recs) {
					recs.forEach( rec => {
						if (rec.geo1)
							sql.query(
								"update openv.cache set x1=?, x2=? where ?", 
								[ rec.geo1[0][0].x, rec.geo1[0][0].y, {ID: rec.ID} ], 
								err => {
									Log(err);
							});
					});
				});
				break;

			case 5: 
				var parms = {
ring: "[degs] closed ring [lon, lon], ... ]  specifying an area of interest on the earth's surface",
"chip length": "[m] length of chip across an edge",
"chip samples": "[pixels] number of pixels across edge of chip"
				};
				//get all tables and revise field comments with info data here -  archive parms - /parms in flex will
				//use getfileds to get comments and return into

			case 6:
				var 
					RAN from "../randpr"),
					ran = new RAN({
						models: ["sinc"],
						Mmax: 150,  // max coherence intervals
						Mstep: 5 	// step intervals
					});

				ran.config( function (pc) {
					var 
						vals = pc.values,
						vecs = pc.vectors,
						N = vals.length, 
						ref = vals[N-1];

					vals.forEach( (val, idx) => {
						var
							save = {
								correlation_model: pc.model,
								coherence_intervals: pc.intervals,
								eigen_value: val,
								eigen_index: idx,
								ref_value: ref,
								max_intervals: ran.Mmax,
								eigen_vector: JSON.stringify( vecs[idx] )
							};

						sql.query("INSERT INTO openv.pcs SET ? ON DUPLICATE KEY UPDATE ?", [save,save] );	
					});
				});
				break;	
		}
});

Example

// npm test T8
// Conduct neo4j database maintenance

const $ from "../man/man.js");
config();
neoThread( neo => {
	neo.cypher( "MATCH (n:gtd) RETURN n", {}, (err,nodes) => {
		Log("nodes",err,nodes.length,nodes[0]);
		var map = {};
		nodes.forEach( (node,idx) => map[node.n.name] = idx );
		//Log(">map",map);

		neo.cypher( "MATCH (a:gtd)-[r]->(b:gtd) RETURN r", {}, (err,edges) => {
			Log("edges",err,edges.length,edges[0]);
			var 
				N = nodes.length,	
				cap = $([N,N], (u,v,C) => C[u][v] = 0 ),
				lambda = $([N,N], (u,v,L) => L[u][v] = 0),
				lamlist = $(N, (n,L) => L[n] = [] );

			edges.forEach( edge => cap[map[edge.r.srcId]][map[edge.r.tarId]] = 1 );

			//Log(">cap",cap);

			for (var s=0; s<N; s++)
				for (var t=s+1; t<N; t++) {
					var 
						{cutset} = $.MaxFlowMinCut(cap,s,t),
						cut = lambda[s][t] = lambda[t][s] = cutset.length;

					lamlist[cut].push([s,t]);
				}

			lamlist.forEach( (list,r) => {
				if ( r && list.length ) Log(r,list);
			});

		});
	});
});

TOTEM
- static
- inner

TOTEM.openvACL

Route nodes according to security requirements.

Kind: static constant of TOTEM

TOTEM.filters

Endpoint filters cb(data data as string || error)

Kind: static constant of TOTEM
Cfg: Object

TOTEM.watchDogs

Kind: static constant of TOTEM

TOTEM.siteContext

Site context extended by the mysql derived query when service starts

Kind: static constant of TOTEM
Cfg: Object

TOTEM.uploadFile(client, srcStream, sinkPath, tags, cb)

Uploads a source stream srcStream to a target file sinkPath owned by the specified client; optional tags are tagged to the upload and the callback cb is made if the upload was successful.

Kind: static method of TOTEM

Param	Type	Description
client	String	file owner
srcStream	Stream	source stream
sinkPath	String	path to target file
tags	Object	hash of tags to add to file
cb	function	callback(file) if upload successful

TOTEM~agents : object

Endpoint agents each of the form agent(req,res)

Kind: inner namespace of TOTEM

~agents : object

agents.attach(req, res)

Register named agents at the specified port serviced by specified number of server cores, charging clients a specified usage cost, or collect util info from registered agents. See attachAgents for details.

Kind: static method of agents

Param	Type	Description
req	requestHash	session request
res	responseCallback	session response

agents.build(req, res)

Endpoint to build script for specified &install keys.

Kind: static method of agents

Param	Type	Description
req	requestHash	session request
res	responseCallback	session response

agents.favicon(req, res)

Endpoint to return favorite icon

Kind: static method of agents

Param	Type	Description
req	requestHash	session request
res	responseCallback	session response

agents.help(req, res)

Endpoint to return system help on requested agent and section goto.

Kind: static method of agents

Param	Type	Description
req	requestHash	session request
res	responseCallback	session response

agents.ping(req, res)

Endpoint to test connectivity.

Kind: static method of agents

Param	Type	Description
req	Object	Totem request
res	function	Totem response

agents.GET(req, res)

Endpoint to select records from a dataset or redirect to an agent.

Kind: static method of agents

Param	Type	Description
req	requestHash	session request
res	responseCallback	session response

agents.PUT(req, res)

Endpoint to update records in a dataset within the client's focus.

Kind: static method of agents

Param	Type	Description
req	requestHash	session request
res	responseCallback	session response

agents.DELETE(req, res)

Endpoint to delte records from a dataset within the client's focus.

Kind: static method of agents

Param	Type	Description
req	requestHash	session request
res	responseCallback	session response

agents.POST(req, res)

Endpoint to add records to a dataset within the client's focus.

Kind: static method of agents

Param	Type	Description
req	requestHash	session request
res	responseCallback	session response

agents.EXEC(req, res)

Endpoint to execute records in a dataset within the client's focus.

Kind: static method of agents

Param	Type	Description
req	requestHash	session request
res	responseCallback	session response

TOTEM~_Errors

Client error messages

Kind: inner constant of TOTEM

~_Errors

_Errors.noFile

Kind: static property of _Errors

_Errors.badFilter

Kind: static property of _Errors

_Errors.noData

Kind: static property of _Errors

_Errors.noAgent

Kind: static property of _Errors

_Errors.noDB

Kind: static property of _Errors

_Errors.badReturn

Kind: static property of _Errors

_Errors.badLogin

Kind: static property of _Errors

_Errors.noSocket

Kind: static property of _Errors

_Errors.badQuery

Kind: static property of _Errors

_Errors.noEndpoint

Kind: static property of _Errors

_Errors.noID

Kind: static property of _Errors

_Errors.noPost

Kind: static property of _Errors

_Errors.noSQL

Kind: static property of _Errors

_Errors.noNEO

Kind: static property of _Errors

_Errors.noReader

Kind: static property of _Errors

TOTEM~agentThread(agent, req, res)

Start an agent thread by Route NODE = /DATASET.TYPE requests using the configured agents.

The routeThread adds client data:

encrypted: bool		// true if request on encrypted server
site: {...}			// site context
mimi: "type"		// mime type of response

Kind: inner method of TOTEM

Param	Type	Description
agent	function	agent to run
req	Object	session request
res	responseCallback	session response

TOTEM~dataThread(req, cb)

Start a dataset thread by appending a

sql: {...}			// sql connector
ds:	"focus.table"	// fully qualified sql table
focus: user focus
//action: "select|update| ..."	// corresponding crude name

to the req request with callback cb(req).

Kind: inner method of TOTEM

Param	Type	Description
req	Object	Totem endpoint request
cb	function	callback(revised req)

TOTEM~startDogs()

Start watchdogs

Kind: inner method of TOTEM

TOTEM~initServer()

Initialize the service.

Kind: inner method of TOTEM

TOTEM~stopServer()

Stop the server.

Kind: inner method of TOTEM

String

String
- ~linkify(ref) ⇐ String
- ~mailify(ref) ⇐ String
- ~parseXML(cb) ⇐ String

String~linkify(ref) ⇐ String

Expands "*"-tagged string into html link.

Kind: inner method of String
Extends: String

Param	Type
ref	String

Example

"test [abc](http:/junk)".linkify()
==> 'test <a href="http:/junk">abc</a>'

Example

"here is a link".linkify("test")
==> '<a href="test">here is a link</a>'

String~mailify(ref) ⇐ String

Expands string into a html email link.

Kind: inner method of String
Extends: String

Param	Type
ref	String

Example

"contact me".mailify("user.one")
'<a href="mailto:contact me">user.one</a>'

String~parseXML(cb) ⇐ String

Parse XML string into json then callsback cb(json).

Kind: inner method of String
Extends: String

Param	Type	Description
cb	function	callback( json		null if error )

Array

Array

Array~groupify(dot)

Groups "x.y.z", "u,v", "x" , ... list entries into "x(y(z)), u(v), x, ..." string.

Kind: inner method of Array

Param	Type	Description
dot	string	item seperator

Example

["a.b","x.y.z"].groupify(".")
==> 'a(b),x(y(z))'

Array~blog(req, keys, cb)

Blogs each string in the list.

Kind: inner method of Array
See: totem:blogify

Param	Type	Description
req	Object	Request context
keys	List	list of keys to blog
cb	function	callback(recs) blogified version of records

Array~joinify(cb)

Joins a list with an optional callback cb(head,list) to join the current list with the current head.

Kind: inner method of Array

Param	Type
cb	function

Example

[	a: null,
	g1: [ b: null, c: null, g2: [ x: null ] ],
	g3: [ y: null ] ].joinify()

==>
	"a,g1(b,c,g2(x)),g3(y)"

TOTEM

TOTEM
- static
- inner

TOTEM.openvACL

Route nodes according to security requirements.

Kind: static constant of TOTEM

TOTEM.filters

Endpoint filters cb(data data as string || error)

Kind: static constant of TOTEM
Cfg: Object

TOTEM.watchDogs

Kind: static constant of TOTEM

TOTEM.siteContext

Site context extended by the mysql derived query when service starts

Kind: static constant of TOTEM
Cfg: Object

TOTEM.uploadFile(client, srcStream, sinkPath, tags, cb)

Uploads a source stream srcStream to a target file sinkPath owned by the specified client; optional tags are tagged to the upload and the callback cb is made if the upload was successful.

Kind: static method of TOTEM

Param	Type	Description
client	String	file owner
srcStream	Stream	source stream
sinkPath	String	path to target file
tags	Object	hash of tags to add to file
cb	function	callback(file) if upload successful

TOTEM~agents : object

Endpoint agents each of the form agent(req,res)

Kind: inner namespace of TOTEM

~agents : object

agents.attach(req, res)

Kind: static method of agents

Param	Type	Description
req	requestHash	session request
res	responseCallback	session response

agents.build(req, res)

Endpoint to build script for specified &install keys.

Kind: static method of agents

Param	Type	Description
req	requestHash	session request
res	responseCallback	session response

agents.favicon(req, res)

Endpoint to return favorite icon

Kind: static method of agents

Param	Type	Description
req	requestHash	session request
res	responseCallback	session response

agents.help(req, res)

Endpoint to return system help on requested agent and section goto.

Kind: static method of agents

Param	Type	Description
req	requestHash	session request
res	responseCallback	session response

agents.ping(req, res)

Endpoint to test connectivity.

Kind: static method of agents

Param	Type	Description
req	Object	Totem request
res	function	Totem response

agents.GET(req, res)

Endpoint to select records from a dataset or redirect to an agent.

Kind: static method of agents

Param	Type	Description
req	requestHash	session request
res	responseCallback	session response

agents.PUT(req, res)

Endpoint to update records in a dataset within the client's focus.

Kind: static method of agents

Param	Type	Description
req	requestHash	session request
res	responseCallback	session response

agents.DELETE(req, res)

Endpoint to delte records from a dataset within the client's focus.

Kind: static method of agents

Param	Type	Description
req	requestHash	session request
res	responseCallback	session response

agents.POST(req, res)

Endpoint to add records to a dataset within the client's focus.

Kind: static method of agents

Param	Type	Description
req	requestHash	session request
res	responseCallback	session response

agents.EXEC(req, res)

Endpoint to execute records in a dataset within the client's focus.

Kind: static method of agents

Param	Type	Description
req	requestHash	session request
res	responseCallback	session response

TOTEM~_Errors

Client error messages

Kind: inner constant of TOTEM

~_Errors

_Errors.noFile

Kind: static property of _Errors

_Errors.badFilter

Kind: static property of _Errors

_Errors.noData

Kind: static property of _Errors

_Errors.noAgent

Kind: static property of _Errors

_Errors.noDB

Kind: static property of _Errors

_Errors.badReturn

Kind: static property of _Errors

_Errors.badLogin

Kind: static property of _Errors

_Errors.noSocket

Kind: static property of _Errors

_Errors.badQuery

Kind: static property of _Errors

_Errors.noEndpoint

Kind: static property of _Errors

_Errors.noID

Kind: static property of _Errors

_Errors.noPost

Kind: static property of _Errors

_Errors.noSQL

Kind: static property of _Errors

_Errors.noNEO

Kind: static property of _Errors

_Errors.noReader

Kind: static property of _Errors

TOTEM~agentThread(agent, req, res)

Start an agent thread by Route NODE = /DATASET.TYPE requests using the configured agents.

The routeThread adds client data:

encrypted: bool		// true if request on encrypted server
site: {...}			// site context
mimi: "type"		// mime type of response

Kind: inner method of TOTEM

Param	Type	Description
agent	function	agent to run
req	Object	session request
res	responseCallback	session response

TOTEM~dataThread(req, cb)

Start a dataset thread by appending a

sql: {...}			// sql connector
ds:	"focus.table"	// fully qualified sql table
focus: user focus
//action: "select|update| ..."	// corresponding crude name

to the req request with callback cb(req).

Kind: inner method of TOTEM