3.15.0 • Published 1 year ago

@totemorg/totem v3.15.0

Weekly downloads
-
License
ISC
Repository
-
Last release
1 year ago

TOTEM

TOTEM provides a barebones web service with customizable features:

  • attach agents (local, registered, and notebook) to process endpoint requests
  • configure https service
  • denial-of-service protection
  • secure connections
  • client logins and profiles
  • PKI encryption and authentication
  • fault protected run states
  • indexing, uploading, downloading and cacheing static files
  • crud interface to mysql and neo4j databases
  • task queuing and regulation
  • periodic watchdog and polling services
  • file streams
  • data fetching using various protocols
  • smartcard reader
  • site skinning

TOTEM provides the following:

/DATASET.TYPE ? QUERY
/AREA/FILE ? QUERY
/AREA/ ? QUERY
/json/STORE ? QUERY
/graph/GRAPH ? QUERY
/AGENT ? QUERY

where the optional TYPE:

db | csv | txt | html | json 

returns the dataset in the specified form.

Install

npm install @totemorg/totem		# install base service
npm update

Start

npm run ?env					# Show env variables
npm run ?config					# show configuration
npm run ?						# List start options
npm run start PLUGIN PLUGIN...	# Start totem with optional plugins

npm run setprot					# Configure for protected mode
npm run setdebug				# Configure for debugging mode
npm run setoper					# Configure for operational mode
npm run setprod					# Configure for production mode
npm run	startdbs				# Start database servers

Manage

npm run verminor				# Roll minor version
npm run vermajor				# Roll major version
npm run redoc					# Regen documentation
npm run pubminor				# republish as minor version
npm run pubmajor				# republish as major version

Usage

import * as TOTEM from "@totemorg/totem";

TOTEM.config({
	key: value, 						// set key
	"key.key": value, 					// indexed set
	"key.key.": value					// indexed append
}, sql => {
	console.log( sql ? "look mom - Im running!" : "something evil is lurking" );
});

where configuration keys follow ENUMS deep copy conventions.

Program Reference

TOTEM

Provides a basic web service. This module documented IAW jsdoc.

Requires: module:enums, module:jsdb, module:securelink, module:skin, module:agent, module:fs, module:constants, module:cluster, module:child_process, module:stream, module:vm, module:crypto, module:mime, module:xml2js, module:toobusy-js, module:json2csv, module:js2xmlparser, module:cheerio, module:markdown
Author: ACMESDS
Example

// npm test T1
// Create simple service but dont start it.
Log({
	msg: "Im simply a Totem interface so Im not even running as a service", 
	default_fetcher_endpts: TOTEM.nodeRouters,
	default_protect_mode: TOTEM.guard,
	default_cores_used: TOTEM.cores
});

Example

// npm test T2
// Totem service running in fault protection mode, no database, no UI; but I am running
// with 2 workers and the default endpoint routes.

config({
	mysql: null,
	guard: true,
	cores: 2
}, sql => {

	Log( 
`I'm a Totem service running in fault protection mode, no database, no UI; but I am running
with 2 workers and the default endpoint routes` );

});

Example

// npm test T3
// A Totem service with no workers.

config({
}, sql => {
	Log( 
`I'm a Totem service with no workers. I do, however, have a mysql database from which I've derived 
my startup options (see the openv.apps table for the Nick="Totem1") with just the standard endpoints.
	`
	);
});

Example

// npm test T4
// Only 1 worker, unprotected, a mysql database, and two endpoints.

config({
	nodeRouters: {
		dothis: function dothis(req,res) {  //< named handlers are shown in trace in console
			res( "123" );

			Log("", {
				do_query: req.query
			});
		},

		dothat: function dothat(req,res) {

			if (req.query.x)
				res( [{x:req.query.x+1,y:req.query.x+2}] );
			else
				res( _Error("We have a problem huston") );

			Log("", {
				msg: `Like dothis, but needs an ?x=value query`, 
				or_query: req.query,
				or_user: req.client
			});
		}
	}
}, sql => {
	Log("", {
		msg:
`As always, if the openv.apps Encrypt is set for the Nick="Totem" app, this service is now **encrypted** [*]
and has https (vs http) endpoints, here /dothis and /dothat endpoints.  Ive only requested only 1 worker (
aka core), Im running unprotected, and have a mysql database.  
[*] If my NICK.pfx does not already exists, Totem will create its password protected NICK.pfx cert from the
associated public NICK.crt and private NICK.key certs it creates.`,
		my_endpoints: T.nodeRouters
	});
});

Example

// npm test T5
// no cores but a mysql database and an anti-bot shield

config({
	"login.challenge.extend": 20
}, sql => {
	Log("", {
		msg:
`I am Totem client, with no cores but I do have mysql database and I have an anti-bot shield!!  Anti-bot
shields require a Encrypted service, and a UI (like that provided by DEBE) to be of any use.`, 
		mysql_derived_parms: T.siteContext
	});
});

Example

// npm test T6
// Testing tasker with database, 3 cores and an additional /test endpoint.

config({
	guard: false,	// ex override default 
	cores: 3,		// ex override default

	"nodeRouters.": {  // define endpoints
		test: function (req,res) {
			res(" here we go");  // endpoint must always repond to its client 
			if (isMaster)  // setup tasking examples on on master
				switch (req.query.opt || 1) {  // test example runTask
					case 1: 
						T.runTask({  // setup tasking for loops over these keys
							keys: "i,j",
							i: [1,2,3],
							j: [4,5]
						}, 
							// define the task which returns a message msg
							($) => "hello i,j=" + [i,j] + " from worker " + _Worker + " on " + $.node, 

							// define the message msg handler
							(msg) => console.log(msg)
						);
						break;

					case 2:
						T.runTask({
							qos: 1,
							keys: "i,j",
							i: [1,2,3],
							j: [4,5]
						}, 
							($) => "hello i,j=" + [i,j] + " from worker " + _Worker + " on " + $.node, 
							(msg) => console.log(msg)
						);
						break;

					case 3:
						break;
				}

		}
	}

}, sql => {
	Log( "Testing runTask with database and 3 cores at /test endpoint" );
});

Example

// npm test T7
// Conduct db maintenance

config({
}, sql => {				
	Log( "db maintenance" );

	if (isMaster)
		switch (process.argv[3]) {
			case 1: 
				sql.query( "select voxels.id as voxelID, chips.id as chipID from openv.voxels left join openv.chips on voxels.Ring = chips.Ring", function (err,recs) {
					recs.forEach( rec => {
						sql.query("update openv.voxels set chipID=? where ID=?", [rec.chipID, rec.voxelID], err => {
							Log(err);
						});
					});
				});
				break;

			case 2:
				sql.query("select ID, Ring from openv.voxels", function (err, recs) {
					recs.forEach( rec => {
						sql.query(
							"update openv.voxels set Point=geomFromText(?) where ?", 
							[ `POINT(${rec.Ring[0][0].x} ${rec.Ring[0][0].y})` , {ID: rec.ID} ], 
							err => {
								Log(err);
						});
					});
				});
				break;

			case 3:
				sql.query( "select voxels.id as voxelID, cache.id as chipID from openv.voxels left join openv.cache on voxels.Ring = fileCache.geo1", function (err,recs) {
					Log(err);
					recs.forEach( rec => {
						sql.query("update openv.voxels set chipID=? where ID=?", [rec.chipID, rec.voxelID], err => {
							Log(err);
						});
					});
				});
				break;

			case 4:
				sql.query("select ID, geo1 from openv.cache where bank='chip'", function (err, recs) {
					recs.forEach( rec => {
						if (rec.geo1)
							sql.query(
								"update openv.cache set x1=?, x2=? where ?", 
								[ rec.geo1[0][0].x, rec.geo1[0][0].y, {ID: rec.ID} ], 
								err => {
									Log(err);
							});
					});
				});
				break;

			case 5: 
				var parms = {
ring: "[degs] closed ring [lon, lon], ... ]  specifying an area of interest on the earth's surface",
"chip length": "[m] length of chip across an edge",
"chip samples": "[pixels] number of pixels across edge of chip"
				};
				//get all tables and revise field comments with info data here -  archive parms - /parms in flex will
				//use getfileds to get comments and return into

			case 6:
				var 
					RAN from "../randpr"),
					ran = new RAN({
						models: ["sinc"],
						Mmax: 150,  // max coherence intervals
						Mstep: 5 	// step intervals
					});

				ran.config( function (pc) {
					var 
						vals = pc.values,
						vecs = pc.vectors,
						N = vals.length, 
						ref = vals[N-1];

					vals.forEach( (val, idx) => {
						var
							save = {
								correlation_model: pc.model,
								coherence_intervals: pc.intervals,
								eigen_value: val,
								eigen_index: idx,
								ref_value: ref,
								max_intervals: ran.Mmax,
								eigen_vector: JSON.stringify( vecs[idx] )
							};

						sql.query("INSERT INTO openv.pcs SET ? ON DUPLICATE KEY UPDATE ?", [save,save] );	
					});
				});
				break;	
		}
});		

Example

// npm test T8
// Conduct neo4j database maintenance

const $ from "../man/man.js");
config();
neoThread( neo => {
	neo.cypher( "MATCH (n:gtd) RETURN n", {}, (err,nodes) => {
		Log("nodes",err,nodes.length,nodes[0]);
		var map = {};
		nodes.forEach( (node,idx) => map[node.n.name] = idx );
		//Log(">map",map);

		neo.cypher( "MATCH (a:gtd)-[r]->(b:gtd) RETURN r", {}, (err,edges) => {
			Log("edges",err,edges.length,edges[0]);
			var 
				N = nodes.length,	
				cap = $([N,N], (u,v,C) => C[u][v] = 0 ),
				lambda = $([N,N], (u,v,L) => L[u][v] = 0),
				lamlist = $(N, (n,L) => L[n] = [] );

			edges.forEach( edge => cap[map[edge.r.srcId]][map[edge.r.tarId]] = 1 );

			//Log(">cap",cap);

			for (var s=0; s<N; s++)
				for (var t=s+1; t<N; t++) {
					var 
						{cutset} = $.MaxFlowMinCut(cap,s,t),
						cut = lambda[s][t] = lambda[t][s] = cutset.length;

					lamlist[cut].push([s,t]);
				}

			lamlist.forEach( (list,r) => {
				if ( r && list.length ) Log(r,list);
			});

		});
	});
});	

TOTEM.openvACL

Route nodes according to security requirements.

Kind: static constant of TOTEM

TOTEM.filters

Endpoint filters cb(data data as string || error)

Kind: static constant of TOTEM
Cfg: Object

TOTEM.watchDogs

Kind: static constant of TOTEM

TOTEM.siteContext

Site context extended by the mysql derived query when service starts

Kind: static constant of TOTEM
Cfg: Object

TOTEM.uploadFile(client, srcStream, sinkPath, tags, cb)

Uploads a source stream srcStream to a target file sinkPath owned by the specified client; optional tags are tagged to the upload and the callback cb is made if the upload was successful.

Kind: static method of TOTEM

ParamTypeDescription
clientStringfile owner
srcStreamStreamsource stream
sinkPathStringpath to target file
tagsObjecthash of tags to add to file
cbfunctioncallback(file) if upload successful

TOTEM~agents : object

Endpoint agents each of the form agent(req,res)

Kind: inner namespace of TOTEM

agents.attach(req, res)

Register named agents at the specified port serviced by specified number of server cores, charging clients a specified usage cost, or collect util info from registered agents. See attachAgents for details.

Kind: static method of agents

ParamTypeDescription
reqrequestHashsession request
resresponseCallbacksession response

agents.build(req, res)

Endpoint to build script for specified &install keys.

Kind: static method of agents

ParamTypeDescription
reqrequestHashsession request
resresponseCallbacksession response

agents.favicon(req, res)

Endpoint to return favorite icon

Kind: static method of agents

ParamTypeDescription
reqrequestHashsession request
resresponseCallbacksession response

agents.help(req, res)

Endpoint to return system help on requested agent and section goto.

Kind: static method of agents

ParamTypeDescription
reqrequestHashsession request
resresponseCallbacksession response

agents.ping(req, res)

Endpoint to test connectivity.

Kind: static method of agents

ParamTypeDescription
reqObjectTotem request
resfunctionTotem response

agents.GET(req, res)

Endpoint to select records from a dataset or redirect to an agent.

Kind: static method of agents

ParamTypeDescription
reqrequestHashsession request
resresponseCallbacksession response

agents.PUT(req, res)

Endpoint to update records in a dataset within the client's focus.

Kind: static method of agents

ParamTypeDescription
reqrequestHashsession request
resresponseCallbacksession response

agents.DELETE(req, res)

Endpoint to delte records from a dataset within the client's focus.

Kind: static method of agents

ParamTypeDescription
reqrequestHashsession request
resresponseCallbacksession response

agents.POST(req, res)

Endpoint to add records to a dataset within the client's focus.

Kind: static method of agents

ParamTypeDescription
reqrequestHashsession request
resresponseCallbacksession response

agents.EXEC(req, res)

Endpoint to execute records in a dataset within the client's focus.

Kind: static method of agents

ParamTypeDescription
reqrequestHashsession request
resresponseCallbacksession response

TOTEM~_Errors

Client error messages

Kind: inner constant of TOTEM

_Errors.noFile

Kind: static property of _Errors

_Errors.badFilter

Kind: static property of _Errors

_Errors.noData

Kind: static property of _Errors

_Errors.noAgent

Kind: static property of _Errors

_Errors.noDB

Kind: static property of _Errors

_Errors.badReturn

Kind: static property of _Errors

_Errors.badLogin

Kind: static property of _Errors

_Errors.noSocket

Kind: static property of _Errors

_Errors.badQuery

Kind: static property of _Errors

_Errors.noEndpoint

Kind: static property of _Errors

_Errors.noID

Kind: static property of _Errors

_Errors.noPost

Kind: static property of _Errors

_Errors.noSQL

Kind: static property of _Errors

_Errors.noNEO

Kind: static property of _Errors

_Errors.noReader

Kind: static property of _Errors

TOTEM~agentThread(agent, req, res)

Start an agent thread by Route NODE = /DATASET.TYPE requests using the configured agents.

The routeThread adds client data:

encrypted: bool		// true if request on encrypted server
site: {...}			// site context
mimi: "type"		// mime type of response

Kind: inner method of TOTEM

ParamTypeDescription
agentfunctionagent to run
reqObjectsession request
resresponseCallbacksession response

TOTEM~dataThread(req, cb)

Start a dataset thread by appending a

sql: {...}			// sql connector
ds:	"focus.table"	// fully qualified sql table
focus: user focus
//action: "select|update| ..."	// corresponding crude name

to the req request with callback cb(req).

Kind: inner method of TOTEM

ParamTypeDescription
reqObjectTotem endpoint request
cbfunctioncallback(revised req)

TOTEM~startDogs()

Start watchdogs

Kind: inner method of TOTEM

TOTEM~initServer()

Initialize the service.

Kind: inner method of TOTEM

TOTEM~stopServer()

Stop the server.

Kind: inner method of TOTEM

String

String~linkify(ref) ⇐ String

Expands "*"-tagged string into html link.

Kind: inner method of String
Extends: String

ParamType
refString

Example

"test [abc](http:/junk)".linkify()
==> 'test <a href="http:/junk">abc</a>'

Example

"here is a link".linkify("test")
==> '<a href="test">here is a link</a>'

String~mailify(ref) ⇐ String

Expands string into a html email link.

Kind: inner method of String
Extends: String

ParamType
refString

Example

"contact me".mailify("user.one")
'<a href="mailto:contact me">user.one</a>'

String~parseXML(cb) ⇐ String

Parse XML string into json then callsback cb(json).

Kind: inner method of String
Extends: String

ParamTypeDescription
cbfunctioncallback( jsonnull if error )

Array

Array~groupify(dot)

Groups "x.y.z", "u,v", "x" , ... list entries into "x(y(z)), u(v), x, ..." string.

Kind: inner method of Array

ParamTypeDescription
dotstringitem seperator

Example

["a.b","x.y.z"].groupify(".")
==> 'a(b),x(y(z))'

Array~blog(req, keys, cb)

Blogs each string in the list.

Kind: inner method of Array
See: totem:blogify

ParamTypeDescription
reqObjectRequest context
keysListlist of keys to blog
cbfunctioncallback(recs) blogified version of records

Array~joinify(cb)

Joins a list with an optional callback cb(head,list) to join the current list with the current head.

Kind: inner method of Array

ParamType
cbfunction

Example

[	a: null,
	g1: [ b: null, c: null, g2: [ x: null ] ],
	g3: [ y: null ] ].joinify()

==>
	"a,g1(b,c,g2(x)),g3(y)"

TOTEM

TOTEM.openvACL

Route nodes according to security requirements.

Kind: static constant of TOTEM

TOTEM.filters

Endpoint filters cb(data data as string || error)

Kind: static constant of TOTEM
Cfg: Object

TOTEM.watchDogs

Kind: static constant of TOTEM

TOTEM.siteContext

Site context extended by the mysql derived query when service starts

Kind: static constant of TOTEM
Cfg: Object

TOTEM.uploadFile(client, srcStream, sinkPath, tags, cb)

Uploads a source stream srcStream to a target file sinkPath owned by the specified client; optional tags are tagged to the upload and the callback cb is made if the upload was successful.

Kind: static method of TOTEM

ParamTypeDescription
clientStringfile owner
srcStreamStreamsource stream
sinkPathStringpath to target file
tagsObjecthash of tags to add to file
cbfunctioncallback(file) if upload successful

TOTEM~agents : object

Endpoint agents each of the form agent(req,res)

Kind: inner namespace of TOTEM

agents.attach(req, res)

Register named agents at the specified port serviced by specified number of server cores, charging clients a specified usage cost, or collect util info from registered agents. See attachAgents for details.

Kind: static method of agents

ParamTypeDescription
reqrequestHashsession request
resresponseCallbacksession response

agents.build(req, res)

Endpoint to build script for specified &install keys.

Kind: static method of agents

ParamTypeDescription
reqrequestHashsession request
resresponseCallbacksession response

agents.favicon(req, res)

Endpoint to return favorite icon

Kind: static method of agents

ParamTypeDescription
reqrequestHashsession request
resresponseCallbacksession response

agents.help(req, res)

Endpoint to return system help on requested agent and section goto.

Kind: static method of agents

ParamTypeDescription
reqrequestHashsession request
resresponseCallbacksession response

agents.ping(req, res)

Endpoint to test connectivity.

Kind: static method of agents

ParamTypeDescription
reqObjectTotem request
resfunctionTotem response

agents.GET(req, res)

Endpoint to select records from a dataset or redirect to an agent.

Kind: static method of agents

ParamTypeDescription
reqrequestHashsession request
resresponseCallbacksession response

agents.PUT(req, res)

Endpoint to update records in a dataset within the client's focus.

Kind: static method of agents

ParamTypeDescription
reqrequestHashsession request
resresponseCallbacksession response

agents.DELETE(req, res)

Endpoint to delte records from a dataset within the client's focus.

Kind: static method of agents

ParamTypeDescription
reqrequestHashsession request
resresponseCallbacksession response

agents.POST(req, res)

Endpoint to add records to a dataset within the client's focus.

Kind: static method of agents

ParamTypeDescription
reqrequestHashsession request
resresponseCallbacksession response

agents.EXEC(req, res)

Endpoint to execute records in a dataset within the client's focus.

Kind: static method of agents

ParamTypeDescription
reqrequestHashsession request
resresponseCallbacksession response

TOTEM~_Errors

Client error messages

Kind: inner constant of TOTEM

_Errors.noFile

Kind: static property of _Errors

_Errors.badFilter

Kind: static property of _Errors

_Errors.noData

Kind: static property of _Errors

_Errors.noAgent

Kind: static property of _Errors

_Errors.noDB

Kind: static property of _Errors

_Errors.badReturn

Kind: static property of _Errors

_Errors.badLogin

Kind: static property of _Errors

_Errors.noSocket

Kind: static property of _Errors

_Errors.badQuery

Kind: static property of _Errors

_Errors.noEndpoint

Kind: static property of _Errors

_Errors.noID

Kind: static property of _Errors

_Errors.noPost

Kind: static property of _Errors

_Errors.noSQL

Kind: static property of _Errors

_Errors.noNEO

Kind: static property of _Errors

_Errors.noReader

Kind: static property of _Errors

TOTEM~agentThread(agent, req, res)

Start an agent thread by Route NODE = /DATASET.TYPE requests using the configured agents.

The routeThread adds client data:

encrypted: bool		// true if request on encrypted server
site: {...}			// site context
mimi: "type"		// mime type of response

Kind: inner method of TOTEM

ParamTypeDescription
agentfunctionagent to run
reqObjectsession request
resresponseCallbacksession response

TOTEM~dataThread(req, cb)

Start a dataset thread by appending a

sql: {...}			// sql connector
ds:	"focus.table"	// fully qualified sql table
focus: user focus
//action: "select|update| ..."	// corresponding crude name

to the req request with callback cb(req).

Kind: inner method of TOTEM

ParamTypeDescription
reqObjectTotem endpoint request
cbfunctioncallback(revised req)

TOTEM~startDogs()

Start watchdogs

Kind: inner method of TOTEM

TOTEM~initServer()

Initialize the service.

Kind: inner method of TOTEM

TOTEM~stopServer()

Stop the server.

Kind: inner method of TOTEM

Contacting, Contributing, Following

Feel free to

License

MIT


© 2012 ACMESDS

3.15.0

1 year ago

3.14.0

1 year ago

3.11.0

1 year ago

3.13.0

1 year ago

3.12.0

1 year ago

3.9.0

2 years ago

3.10.0

2 years ago

3.4.0

2 years ago

3.3.0

2 years ago

3.2.0

2 years ago

3.7.0

2 years ago

3.6.0

2 years ago

3.5.0

2 years ago

3.1.0

2 years ago

2.25.0

3 years ago

2.24.0

3 years ago

2.27.0

3 years ago

2.26.0

3 years ago

2.32.0

3 years ago

2.31.0

3 years ago

2.34.0

3 years ago

2.33.0

3 years ago

2.36.0

3 years ago

2.35.0

3 years ago

2.23.0

3 years ago

2.22.0

3 years ago

2.20.0

3 years ago

2.19.0

3 years ago

2.18.0

3 years ago

2.17.0

3 years ago

2.16.0

3 years ago

2.15.0

3 years ago

2.14.0

3 years ago

2.13.0

3 years ago

2.12.0

3 years ago

2.11.0

3 years ago

2.9.0

3 years ago

2.7.0

3 years ago

2.6.0

3 years ago

2.5.0

3 years ago

2.4.0

3 years ago

2.3.0

3 years ago

2.2.0

3 years ago