@totemorg/core v1.2.0

ENUMS

Provides methods to fetch, enumerate, stream, and index data. Also provides connections to the mysql, neo4j, txmail, rxmail services per env variables.

Install

npm install @totemorg/enums	
npm update

Start

npm run	startdbs				# Start database servers
npm run start 					# Start/unit test

Manage

npm run verminor				# Roll minor version
npm run vermajor				# Roll major version
npm run redoc					# Regen documentation
npm run pubminor				# republish as minor version
npm run pubmajor				# republish as major version

Usage

import { Copy, Each, Extend, Stream, ... } from "@totemorg/enums";

See program reference for more information.

Program Reference

CORE

Provides enumeration and indexing methods, stream and serialization methods, MySQL and Neo4J connectors, attach agent endpoints, regulate job workflows, fetch data, file, graph, and json stores.

This module documented IAW jsdoc.

Requires: module:os, module:cluster, module:fs, module:http, module:https, module:vm, module:cp, module:crypto, module:stream, module:mysql, module:neo4j-driver, module:nodemailer, module:nodemailer-smtp-transport
Author: ACMESDS

• CORE
  • static
    • .Attach(opts, agents, cb) ⇒
      • ~getGraph(req, res)
      • ~getStore(req, res)
      • ~getFile(req, res)
      • ~getAgent(spec, agents)
    • .Pipe()
    • .Fetch(ref, data, ack)
  • inner
    • ~TRACE(msg, ...args)
    • ~LexisNexis()

CORE.Attach(opts, agents, cb) ⇒

Attach agents to the agent cloud, where each agent(req,res) specifies its endpoint as follows:

NAME: agent                     // handle notebook/data requests
"/AREA": agent                  // handle file requests
".TYPE": agent                  // handle conversion requests 
"/": agent 	                    // default handler for file requests 
".": agent	                    // default handler for type conversions
NAME_MACHINE: agent             // handle requests via a MACHINE = py|cv|m|jx|db| ... 
_IGNORE: agent                  // ignored
"#EVENT": agent                 // handle notebook EVENT = charge|run requests
GET|PUT|POST|DELETE: agent      // handle CRUD data requests 

The supplied agents hash is automatically extended with "/json", "/graph", and "/file" reader agents.

When a request arrives to a agent server , Attach first

• attempts to resolve the request to the master agent, then (if SQL-equipped)
• attempts to resolve the request to a remote agent, then
• attempts to resolve the request to a notebook agent, then
• defaults to a CRUD agent based on the request type.

The following options opts can be provided when attaching agents:

• cores number of servers (workers) attached to the agents.

• port number that servers listen on.

• CORS flag to support Cross-Origin-Scripting.

• never = file cacheing exemption hash { AREA: 1, PATH: 1, ... }.

• parse hash of (post,body) callbacks to parse posts in request body.
• mimes mime hash sets the mime header for requested TYPE.
• https options hash when creating a HTTPS server.
• directs = file remapping hash {AREA: "./PathPrefix", ...}.

• filters = data filtering hash {TYPE: (res,req,res) => res(...), ...}.

• access(req) method to validate client request req.

• proxy(cert,headers) method to adjust cert from http headers.
• init(server,agents) method to initialize the agent server after listener attached.

• busy() method to block DOS attacks. Machines* require additional modules:

• py,cv,m requires @totemorg/agent

• jx requires @totemorg/man
• db requires @totemorg/enums

to support installed python, opencv, matlab, mathjx, mysql db, and neo4j db.

Each agent(req,res) is provided a res response method accepting a:

string
object
array
error
function

and a req request containing:

cookie: "...."		// client cookie string
agent: "..."		// client browser info
ipAddress: "..."	// client ip address
referer: "http://site"		// url during a cross-site request
method: "GET|PUT|POST|DELETE" 		// http request method
now: date			// date stamp when request started
post: "..."			// raw body text
url	: "/query"		// requested url path
reqSocket: socket	// socket to retrieve client cert, post etc
resSocket: socket	// method to create socket to accept response
cert: {...} 		// full client cert

path: "/[area/...]name.type"	// full node path
area: "name"		// file area being requested
table: "name"		// name of dataset/table being requested
type: "type" 		// type descriptor 

query: {...} 		// raw keys from url
where: {...} 		// sql-ized query keys from url
body: {...}			// body keys from request 
flags: {...} 		// flag keys from url
index: {...}		// sql-ized index keys from url

To attach your nodejs-ready machine to the agent cloud:

curl "http://totem/attach" | node [- ?|option=value|list|file.js] 

where option overrides the default port,cores,agents,etc.

Kind: static method of CORE
Returns: MACHINES Context of machines required by the agents

• .Attach(opts, agents, cb) ⇒
  • ~getGraph(req, res)
  • ~getStore(req, res)
  • ~getFile(req, res)
  • ~getAgent(spec, agents)

Attach~getGraph(req, res)

Graph reader agent returns neo4j graph at the requested path = /AGENT/GRAPH?QUERY

Kind: inner method of Attach

Attach~getStore(req, res)

Json reader Agent returns json store at the requested path = /AGENT/TABLE/STORE$EXPRESSION.

Kind: inner method of Attach

Attach~getFile(req, res)

File reader agent returns file at the requested path = /AREA/FILE

Kind: inner method of Attach

Attach~getAgent(spec, agents)

Return agent from the agents hash with the given {type,area,method,table} spec.

Kind: inner method of Attach

CORE.Pipe()

Kind: static method of CORE

CORE.Fetch(ref, data, ack)

Fetch data from a specified ref url

PROTOCOL://HOST/FILE ? QUERY 

where PROTOCOL is

• http(s) = http (https) protocol
• curl(s) = curl (curls uses certs/fetch.pfx to authenticate)
• wget(s) = wget (wgets uses certs/fetch.pfx to authenticate)
• mask = http access via rotated proxies
• lexnex = Lexis-Nexis oauth access to documents
• mysql = mysql database
• neo4j = neo4j database
• csv = csv file
• stream = stream file
• json = json file
• jpg = image file
• list = list file
• null = null file
• file = file/folder
• SITEREF = shortcut name

File paths may also contain wild-* cards. The callback name (GET || PUT || POST || DELETE || JSON) determines the request method. Optional QUERY keys:

batch 	= NUMBER of records to fetch 
offset	= NUMBER of records to offset fetch
select	= "FIELD, ..." to fetch
keys	= [...] header keys in csv source
comma	= "delim" for csv source
newline = "delim" for csv source

Kind: static method of CORE

Example

Fetch( ref, text => { // get request
})

Example

Fetch( ref, function json(ctx) {    // get request with json ctx
})

Example

Fetch( ref, [ ... ], function post(stat) { 	// post request with data hash list
})

Example

Fetch( ref, { ... }, function put(stat) { 	// put request with data hash
})

Example

Fetch( ref, null, function deete(stat) {	// delete request 
})

CORE~TRACE(msg, ...args)

Trace log message and args.

Kind: inner method of CORE

CORE~LexisNexis()

Start N Lexis-Nexis scrapes residing at the endpoint endpt + random counter when R is non-zero.

Kind: inner method of CORE

array

• array
  • .mysql
  • .neo4j
  • .txmail
  • .rxmail
  • .Log
  • .neoThread(cb)
  • .sqlThread(cb)
  • .Start(host, ctx)
  • .isFunction()
  • .isString()
  • .isObject()
  • .isNumber()
  • .Copy(src, tar, deep) ⇒ Object
  • .Each(A, cb)
  • .Notify()

array.mysql

Stash for MySQL configuation options.

Kind: static constant of array

array.neo4j

Stash for Neo4J configuation options.

Kind: static constant of array

array.txmail

Stash for Send-Mail configuation options.

Kind: static constant of array

array.rxmail

Stash for Receive-Mail configuation options.

Kind: static constant of array

array.Log

Dump message to the console.

Kind: static constant of array

array.neoThread(cb)

Provide a Neo4J connector to the callback.

Kind: static method of array

array.sqlThread(cb)

Provide a MySQL connector to the callback.

Kind: static method of array

array.Start(host, ctx)

Start a unit test using an optional $(cmd,ctx) command processor or a CB() callback provided in the ctx context.
Inspect a KEY variable, file, or notebook or function with ?KEY.

Kind: static method of array

array.isFunction()

Kind: static method of array

array.isString()

Kind: static method of array

array.isObject()

Kind: static method of array

array.isNumber()

Kind: static method of array

array.Copy(src, tar, deep) ⇒ Object

Copy source hash src to target hash tar. If the copy is shallow (deep = false), a Copy({...}, {}) is equivalent to new Object({...}). In a deep copy, (e.g. deep = "."), src keys are treated as keys into the target thusly:

{	
	A: value,			// sets target[A] = value

	"A.B.C": value, 	// sets target[A][B][C] = value

	"A.B.C.": {			// appends X,Y to target[A][B][C]
		X:value, Y:value, ...
	},	
}

Kind: static method of array
Returns: Object - target hash

array.Each(A, cb)

Enumerate Object A over its keys with callback cb(key,val).

Kind: static method of array

array.Notify()

Kind: static method of array

Array

String

String.Task(spec, ctx, cb)

Task callback cb(t,ctx) at triggered events defined by the event context ctx:

.start = starting number in relative-step mode || start datetime in clock-time mode (now clock-time mode)
.end = end time (start + 100 steps)
.on = number triggers steps enabled (0 disable)
.off = number of trigger steps disabled (0 disable)
.step = number of ms between triggers || "NUMBER [s|m|h|d|y]"  (1000 ms)
.queue = "task queue" to share job status ("" disable)
.client = "client name" to charge this request ("" disable)
.priority = number of seconds to monitor task signoff status (0 disable)
.source = "record source url" to fetch and feed to callback ("" disable)
.select = "field, ..." to select from source ("" none)
.batch = number of source records feed to callback cb(t,recs)

Kind: static method of String

Contacting, Contributing, Following

Feel free to

• submit and status TOTEM issues
• contribute to TOTEM notebooks
• revise TOTEM requirements
• browse TOTEM holdings
• or follow TOTEM milestones

License

MIT

© 2012 ACMESDS