1.6.0 • Published 6 months ago

@totemorg/agent v1.6.0

Weekly downloads
-
License
ISC
Repository
-
Last release
6 months ago

AGENT

Add TOTEM agents to your compute cloud. Regulate workflows. Fetch data.

Install

npm install @totemorg/enums	

Manage

npm run start ?					# List start options
npm run verminor				# Roll minor version
npm run vermajor				# Roll major version
npm run redoc					# Regen documentation
npm run pubminor				# republish as minor version
npm run pubmajor				# republish as major version

Usage

To add your compute agents to TOTEM's compute cloud, simply register your agents using the following code pattern

// revise as needed
const 
	need$ = false,
	needFetch = false,
	totems = {
		RLEU: "https://RLENET.126:8443",
		RLEC: "https://RLENET.216:8443",
		WWW: "http://totem.hopto.org",
		SBU: "https://totem.nga.mil:8443",
		COE: "https://totem.coe.nga.mil",
		LOCAL: "http://localhost:8080"
	},
	totem = totems.LOCAL,
	port = 3333,
	agents = {	// define your agents here
	};

// do not alter
const
	{ $ } = need$
		? require("/mnt/totem/man")
		: { $ : null },

	{ Fetch } = needFetch 
			? require("/mnt/totem/enums")
			: { Fetch: ( url, cb ) => {
	const {pathname,protocol} = new URL(url);
	switch (protocol) {
		case "http:":
			return require("http").get(url, res => {
				var txt = "";
				res.on("data", data => txt += data.toString());
				res.on("end", () => cb(txt) );
			}).end();

		case "file:": 
			return require("fs").readFile( "."+pathname, "utf-8", (err,txt) => cb(err?null:txt) );
	}
} };

Fetch(`${totem}/agent?port=${port}&keys=${Object.keys(agents)}`, setup => eval(setup));

For example, to create add, cat, getinfo, dft, python, R and opencv agents that listen for requests on port 3333, replace the agents = {}; with

agents = {	// define your agents here
	add: (req,res) => {	// an agent to add x and y
		const 
			{x,y} = req.query,
			calc = (x,y) => x+y;

		res( calc(x,y) );
	},

	cat: (req,res) => {	// an agent to concatenate x with y
		const 
			{x,y} = req.query,
			calc = (x,y) => [x,y];

		res( calc(x,y) );
	},

	backlog: (req,res) => {	// check my task queue
		Fetch( `${totem}/agent?tasks=all`, msg => {
			const tasks = JSON.parse(msg);

			res( `I've got ${tasks.length} to work on` );
			tasks.forEach( task => {
				console.log("working", task);
			});
		});
	},

	getfile: (req,res) => {	// send a text file
		Fetch( "file:/root/test.txt", txt => res(txt) );
	},

	// following agents require $ 

	dft: (req,res) => {	// fft of a large-dim x-post array + with a small-dim a-query offset 
		const
			{ y } = ctx = $("y=dft(x+a)", {		// context for $
				x: req.body.x || [0,0,0,0,0,0,0,1,1,1,0,0,0,0,0,0,0],
				a: req.query.a
			});

		res( y.$("re&im") );
	},

	python: (req,res) => {	// test python engine
		const
			{ a } = $.py(`
		import numpy as np;

		print 'console log: an array if i need it', np.array([1,2,3]);

		def f(x,y):
			return x+y;

		a=f(x,y)
		`, {
				x: req.query.x,
				y: req.query.y
			});

		res( a );
	},

	R: (req,res) => {	// test R engine
		const
			{ a } = ctx = $.R(`
		print('you da man');
		print('R input ctx=');str(CTX);
		CTX$d = 'this is a test';
		CTX$e = list(x=1,y=2,z=3);
		CTX$a = CTX$x + CTX$y;
		CTX$g = list(4,5,6);
		CTX$h = TRUE;
		`, {
				x: req.query.x,
				y: req.query.y
			});

		//console.log(ctx);
		res( ctx );
	},

	opencv: (req,res) => {	// test opencv/caffe engine
		const 
			ctx = $.cv("dummy code", {
				output: {	// classifier output port parms
					scale: 0,
					dim: 100,
					delta: 1,
					hits: 10,
					cascade: ["path1", "path2"]
				},
				input: {// image input port parms
				}
			});

		res( ctx );
	}
};

where

  • the last 5 agents require need$ enabled to include TOTEMs matrix minupulator
  • the last 3 agents demonstrate R-opencv-python support (assummed installed on your host).
  • enable needFetch to use TOTEM's full-featured Fetch.

Program Reference

agent

Attach endpoint agents to your compute cloud. Regulate workflows. Fetch data.

Requires: module:http, module:https, module:os, module:busy, module:mime, module:@totemorg/enums

agent.fileCache

File fileCache

Kind: static constant of agent

agent.getAgent()

Kind: static method of agent

agent.getFile()

Kind: static method of agent

agent.getGraph()

Kind: static method of agent

agent.getJson()

Kind: static method of agent

agent.attachAgents(server, port, agents, init)

Attach (req,res)-agent(s) to the server listening on the specified port. Callsback the supplied (req,res)-agent or the (req,res)-agent as derived from the req requested path, table, type, area.

The outer-most socThread adds socket data to the request:

	cookie: "...."		// client cookie string
	agent: "..."		// client browser info
	ipAddress: "..."	// client ip address
	referer: "http://site"		// url during a cross-site request
	method: "GET|PUT|..." 		// http request method
	now: date			// date stamp when requested started
	post: "..."			// raw body text
	url	: "/query"		// requested url path
	reqSocket: socket	// socket to retrieve client cert, post etc
	resSocket: socket	// method to create socket to accept response
	cert: {...} 		// full client cert

The dataThread adds dataset information:

	sql: {...}			// sql connector
	ds:	"focus.table"	// fully qualified sql table
	action: "select|update| ..."	// corresponding crude name

The routeThread adds client data:

	encrypted: bool		// true if request on encrypted server
	site: {...}			// site context
	mimi: "type"		// mime type of response
	
The resThread adds node information and url parameters:

	path: "/[area/...]name.type"	// full node path
	area: "name"		// file area being requested
	table: "name"		// name of dataset/table being requested
	type: "type" 		// type descriptor 

	query: {...} 		// raw keys from url
	where: {...} 		// sql-ized query keys from url
	body: {...}			// body keys from request 
	flags: {...} 		// flag keys from url
	index: {...}		// sql-ized index keys from url

And the inner-most agentThread adds client information:

	client: "..."		// client's name derived from cert or cookie
	profile: {...},		// client's profile

Kind: static method of agent

ParamTypeDescription
serverObjectServer being started
portNumericPort number to listen for agent requests
agentsfunction | Objectthe (req,res)-agent or a hash of (req,res)-agents
initfunctionOptional callback after server started

agent.Regulate()

Regulate callbacks to cb(t,ctx) on events defined by a trigger spec:

	spec = STEP / ONSTEPS / OFFSTEPS / START / END

where

	subspec = NUMBER | NUMBER [s|m|h|d|y] | dd-mm-yr

The context ctx can define:

	ctx.queue = "task queue" to share job status
	ctx.batch = NUMBER of records returned in callback cb(t,recs)
	ctx.client = "client name" to chage this request
	ctx.source = "sql table" record source
	ctx.fields = "field, ..." source fields
	ctx.priority = NUMBER of seconds to monitor signoff-status if required by client

Kind: static method of agent

agent.Fetch(ref, cb, cb)

Fetch (GET || PUT || POST || DELETE) information at a specified ref url

    PROTOCOL://HOST/FILE ? QUERY & FLAGS
    SITEREF

by using a `data` callback || Array || Object || null where PROTOCOL is

    http(s) 	=	http (https) protocol
    curl(s) 	=	curl (curls uses certs/fetch.pfx to authenticate)
    wget(s)		=	wget (wgets uses certs/fetch.pfx to authenticate)
    mask 		=	http access via rotated proxies
    file		=	file or folder path can include wild cards
    lexnex 		=	Lexis-Nexis oauth access to documents
    mysql		=	mysql database
    neo4j		= 	neo4j database

When a file path is "/"-terminated, a folder index is returned.  File paths 
may also contain wild-* cards.  

Optional FLAGS

    batch 	= NUMBER of records to fetch 
    offset	= NUMBER of records to offset fetch
    select	="FIELD, ..." to fetch
    keys	= [...] header keys in csv source
    comma	= "delim" for csv source
    newline = "delim" for csv source

Kind: static method of agent

ParamTypeDescription
refStringsource URL
cbstring | array | function | nullcallback or data
cbfunctionoptional callback when first cb is data

Example

Fetch( ref, text => {			// get request
    })

    

Example

Fetch( ref, [ ... ], stat => { 	// post request with data hash list
    })

    

Example

Fetch( ref, { ... }, stat => { 	// put request with data hash
    })

    

Example

Fetch( ref, null, stat => {		// delete request 
    })

agent~quickLinks

Fetch quick SITEREFs

    https://www.programmableweb.com/search/military

    ACLED
    https://www.programmableweb.com/api/acled-rest-api-v25
    ACCT teliy40602@plexfirm.com / ACLEDsnivel1
    API https://developer.acleddata.com/rehd/cms/views/acled_api/documents/API-User-Guide.pdf
    SITE https://developer.acleddata.com/
    The Armed Conflict Location & Event Data Project (ACLED) is a real-time data and and crisis analysis and mapping project on political violence and protest across the world. ACLED's mission is to produce dis-aggregated, locally informed data and analysis globally in real time. An ACLED REST API enables users to retrieve data about actors, actor type, country, region and get data in JSON, XML, CSV or text. Filter data by date, area, pagination, etc.

    Animetrics FIMS
    https://www.programmableweb.com/api/animetrics-fims-cloud-rest-api
    http://animetrics.com/fims-cloud
    Aimed at the law enforcement, security and military intelligence industries, Animetrics' FaceR Identity Management Solution (FIMS) allows organizations to bring mobile security and video surveillance facial-biometrics applications into the field for use in real time. FIMS Cloud is used to centralize and access a user's cloud based photographic stores using facial recognition. FIMS Cloud organizes, searches and centralizes access to photographic stores using 1:many web service based verification engine. Access to the service is provided via a RESTful API. Public documentation is not available.

    Navlost WXT Weather Tesseract
    https://www.programmableweb.com/api/navlost-wxt-weather-tesseract
    The WXT Weather Service provides atmospheric weather information through a REST architecture, HTTP requests, and JSON formats. It integrates METAR/TAF information, sun, and moon calculations, targeting aviation and energy applications. Advanced features include: -Upper atmosphere information (e.g., research, aviation, rocketry, military) -Automated, push-type notification of arbitrary weather-related events (alert service) -Calculation of arbitrary results derived from weather forecast information via a server-side scripting language. The default response type is application/json, although other formats may be supported. At the present time, there is partial support for comma-separated value (CSV) responses.
    https://wxt.navlost.eu/api/v1/
    https://wxt.navlost.eu/doc/api/

Kind: inner constant of agent

agent~maxFiles

Max files to Fetch when indexing a folder

Kind: inner constant of agent

agent~maxRetry

Fetch wget/curl maxRetry

Kind: inner constant of agent

agent~certs

Fetching certs

Kind: inner constant of agent

Contacting, Contributing, Following

Feel free to


© 2012 ACMESDS

1.6.0

6 months ago

1.5.0

6 months ago

1.4.0

6 months ago

1.3.0

6 months ago