@totemorg/agent v1.6.0
AGENT
Add TOTEM agents to your compute cloud. Regulate workflows. Fetch data.
Install
npm install @totemorg/enums
Manage
npm run start ? # List start options
npm run verminor # Roll minor version
npm run vermajor # Roll major version
npm run redoc # Regen documentation
npm run pubminor # republish as minor version
npm run pubmajor # republish as major version
Usage
To add your compute agents to TOTEM's compute cloud, simply register your agents using the following code pattern
// revise as needed
const
need$ = false,
needFetch = false,
totems = {
RLEU: "https://RLENET.126:8443",
RLEC: "https://RLENET.216:8443",
WWW: "http://totem.hopto.org",
SBU: "https://totem.nga.mil:8443",
COE: "https://totem.coe.nga.mil",
LOCAL: "http://localhost:8080"
},
totem = totems.LOCAL,
port = 3333,
agents = { // define your agents here
};
// do not alter
const
{ $ } = need$
? require("/mnt/totem/man")
: { $ : null },
{ Fetch } = needFetch
? require("/mnt/totem/enums")
: { Fetch: ( url, cb ) => {
const {pathname,protocol} = new URL(url);
switch (protocol) {
case "http:":
return require("http").get(url, res => {
var txt = "";
res.on("data", data => txt += data.toString());
res.on("end", () => cb(txt) );
}).end();
case "file:":
return require("fs").readFile( "."+pathname, "utf-8", (err,txt) => cb(err?null:txt) );
}
} };
Fetch(`${totem}/agent?port=${port}&keys=${Object.keys(agents)}`, setup => eval(setup));
For example, to create add
, cat
, getinfo
, dft
, python
, R
and opencv
agents that listen for requests on port 3333, replace
the agents = {};
with
agents = { // define your agents here
add: (req,res) => { // an agent to add x and y
const
{x,y} = req.query,
calc = (x,y) => x+y;
res( calc(x,y) );
},
cat: (req,res) => { // an agent to concatenate x with y
const
{x,y} = req.query,
calc = (x,y) => [x,y];
res( calc(x,y) );
},
backlog: (req,res) => { // check my task queue
Fetch( `${totem}/agent?tasks=all`, msg => {
const tasks = JSON.parse(msg);
res( `I've got ${tasks.length} to work on` );
tasks.forEach( task => {
console.log("working", task);
});
});
},
getfile: (req,res) => { // send a text file
Fetch( "file:/root/test.txt", txt => res(txt) );
},
// following agents require $
dft: (req,res) => { // fft of a large-dim x-post array + with a small-dim a-query offset
const
{ y } = ctx = $("y=dft(x+a)", { // context for $
x: req.body.x || [0,0,0,0,0,0,0,1,1,1,0,0,0,0,0,0,0],
a: req.query.a
});
res( y.$("re&im") );
},
python: (req,res) => { // test python engine
const
{ a } = $.py(`
import numpy as np;
print 'console log: an array if i need it', np.array([1,2,3]);
def f(x,y):
return x+y;
a=f(x,y)
`, {
x: req.query.x,
y: req.query.y
});
res( a );
},
R: (req,res) => { // test R engine
const
{ a } = ctx = $.R(`
print('you da man');
print('R input ctx=');str(CTX);
CTX$d = 'this is a test';
CTX$e = list(x=1,y=2,z=3);
CTX$a = CTX$x + CTX$y;
CTX$g = list(4,5,6);
CTX$h = TRUE;
`, {
x: req.query.x,
y: req.query.y
});
//console.log(ctx);
res( ctx );
},
opencv: (req,res) => { // test opencv/caffe engine
const
ctx = $.cv("dummy code", {
output: { // classifier output port parms
scale: 0,
dim: 100,
delta: 1,
hits: 10,
cascade: ["path1", "path2"]
},
input: {// image input port parms
}
});
res( ctx );
}
};
where
- the last 5 agents require need$ enabled to include TOTEMs matrix minupulator
- the last 3 agents demonstrate R-opencv-python support (assummed installed on your host).
- enable needFetch to use TOTEM's full-featured Fetch.
Program Reference
agent
Attach endpoint agents to your compute cloud. Regulate workflows. Fetch data.
Requires: module:http, module:https, module:os, module:busy, module:mime, module:@totemorg/enums
agent.fileCache
File fileCache
Kind: static constant of agent
agent.getAgent()
Kind: static method of agent
agent.getFile()
Kind: static method of agent
agent.getGraph()
Kind: static method of agent
agent.getJson()
Kind: static method of agent
agent.attachAgents(server, port, agents, init)
Attach (req,res)-agent(s)
to the server
listening on the specified port
. Callsback the supplied (req,res)-agent
or
the (req,res)-agent
as derived from the req
requested path
, table
, type
, area
.
The outer-most socThread adds socket data to the request:
cookie: "...." // client cookie string
agent: "..." // client browser info
ipAddress: "..." // client ip address
referer: "http://site" // url during a cross-site request
method: "GET|PUT|..." // http request method
now: date // date stamp when requested started
post: "..." // raw body text
url : "/query" // requested url path
reqSocket: socket // socket to retrieve client cert, post etc
resSocket: socket // method to create socket to accept response
cert: {...} // full client cert
The dataThread adds dataset information:
sql: {...} // sql connector
ds: "focus.table" // fully qualified sql table
action: "select|update| ..." // corresponding crude name
The routeThread adds client data:
encrypted: bool // true if request on encrypted server
site: {...} // site context
mimi: "type" // mime type of response
The resThread adds node information and url parameters:
path: "/[area/...]name.type" // full node path
area: "name" // file area being requested
table: "name" // name of dataset/table being requested
type: "type" // type descriptor
query: {...} // raw keys from url
where: {...} // sql-ized query keys from url
body: {...} // body keys from request
flags: {...} // flag keys from url
index: {...} // sql-ized index keys from url
And the inner-most agentThread adds client information:
client: "..." // client's name derived from cert or cookie
profile: {...}, // client's profile
Kind: static method of agent
Param | Type | Description |
---|---|---|
server | Object | Server being started |
port | Numeric | Port number to listen for agent requests |
agents | function | Object | the (req,res)-agent or a hash of (req,res)-agents |
init | function | Optional callback after server started |
agent.Regulate()
Regulate callbacks to cb(t,ctx) on events defined by a trigger spec:
spec = STEP / ONSTEPS / OFFSTEPS / START / END
where
subspec = NUMBER | NUMBER [s|m|h|d|y] | dd-mm-yr
The context ctx can define:
ctx.queue = "task queue" to share job status
ctx.batch = NUMBER of records returned in callback cb(t,recs)
ctx.client = "client name" to chage this request
ctx.source = "sql table" record source
ctx.fields = "field, ..." source fields
ctx.priority = NUMBER of seconds to monitor signoff-status if required by client
Kind: static method of agent
agent.Fetch(ref, cb, cb)
Fetch (GET || PUT || POST || DELETE) information at a specified ref
url
PROTOCOL://HOST/FILE ? QUERY & FLAGS
SITEREF
by using a `data` callback || Array || Object || null where PROTOCOL is
http(s) = http (https) protocol
curl(s) = curl (curls uses certs/fetch.pfx to authenticate)
wget(s) = wget (wgets uses certs/fetch.pfx to authenticate)
mask = http access via rotated proxies
file = file or folder path can include wild cards
lexnex = Lexis-Nexis oauth access to documents
mysql = mysql database
neo4j = neo4j database
When a file path is "/"-terminated, a folder index is returned. File paths
may also contain wild-* cards.
Optional FLAGS
batch = NUMBER of records to fetch
offset = NUMBER of records to offset fetch
select ="FIELD, ..." to fetch
keys = [...] header keys in csv source
comma = "delim" for csv source
newline = "delim" for csv source
Kind: static method of agent
Param | Type | Description |
---|---|---|
ref | String | source URL |
cb | string | array | function | null | callback or data |
cb | function | optional callback when first cb is data |
Example
Fetch( ref, text => { // get request
})
Example
Fetch( ref, [ ... ], stat => { // post request with data hash list
})
Example
Fetch( ref, { ... }, stat => { // put request with data hash
})
Example
Fetch( ref, null, stat => { // delete request
})
agent~quickLinks
Fetch quick SITEREFs
https://www.programmableweb.com/search/military
ACLED
https://www.programmableweb.com/api/acled-rest-api-v25
ACCT teliy40602@plexfirm.com / ACLEDsnivel1
API https://developer.acleddata.com/rehd/cms/views/acled_api/documents/API-User-Guide.pdf
SITE https://developer.acleddata.com/
The Armed Conflict Location & Event Data Project (ACLED) is a real-time data and and crisis analysis and mapping project on political violence and protest across the world. ACLED's mission is to produce dis-aggregated, locally informed data and analysis globally in real time. An ACLED REST API enables users to retrieve data about actors, actor type, country, region and get data in JSON, XML, CSV or text. Filter data by date, area, pagination, etc.
Animetrics FIMS
https://www.programmableweb.com/api/animetrics-fims-cloud-rest-api
http://animetrics.com/fims-cloud
Aimed at the law enforcement, security and military intelligence industries, Animetrics' FaceR Identity Management Solution (FIMS) allows organizations to bring mobile security and video surveillance facial-biometrics applications into the field for use in real time. FIMS Cloud is used to centralize and access a user's cloud based photographic stores using facial recognition. FIMS Cloud organizes, searches and centralizes access to photographic stores using 1:many web service based verification engine. Access to the service is provided via a RESTful API. Public documentation is not available.
Navlost WXT Weather Tesseract
https://www.programmableweb.com/api/navlost-wxt-weather-tesseract
The WXT Weather Service provides atmospheric weather information through a REST architecture, HTTP requests, and JSON formats. It integrates METAR/TAF information, sun, and moon calculations, targeting aviation and energy applications. Advanced features include: -Upper atmosphere information (e.g., research, aviation, rocketry, military) -Automated, push-type notification of arbitrary weather-related events (alert service) -Calculation of arbitrary results derived from weather forecast information via a server-side scripting language. The default response type is application/json, although other formats may be supported. At the present time, there is partial support for comma-separated value (CSV) responses.
https://wxt.navlost.eu/api/v1/
https://wxt.navlost.eu/doc/api/
Kind: inner constant of agent
agent~maxFiles
Max files to Fetch when indexing a folder
Kind: inner constant of agent
agent~maxRetry
Fetch wget/curl maxRetry
Kind: inner constant of agent
agent~certs
Fetching certs
Kind: inner constant of agent
Contacting, Contributing, Following
Feel free to
- submit and status TOTEM issues
- contribute to TOTEM notebooks
- revise TOTEM requirements
- browse TOTEM holdings
- or follow TOTEM milestones
© 2012 ACMESDS