timequerylog v45.1.0
(See also tql-cli
command line tool.)
JSONL (newline-separated JSON) logging separated into a files per hour of day with simple query between start and end time with match function.
The idea is to make it more efficient to query logs for a specific time period or type of data/event by breaking up the files in a consistent way.
You may want to use sub-types to break out data to improve efficiency when the
sub-type data doesn't always need to be queried immediately.
For example, instead of logging all details of an error
type you may log
a main error
type with the core details of errors and a separate error-details
type including info such as stack traces.
Usage
Log some data
import {log} from 'timequerylog';
log('request', {url:'http://www.reddit.com/'});
Read last record for a type
import {latest} from 'timequerylog';
latest('request').then(row => console.log(row));
Query data with a timestamp after 1/1/1995 GMT
import {query} from 'timequerylog';
query('request', new Date('01-01-1995'), new Date())
.then(rows => console.log(rows));
Set directory for logging, don't repeat rows
import {config} from 'timequerylog';
config({path:process.cwd()+'/datalog', noRepeat: true});
Log using a different time than the present
log('req',{cat:100}, new Date('2015-05-01T23:50:59.392Z'));
Query logs from last 30 minutes matching function
query('req', null, null, d => d.amount > 1000).then(console.log);
Return logs as an object mode stream
const matched = queryOpts({type:'request', start: new Date('2016-10-22 10:00 AM'),
end: new Date('2016-10-23')});
matched.on('data', console.log);
Match, map results, convert time to MS since epoch, and return CSV text stream
let i = 0;
const csvStream = queryOpts({type:'event', csv: true, timeMS: true,
match: r => r.category == 'update',
start: new Date('1995-12-25'),
map: (r)=>{r.row = i++;return r}});
csvStream.pipe(process.stdout);
Increment
let n = await incr('test'); // n = 1
n = await incr('test'); // n = 2
let m = await incr('test2',Date.now())
// if m was not previously incremented
// then m is Date.now()
// otherwise previous m + 1
const unixTime = Math.round(Date.now()/1000));
// actually saves unixTime - 1
await setIncr('test2', unixTime);
n = await incr('test2');
// n == unixTime
// need to load first with async call
let load = incr('test3',Date.now());
// loaded into memory, can now incr in memory
// and save after immediate return
let imm = incrNow('test3');
// if the process is killed suddenly
// without Node.js events completing, may not save
// updated value..
getTypes, queryMultiArray, high res time (hrms)
async function testMulti() {
console.log("Current high-res time:", hrms());
const types = await getTypes('*');
console.log(types);
const req = await getTypes('r*');
console.log(req);
log('testhr', {n:'tom',hrtime:hrms()});
log('testhr', {n:'bob',hrtime:hrms()});
await delay(10);
const args = { typeGlob: '*', start: new Date('1980-01-01'),
end: Date.now() };
const results = await queryMultiArray(args);
console.log(results);
}
Functions/Options
log(type, object, time = current time)
Uses JSON stream to log some data to the file ./[type]_GMT/DATE/HOUR
.
Returns a promise.
queryRecent(type)
Return a promise with data from the last 30 minutes for type
. Searches JSON streamed files starting
from directory ./[type]_GMT
.
query(type, startDate, endDate, matchFunction)
Returns a promise with data for type
between startDate
and endDate
where matchFunction
returns true. Searches JSONL files starting from directory ./[type]_GMT
.
config(opts)
Set configuration options. Defaults are: {path: process.cwd(), noRepeat: false},
noRepeat:{[type]:true}
makes it ignore rows that are duplicates (besides time).
To delete files older than X days, specify
config({path:'datalog'}, deleteOldDays: {event: 100}}
) --
deletes old 'event' type files 100 days or older.
queryOpts({type, start, end, match = (d=>true), csv=false,map, timeMS=false})
This query function takes an options object and returns either an objectMode stream or a CSV stream.
If end
is not specified the current time is used. If start
is not specified then end
-30 minutes is
used. map
is an optional function to modify rows. timeMS
will return time as MS since epoch.
incr(key, init=0) or incrNow(key, init) (non-promise version)
Increment key
and return new value. Stored in file key_INCR
. Optionally init at some number.
If you awaitincr(key,undefined,true)
first to load you can use the immediate (non-async) incrNow
to update the value in memory without waiting for the file save to complete storing the new number.
setIncr(key, val)
Set counter for key
to val-1
so next call to incr(key)
returns val
.
5 years ago
5 years ago
5 years ago
5 years ago
6 years ago
6 years ago
6 years ago
7 years ago
8 years ago
8 years ago
8 years ago
8 years ago
8 years ago
8 years ago
8 years ago
8 years ago
8 years ago
8 years ago
8 years ago
8 years ago
8 years ago
8 years ago
8 years ago
8 years ago
8 years ago
8 years ago
8 years ago
8 years ago
8 years ago
8 years ago
8 years ago
8 years ago
8 years ago
8 years ago
8 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
10 years ago
10 years ago
10 years ago
10 years ago
10 years ago
10 years ago
10 years ago
10 years ago
10 years ago
10 years ago
10 years ago
10 years ago
10 years ago