45.1.0 • Published 5 years ago

timequerylog v45.1.0

Weekly downloads
5
License
ISC
Repository
github
Last release
5 years ago

(See also tql-cli command line tool.)

JSONL (newline-separated JSON) logging separated into a files per hour of day with simple query between start and end time with match function.

The idea is to make it more efficient to query logs for a specific time period or type of data/event by breaking up the files in a consistent way.

You may want to use sub-types to break out data to improve efficiency when the sub-type data doesn't always need to be queried immediately. For example, instead of logging all details of an error type you may log a main error type with the core details of errors and a separate error-details type including info such as stack traces.

Usage

Log some data

import {log} from 'timequerylog';

log('request', {url:'http://www.reddit.com/'});

Read last record for a type

import {latest} from 'timequerylog';
latest('request').then(row => console.log(row));

Query data with a timestamp after 1/1/1995 GMT

import {query} from 'timequerylog';

query('request', new Date('01-01-1995'), new Date())
.then(rows => console.log(rows));

Set directory for logging, don't repeat rows

import {config} from 'timequerylog';

config({path:process.cwd()+'/datalog', noRepeat: true});

Log using a different time than the present

log('req',{cat:100}, new Date('2015-05-01T23:50:59.392Z'));

Query logs from last 30 minutes matching function

query('req', null, null, d => d.amount > 1000).then(console.log);

Return logs as an object mode stream

const matched = queryOpts({type:'request', start: new Date('2016-10-22 10:00 AM'),
                           end: new Date('2016-10-23')});
matched.on('data', console.log);

Match, map results, convert time to MS since epoch, and return CSV text stream

  let i = 0;
  const csvStream = queryOpts({type:'event', csv: true, timeMS: true,
                               match: r => r.category == 'update',
                               start: new Date('1995-12-25'),
                               map: (r)=>{r.row = i++;return r}});
  csvStream.pipe(process.stdout);

Increment

  let n = await incr('test'); // n = 1
  n = await incr('test'); // n = 2
  let m = await incr('test2',Date.now()) 
  // if m was not previously incremented
  // then m is Date.now()
  // otherwise previous m + 1

  const unixTime = Math.round(Date.now()/1000));
  // actually saves unixTime - 1
  await setIncr('test2', unixTime);
  n = await incr('test2');
  // n == unixTime

  // need to load first with async call
  let load = incr('test3',Date.now());
  // loaded into memory, can now incr in memory
  // and save after immediate return
  let imm = incrNow('test3');
  // if the process is killed suddenly
  // without Node.js events completing, may not save
  // updated value..

getTypes, queryMultiArray, high res time (hrms)

async function testMulti() {
  console.log("Current high-res time:", hrms());

  const types = await getTypes('*');
  console.log(types);
  const req = await getTypes('r*');
  console.log(req);
 
  log('testhr', {n:'tom',hrtime:hrms()});
  log('testhr', {n:'bob',hrtime:hrms()});
  await delay(10);
  const args = { typeGlob: '*', start: new Date('1980-01-01'),
                 end: Date.now() };
  const results = await queryMultiArray(args);
  console.log(results);
}

Functions/Options

log(type, object, time = current time)

Uses JSON stream to log some data to the file ./[type]_GMT/DATE/HOUR. Returns a promise.

queryRecent(type)

Return a promise with data from the last 30 minutes for type. Searches JSON streamed files starting from directory ./[type]_GMT.

query(type, startDate, endDate, matchFunction)

Returns a promise with data for type between startDate and endDate where matchFunction returns true. Searches JSONL files starting from directory ./[type]_GMT.

config(opts)

Set configuration options. Defaults are: {path: process.cwd(), noRepeat: false}, noRepeat:{[type]:true} makes it ignore rows that are duplicates (besides time).

To delete files older than X days, specify config({path:'datalog'}, deleteOldDays: {event: 100}}) -- deletes old 'event' type files 100 days or older.

queryOpts({type, start, end, match = (d=>true), csv=false,map, timeMS=false})

This query function takes an options object and returns either an objectMode stream or a CSV stream. If end is not specified the current time is used. If start is not specified then end-30 minutes is used. map is an optional function to modify rows. timeMS will return time as MS since epoch.

incr(key, init=0) or incrNow(key, init) (non-promise version)

Increment key and return new value. Stored in file key_INCR. Optionally init at some number. If you awaitincr(key,undefined,true) first to load you can use the immediate (non-async) incrNow to update the value in memory without waiting for the file save to complete storing the new number.

setIncr(key, val)

Set counter for key to val-1 so next call to incr(key) returns val.

45.1.0

5 years ago

45.0.0

5 years ago

44.0.0

5 years ago

43.0.0

5 years ago

42.0.0

6 years ago

41.0.0

6 years ago

40.0.0

6 years ago

39.0.0

7 years ago

38.0.0

8 years ago

37.0.0

8 years ago

36.0.0

8 years ago

35.0.0

8 years ago

34.0.0

8 years ago

33.0.0

8 years ago

32.0.0

8 years ago

31.0.0

8 years ago

30.0.0

8 years ago

29.0.0

8 years ago

28.0.0

8 years ago

27.0.0

8 years ago

26.0.0

8 years ago

25.0.0

8 years ago

24.1.0

8 years ago

24.0.0

8 years ago

23.0.0

8 years ago

22.0.0

8 years ago

21.0.0

8 years ago

20.0.0

8 years ago

19.0.0

8 years ago

18.1.0

8 years ago

18.0.0

8 years ago

16.1.0

8 years ago

17.0.0

8 years ago

16.0.0

8 years ago

15.0.0

8 years ago

14.1.0

9 years ago

14.0.3

9 years ago

14.0.2

9 years ago

14.0.1

9 years ago

14.0.0

9 years ago

13.0.0

9 years ago

12.0.0

9 years ago

11.1.0

9 years ago

11.0.0

9 years ago

10.1.0

9 years ago

10.0.0

9 years ago

9.0.0

9 years ago

8.0.0

9 years ago

7.0.0

9 years ago

6.2.2

9 years ago

6.2.1

9 years ago

6.2.0

9 years ago

6.1.0

9 years ago

6.0.1

9 years ago

6.0.0

9 years ago

5.0.0

10 years ago

4.6.0

10 years ago

4.5.0

10 years ago

4.4.0

10 years ago

4.3.0

10 years ago

4.2.0

10 years ago

4.1.0

10 years ago

4.0.0

10 years ago

3.0.0

10 years ago

2.0.0

10 years ago

1.2.0

10 years ago

1.1.0

10 years ago

1.0.0

10 years ago