frxt v1.0.11
frxt
freetext filter for sql-esque data
About
frxt is a refractor and update to the library json-ctrl by combining several of the smaller dependencies (json-cnf, json-tkn, json-tim, and jsqlon). As such there are many moving pieces for which this README.md will try to explain and address.
Assumptions
A core assumption of this library is that the stored data is SQL-eqsue. In other words, that the JSON object's outer most keys correspond to record ids and that each record (an object) is structured as if in a SQL table i.e. each record contains the same fields:
json = {
id1: {
field1: value1f1,
field2: value1f2,
field3: value1f3,
},
// ...
idn: {
field1: valueNf1,
field2: valueNf2,
field3: valueNf3,
}
}It may be the case that users need to overwrite the default behavior of how field values are treated. For this three objects exists in which overriding functions can be set. These include:
fieldRenderFunctions: how a given field's value should be altered when rendering it on the page.fieldSortByFunctions: how a given field's value should be altered when comparing against other values for sorting.fieldFilterFunctions: how a given field's value should be altered when comparing against specified filters.
To overwrite a field's function (e.g. fieldX), a function with the following signature is expected:
function overwriteFieldX(id, field, record) {
let value = record[field]
// manipulate value to be returned.
return value
}then set the corresponding object (e.g. fieldRenderFunctions) to have this function with the field name as the key:
fieldRenderFunctions = {
fieldX: overwriteFieldX
}Example
let json = {
'banana': {
'fruit': 'banana'
},
'pear': {
'fruit': 'pear'
},
'apple': {
'fruit': 'apple'
}
}
let inputString = 'fruit is banana or type contains app'
let tknFieldMap = {
'fruit': 'fruit',
'Type': 'fruit',
'name': 'fruit',
}
let [valid, error] = frxt.extractFiltersFromText({
input:inputString,
tknFieldMap:tknFieldMap,
tknFunctionMap:frxt.configs.defaultCNFFunctionMap,
tknConditionalMap:frxt.configs.defaultCNFConditionalMap,
fieldTokens:undefined,
functionTokens:undefined,
conditionalTokens:undefined,
scrubInputPattern:frxt.configs.PunctuationPattern,
cnfLogicConfig:frxt.configs.defaultCNFLogicConfig,
cnfFunctionsConfig:frxt.configs.defaultCNFFunctionConfig,
cnfConditionalConfig:frxt.configs.defaultCNFConditionalConfig,
debug:false
})
expect(valid.length).toBe(2)
expect(error.length).toBe(0)
expect(valid[0].filter.field).toBe('fruit')
expect(valid[1].filter.field).toBe('fruit')
let filters = valid.map(({filter})=>filter)
let res = frxt.conjunctiveNormalFormFilter(
json,
filters,
frxt.configs.defaultCNFLogicConfig,
frxt.configs.defaultCNFFunctionConfig,
frxt.configs.defaultCNFConditionalConfig,
{}
)
expect(res.length).toBe(2)
expect(res[0]).toBe('banana')
expect(res[1]).toBe('apple')Filtering
Filtering of the records in the stored JSON occurs in four steps:
1. records are passed through the specified conjunctive normal form filters. (This utilizes specified fieldFilterFunctions).
2. the records passing step 1 are passed through the global RegEx (This utilizes specified fieldRenderFunctions, as this is what the user sees).
3. the records are sorted via TimSort from the given sortSpecification (This utilizes the specified fieldSortByFunctions).
4. the records of the specified "page" are returned.
Sorting
Sorting is handled via the state variable sortSpecification.
sortSpecification is an array of objects, where each object has the following structure:
sortSpec = {
field: fieldName,
isAscending: boolean
}This specification allows for the fields, in the order provided in sortSpecification to undergo TimSort.
Pagination
Pagination is provided for controlling which records are show to the user. It is worthwhile that this is local pagination, therefore all the data is already in memory. This stems from the combination of functions applied to the data outlined in the section regarding filtering. Consider the question "what are the five elements on page two?". If filters, RegEx and sorting are in play, there is no way to know this without finding all records that pass and sorting them to know which will be on page two.
What's Inside?
defaults: default configurationsfilter: conjunctive normal form filter for SQL-esque JSONfreetext: extract filters from user provided freeform textjsonsql: utilities for SQL-esque JSONsort: timsort for SQL-esque JSONtypes: typescript types for this packageutils: grab bag of utility functions
Configuration
Utility Functions
There is a lot which can be configured to achieve the desired behavior. As these configurations are mostly independent, they are split up into sub-configs. These are passed to the utility functions from the components.
I already touched on the fieldRenderFunctions, fieldSortByFunctions, fieldFilterFunctions above. In addition there are:
1. configCNFLogic: The logical options for CNF.
2. configCNFFunctions: The functions which can be applied during CNF.
3. configCNFConditionals: The conditionals (or tests) which can be applied during CNF.
4. configCNFExtractors:
5. configTKNFunctionMap: A mapping of tokens to functions in the CNF.
6. configTKNConditionalMap: A mapping of tokens to conditionals in the CNF.
configCNFLogic
The two logical operations supported are and and or. You can change how they
are displayed in VFilterView as so:
{
and: {
display: "∧"
},
or: {
display: "∨"
}
}configCNFFunctions
These are the known functions and corresponding types they work on for the free text filter.
{
identity: {
display: "x → x",
function: (x) => x,
types: ["number", "string", "array:number", "undefined"]
},
abs: {
display: "abs",
function: (x) => Math.abs(x),
types: ["number"]
},
mean: {
display: "mean",
function: (nums) => (nums.reduce((add, val) => add + val) / nums.length),
types: ["array:number"]
},
max: {
display: "max",
function: (nums) => Math.max(...nums),
types: ["array:number"]
},
min: {
display: "min",
function: (nums) => Math.min(...nums),
types: ["array:number"]
},
length: {
display: "len",
function: (arr) => arr.length,
types: ["array", "array:number", "array:string", "array:object"]
}
}configCNFConditionals
Here you find which conditionals exist
{
eq: {
display: "=",
conditional: (a, b) => {
// intentional == rather than === to allow for 1 == "1" to be true
return a == b;
},
types: [
"array", "array:string", "array:object", "array:number",
"number", "string", "undefined"
]
},
neq: {
display: "≠",
types: [
"array", "array:string", "array:object", "array:number",
"number", "string", "undefined"
],
conditional: (a, b) => a != b,
},
lt: {
display: "<",
types: ["number"],
conditional: (a, b) => a < b,
},
gt: {
display: ">",
types: ["number"],
conditional: (a, b) => a > b,
},
lte: {
display: "≤",
types: ["number"],
conditional: (a, b) => a <= b,
},
gte: {
display: "≥",
types: ["number"],
conditional: (a, b) => a >= b,
},
ss: {
display: "⊂",
types: ["array", "array:string", "array:number", "string"],
conditional: (a, b) => {
let includes = false
if (Array.isArray(a)) {
for (let i = 0; i < a.length; i++) {
if (a[i] == b) includes = true
if (includes) return includes
}
} else if (typeof a === 'string') {
return a.includes(b)
} else { return includes }
return includes
}
},
nss: {
display: "⟈",
types: ["array", "array:string", "array:number", "string"],
conditional: (a, b) => {
let includes = true
if (Array.isArray(a)) {
return a.every(e => e != b)
} else if (typeof a === 'string') {
return !a.includes(b)
}
return includes
}
}
}configTKNFunctionMap
There are the default tokens that map tokens (the keys) to the corresponding
values in configCNFFunctions.
{
max: "max",
min: "min",
maximum: "max",
minimum: "min",
length: "length",
len: "length",
mean: "mean",
median: "median",
average: "mean",
ave: "mean",
identity: "identity"
}configTKNConditionalMap
There are the default tokens that map tokens (the keys) to the corresponding
values in configCNFConditionals.
{
eq: "eq",
is: "eq",
equal: "eq",
"=": "eq",
"!=": "neq",
"≠": "neq",
">": "gt",
"≥": "gte",
">=": "gte",
"<": "lt",
"≤": "lte",
"<=": "lte",
"is not": "neq",
neq: "neq",
"not equal to": "neq",
gt: "gt",
"greater than": "gt",
"less than": "lt",
lt: "lt",
"less than or equal to": "lte",
"greater than or equal to": "gte",
"member of": "ss",
substring: "ss",
contains: "ss",
includes: "ss"
}