xwebdb v0.0.8
XWebDB
pronounced: Cross Web Database
Documentation work in progess
What is this?
Comparision with other databases
Feature | LocalForage | PouchDB | Dexie.js | XWebDB |
---|---|---|---|---|
Minified Size | 29KB | 142KB | 80KB | 48KB |
Performance^ | good | good | good | fastest |
Query Language | Key/value | Map/Reduce | Mongo-like | Mongo-like |
Sync | no sync | CouchDB sync | paid/server | serverless services (free) |
Live Queries | unsupported | unsupported | supported | supported |
Aggregation | unsupported | unsupported | unsupported | supported |
A Word on Performance
XWebDB has a pretty good performance. It has the fastest insert, bulk-insert, update, delete, and read times even with large databases.
The performance of XWebDB can be attributed to:
- Most notably, a custom data structure that is specifically optimized for browser usage in JavaScript and offers a unique combination of a sorted array and a map. It leverages efficient methods, such as binary search, map get, and array concatenation, to provide high-performance operations for reading, inserting, and deleting elements. Unlike tree-based structures, which often require additional memory for node structures, this data structure minimizes memory overhead by only utilizing memory for the array and the map, resulting in improved performance and reduced memory consumption.
- A built-in caching mechanism, that is quite simple yet efficient.
- Leveraging bulk-operations in IndexedDB whenever possible, as it has been proven to be faster.
However, it is important to note that achieving such high-performance requires maintaining a complete copy of the database in memory. While this approach may seem unconventional, it poses no significant issues for the intended use cases of this database, particularly given today's standards. The memory footprint for storing 10,000 2KB documents is nearly 20MB, which is considered manageable.
Data complexity
- Get
O(1)
- Insert
O(log n)
- Delete
O(log n)
!WARNING While XWebDB may appear to be a promising choice for your next project compared to other databases, it is essential to carefully weigh your decision. Other solutions have undergone rigorous testing, have been battle-tested, and enjoy robust support from larger communities. This is not to discourage you from using XWebDB; in fact, I am currently using it in multiple projects myself. However, it's important to acknowledge that XWebDB is a relatively new project. With time, it is expected to mature and improve. I hope that in the future, this cautionary section can be removed from the documentation. Until then, it is advisable to thoroughly consider your options before making a final decision.
Quick start
Installation
Install using npm
npm install xwebdb
Alternatively, you can include the pre-built and minified file in your HTML:
<script src="https://unpkg.com/xwebdb/dist/xwebdb.min.js"></script>
Database creation and configuration
To create a database, you need to instantiate the Database class with a configuration object. The only required property is ref, which specifies the reference name of the database.
import { Database } from "xwebdb";
// Database creation and configuration
let db = new Database({
ref: "my-database", // Specify the reference name for the database
});
!WARNING The above example is oversimplified and shouldn't be used in a real-world application as it doesn't specify the model to be used in the database for object mapping.
For more advanced configurations, please refer to the Configuration section below.
Defining a model
You can define a document model by extending the Doc class and specifying its properties and methods:
import { Database, Doc } from "xwebdb";
// Define a document class
class Person extends Doc {
firstName: string = ""; // Define a firstName property
lastName: string = ""; // Define a lastName property
get fullName() {
return this.firstName + " " + this.lastName; // Define a computed property fullName
}
}
// Create a database with the specified model
let db = new Database<Person>({
ref: "my-database1",
model: Person, // Specify the document model
});
For more advanced object mapping, please refer to the Object mapping section below.
Operations
Once you have a database instance, you can perform various operations on it, such as creating, finding, updating, and deleting documents. Here are some examples:
// Creating a document
db.insert(Person.new({ firstName: "Ali" }));
// Create a new document with firstName "Ali"
// defining an index
db.createIndex({ fieldName: "firstName", unique: false, sparse: true });
// create an index on "firstName" field
// Finding documents
db.find({ firstName: "Ali" });
// Find documents with firstName "Ali"
// Using operators in queries
db.find({ firstName: { $eq: "Ali" } });
// Find documents with firstName equal to "Ali"
// Use aggregation method chaining
(await db.aggregate({ firstName: $eq }))
.$skip(1)
.$limit(5)
.$sort({ lastName: -1 })
.$addFields((doc) => ({ firstLetter: doc.firstName.charAt(0) }));
// Updating documents
db.update({ firstName: "Ali" }, { $set: { firstName: "Dina" } });
// Update firstName from "Ali" to "Dina"
// Deleting documents
db.delete({ firstName: { $eq: "Ali" } });
// Delete documents with firstName "Ali"
// reload database from persistence layer
db.reload();
// synchronizing database (an RSA must have been configured)
db.sync();
Those operations are explained extensively in their respective sections below (check: Inserting, Indexing, Reading, Counting, Aggregation, Updating, Upserting, Deleting, Reloading, and Synchronization).
Live queries
By leveraging Live Queries, you can perform queries that not only return initial results but also establish a continuous connection to the queried data. This connection ensures that any changes made to the data in the database are automatically reflected in the query results in real-time, without the need for manual refreshing or re-querying. It also means that any changes made to the query resulting object will be persisted to the database.
// Perform a live query
let res1 = await db.live({ firstName: "Ali" });
// same results as above
let res2 = await db.live({ firstName: "Ali" });
// Get regular non-live result
let res3 = await db.find({ firstName: "Ali" });
res1[0].firstName = "Mario";
// Update the firstName property
// The above line updates the database and 'res1'
// it is equivalent to:
// db.update({ firstName: 'Ali' }, { $set: { firstName: 'Mario' } });
// 'res2' will have the updated value automatically
// 'res3' will retain the old value
// since it was obtained using 'find' instead of 'live'
db.insert(Model.new({ firstName: "Ali" }));
// when the operation above adds a new document
// the res1 & res2 will be updated automatically
// and will have the new document
// won't get automatic updates from DB
db.live({ firstName: "Dina" }, { fromDB: false });
// won't set automatic updates to DB
db.live({ firstName: "Dina" }, { toDB: false });
// killing:
// kill live query turning it to regular query
res1.kill();
// won't get automatic updates from DB anymore
res1.kill("fromDB");
// will not reflect changes to DB anymore
res1.kill("toDB");
With live queries, you can build dynamic applications that respond to data changes in real-time. Live queries enable you to use XWebDB directly as a state manager in your front-end framework (react, angular, vue, svelte, solid ...etc). This is discussed extensively in Live queries section.
Configuration
import { Database, Doc } from "xwebdb";
// Model/Schema
class Person extends Doc {
firstName: string = "";
lastName: string = "";
get fullName() {
return this.firstName + " " + this.lastName;
}
}
// Database creation and configuration
let db = new Database<Person>({
ref: "my-database",
// Define a reference to be used as a database name for IndexedDB
model: Person,
// Define model for object mapping
timestampData: true,
// Include "createdAt" and "updatedAt" fields in documents
stripDefaults: true,
// Remove default values from the IndexedDB and remote database
corruptAlertThreshold: 0.2,
// Set tolerance level for data corruption
deferPersistence: 500,
// Resolve promises before persisting operations to IndexedDB
indexes: ["firstName"],
// Define non-unique indexes
cacheLimit: 1000,
// Set cache limit to avoid overwhelming memory
encode: (obj) => JSON.stringify(obj),
// Implement encryption for data persistence
decode: (str) => JSON.parse(str),
// Implement decryption for data retrieval
});
ref
:string
(Required, no default value)
- The provided string will serve as the name for both the database and table in IndexedDB. Ensure uniqueness for each database to avoid data sharing and unexpected behavior.
model
:a class that extends Doc
(Defaults to Doc)
- The model represents the schema and type declaration for your data. It should be a class that extends Doc. The properties of this model define the document's schema, and the values assigned to these properties act as defaults. Using the model ensures consistency and adherence to the schema when creating new documents.
import { Doc } from "xwebdb";
class Person extends Doc {
firstName: string = "default name";
// Default value for 'firstName'
age: number = 25;
// Default value for 'age'
}
// Create a document using .new
Person.new({ firstName: "Ali" });
// The above returns a document
// with the default value for 'age'
// and "Ali" as the value for 'firstName'
- Strong typing for querying and modification comes from the type declarations of this class.
timestampData
:boolean
(Defaults to false)
- When set to true, the database automatically includes "createdAt" and "updatedAt" fields in documents with their respective values as Date objects.
stripDefaults
:boolean
(Defaults to false)
- By default, both the IndexedDB database and the remote database contain all properties of the documents. However, when the property is set to true, default values are stripped during persistence. These default values will be added back through the object mapping mechanism, ensuring the integrity of the data is preserved. It is important to note that if a different model is used that either does not include those default values or includes different ones, the behavior may vary.
corruptAlertThreshold
:number
(Defaults to 0)
- Set a value between 0 and 1 to introduce tolerance for data corruption. A value greater than 0 allows a level of tolerance for corrupted data. The default value of 0 indicates no tolerance for data corruption.
deferPersistence
:false | number
(Defaults to false)
- During document insertion, updating, or deletion, these operations are initially performed on the in-memory copy of the database, subsequently, the changes are reflected in the persisted database, then the promises associated with these operations are resolved. However, if you set this property to a numeric value, the promises will be resolved before the operations are persisted to the IndexedDB database. After a specified number of milliseconds (determined by the value you provided) the operations will be persisted to IndexedDB.
- This approach can offer optimal performance for applications that prioritize speed, since performance bottleneck is actually IndexedDB transactions. But it should be noted that consistency between the in-memory and persisted copies of the database may be compromised due to the time delay. Eventual consistency will occur, unless script execution stopped (like page reload or exit).
indexes
:Array<string>
(Defaults to an empty array)
- This is a way to define the indexes of your database. It's equivalent to calling
db.ensureIndex
However, it offers less options. For example, the indexes created using this approach will not be unique by default. If you require unique indexes, you would need to recreate them usingdb.ensureIndex
and explicitly define them as unique (checkensureIndex
below for more information). - Nevertheless, it can be considered as a shortcut for defining non-unique database indexes.
cacheLimit
:number
(Defaults to 1000)
- To avoid overwhelming user memory with cached data, a cache limit must be set. defaults to 1000 (read more about caching mechanism below).
encode
:(input:string)=>string
(Defaults to undefined)
decode
:(input:string)=>string
(Defaults to undefined)
- Implement the encode and decode methods as reverse functions of each other. By default, documents are persisted as JavaScript objects in the IndexedDB database and sent to the remote database as stringified versions of those objects. Use these methods to implement encryption or other transformations for data persistence and retrieval.
import { Database } from "xwebdb";
function encrypt() {
/* encrpytion code */
}
function decrypt() {
/* decrpytion code */
}
let db = new Database({
ref: "database",
encode: (input: string) => encrpyt(input),
decode: (input: string) => decrypt(input),
});
Object mapping
Object mapping is mechanism by which you define a structure for your data using JavaScript classes.
import { Doc } from "xwebdb";
class Person extends Doc {
firstName: string = "";
lastName: string = "";
birth: number = 20;
// getters
get fullName() {
return this.firstName + " " + this.lastName;
}
get age() {
new Date().getFullYear() - this.birth;
}
// alias
name = fullname;
// helper method
setBirthByAge(age: number) {
this.birth = new Date().getFullYear() - age;
}
}
From the above example you can see the following advantages when defining your model:
- You can set getters in the class and use them when querying.
- You can use aliases for properties.
- You can use helper methods as part of your document.
- You can set default values for properties.
The model class extends Doc, which is mandatory because:
_id
field will be added automatically. XWebDB, by default uses UUID generator that is even faster than the nativecrypto.randomUUID()
.- Properties with default values will be stripped on persistence so your documents will take less size and send less data when syncing. If
stripDefaults
options is set to true on database instantiation.
Having your model as a class allows for more creativity and flexibility, the following example implements a basic level of hierarchy in model definition, since two models share similar type of values:
import { Doc } from "xwebdb";
class Person extends Doc {
// overwrites the default _id generator
_id: string = crypto.randomUUID();
firstName: string = "";
lastName: string = "";
get fullName() {
return this.firstName + " " + this.lastName;
}
}
class Doctor extends Person {
speciality: string = "";
}
class Patient extends Person {
illness: string = "";
}
let doctorsDB = new Database<Doctor>({
model: Doctor,
ref: "doctors",
});
let patientsDB = new Database<Patient>({
model: Patient,
ref: "patients",
});
You can explore more advanced concepts such as OOP, modularity, dependency injection, decorators, mixins, and more.
Sub-documents Mapping
Submodels (Child models/sub-documents) are also supported in object mapping using SubDoc
class and mapSubModel
function.
import { Doc, SubDoc, mapSubModel } from "xwebdb";
/**
* Toy is a sub-document of a sub-document of a document
* Sub document definintion must extend "SubDoc"
*/
class Toy extends SubDoc {
name: string = "";
price: number = 0;
get priceInUSD() {
return this.price * 1.4;
}
}
/**
* Child is a sub-document of a document
* Sub document definintion must extend "SubDoc"
*/
class Child extends SubDoc {
name: string;
age: number = 0;
toys: Toy[] = mapSubModel(Toy, []);
favoriteToy: Toy = mapSubModel(Toy, Toy.new({}));
get numberOfToys() {
return this.toys.length;
}
}
class Parent extends Doc {
name: string = "";
age: number = 9;
male: boolean = false;
mainChild: Child = mapSubModel(Child, Child.new({}));
children: Child[] = mapSubModel(Child, []);
get female() {
return !this.male;
}
}
From the above example you can see that mapSubModel
takes two arguments:
- First one: is model definition of the sub-document.
- Second one: is the default value for this property/field.
Inserting documents
When trying to insert/create a new document use the .new()
method.
db.insert(Parent.new());
// inserts a new "Parent" document.
// fields/properties of the document will all be the default values.
// to define properties other than the defaults
// you can pass them as a plain JS object.
db.insert(
Parent.new({
name: "Ali",
age: 31,
male: true,
mainChild: Child.new({
name: "Kiko",
}),
// properties that are not
// mentioned in this object
// will be the defaults defined
// in the class above
})
);
// Note that the .new() method
// doesn't actually insert a new document.
// it merely returns a document in preparation for insertion.
How would it look when persisiting?
When persisting data, only the actual fields (neither getters nor methods) will be persisted. Using the stripDefaults option on database instantiation will also remove the default values from the persisted data (StripDefaults).
Best practices
- Define getters instead of functions and methods. This enables you to query documents using the getter value, use them as indexes, and simplifies your queries.
class Child extends Doc {
age: number = 9;
oldToys: Toy[] = mapSubModel(Toy, []);
newToys: Toy[] = mapSubModel(Toy, []);
get numberOfToys() {
return this.oldToys.length + this.newToys.length;
}
get toyEachYear() {
return this.numberOfToys / this.age > 1;
}
}
let childrenDB = new Database<Child>({ ref: "children", model: Child });
// simple query
childrenDB.find({ toyEachYear: { $gt: 2 } });
// if you wouldn't use the computed property
// your query will be very complex
// having to use many operators
// like: $or, $size, $gt and maybe even more.
- Always use the static Model.new to prepare new documents before insertion.
// all fields have default values
db.insert(Parent.new());
// all fields have default values except 'age'
db.insert(Parent.new({ age: 30 }));
- Define createdAt and updatedAt in your model when you're using them in you database.
- Never try to directly set a computed property or update it via the update operators.
- Use Model.new in conjugation with the upsert operator
$setOnInsert
(more on upserting in the examples below). - Always define defaults for your fields in the model.
Inserting
To insert documents in the database use the insert
method (or alias: create
), which can either take single document or an array of documents. Remember to always use the Model.new()
when inserting document.
import { Database, Doc } from "xwebdb";
class Person extends Doc {
name: string = "";
age: number = 0;
}
let db = new Database<Person>({ ref: "database", model: Person });
// insert an empty document (with default values)
db.insert(Person.new({}));
// insert a document with fields
db.insert(
Person.new({
name: "ali",
age: 12,
})
);
db.insert(
Person.new({
name: "ali",
// age field will have the default value
})
);
// inserting multiple documents at once
db.insert([Person.new({ name: "ali" }), Person.new({ name: "dina" })]);
The insert
method will return a promise that resolves to the number of inserted documents and an array of the inserted documents. The last line of the above example will return a promise that resolves to the following:
{
"number": 2,
"docs": [
{
"_id": "ad9436a8-ef8f-4f4c-b051-aa7c7d26a20e",
"name": "ali",
"age": 0
},
{
"_id": "38ae1bbd-60a7-4980-bbe9-fce3ffaec51c",
"name": "dina",
"age": 0
}
]
}
Reading
Reading from the database using read
(alias: find
):
db.read({ name: "ali" });
db.find({ name: "ali" }); // alias
The read
(or find
) method takes two arguments:
- The first one is the read query (resembles MongoDB).
- The second one is an object that you can use to
skip
,limit
,sort
, andproject
the matched documents.
Here's a more elaborate example:
db.read(
{ name: "ali" }, // query
{
skip: 2, // skip the first two matches
limit: 10, // limit matches to 10 results
project: {
// pick-type projection
// result will only have
// the following props
name: 1,
age: 1,
// omit-type projection
// result will not have
// the following props
address: 0,
email: 0,
// NOTE: you can either use
// pick-type or omit-type projection
// except for _id which is by default
// always returned and which you can choose to omit
},
sort: {
name: 1, // ascending sort by name
age: -1, // descending sort by age
// single or multiple sort elements
// can be specified at the same time
},
}
);
The query API (first argument of read
) closely resembles MongoDB MQL. You can query documents based on field equality or utilize a range of comparison operators such as $lt
, $lte
, $gt
, $gte
, $in
, $nin
, $ne
, and $eq
. Additionally, logical operators like $or
, $and
, $not
, and $where
are available for more complex querying capabilities.
- Field equality, e.g.
{name:"Ali"}
- Field level operators (comparison at field level), e.g.
{age:{$gt:10}}
- Top level operators (logical at top level), e.g.
{$and:[{age:10},{name:"Ali"}]
.
1. Field level Equality
To specify equality conditions in a query filter document, you can use { <FieldName> : <Value> }
expressions. This allows you to find documents that match specific field values. Here are some examples:
// Select all documents where the name is "ali"
db.find({ name: "ali" });
// Select all documents where the age is exactly 27
// and the height is exactly 180
db.find({
age: 27,
height: 180,
});
In these examples, the filter field is used to specify the equality conditions for the query. You can provide multiple field-value pairs to further refine the query.
However, like MongoDB, when dealing with deeply nested objects, simple field-level equality may not work as expected. Consider the following example:
// Suppose you have the following document:
{
item: "Box",
dimensions: {
height: 30,
width: 20,
weight: 100
}
}
// The following queries won't match:
db.find({
dimensions: { height: 30 }
});
db.find({
dimensions: { height: 30, width: 20 }
});
// The following query will match:
db.find({
dimensions: { height: 30, width: 20, weight: 100 }
});
In the case of deeply nested objects, using field-level equality alone will not work. To query deeply nested documents, you need to use the $deep
operator. The $deep
operator allows you to specify nested fields and their values in a query. More information about the $deep
operator can be found below.
2. Field-level operators
Syntax: { <fieldName>: { <operator>: <specification> } }
2.1. Comparison operators
$eq
Specifies equality condition. The $eq operator matches documents where the value of a field equals the specified value. It is equivalent to { <FieldName> : <Value> }
.
Specification
- Applies to:
Any field type
- Syntax:
{ <fieldName> : { $eq: <value> } }
Example
// Example
db.find({ name: { $eq: "ali" } });
// same as:
db.find({ name: "ali" });
$ne
$ne selects the documents where the value of the field is not equal to the specified value. This includes documents that do not contain the field.
Specification
- Applies to:
Any field type
- Syntax:
{ <fieldName> : { $ne: <value> } }
Example
// selecting all documents where "name"
// does not equal "ali"
db.find({ name: { $ne: "ali" } });
$gt
selects those documents where the value of the field is greater than (i.e. >
) the specified value.
Specification
- Applies to:
number
&Date
fields - Syntax:
{ <fieldName> : { $gt: <value> } }
Example
// applied on a number field
db.find({ year: { $gt: 9 } });
// applied on a date field
db.find({
createdAt: { $gt: new Date(1588134729462) },
});
$lt
selects those documents where the value of the field is less than (i.e. <
) the specified value.
Specification
- Applies to:
number
&Date
fields - Syntax:
{ <fieldName> : { $lt: <value> } }
Example
// applied on a number field
db.find({ year: { $lt: 9 } });
// applied on a date field
db.find({
createdAt: { $lt: new Date(1588134729462) },
});
$gte
selects those documents where the value of the field is greater than or equal to (i.e. >=
) the specified value.
Specification
- Applies to:
number
&Date
fields - Syntax:
{ <fieldName> : { $gte: <value> } }
Example
// applied on a number field
db.find({ year: { $gte: 9 } });
// applied on a date field
db.find({
createdAt: { $gte: new Date(1588134729462) },
});
$lte
selects those documents where the value of the field is less than or equal to (i.e. <=
) the specified value.
Specification
- Applies to:
number
&Date
fields - Syntax:
{ <fieldName> : { $lte: <value> } }
Example
// applied on a number field
db.find({ year: { $lte: 9 } });
// applied on a date field
db.find({
createdAt: { $lte: new Date(1588134729462) },
});
$in
The $in
operator selects the documents where the value of a field equals any value in the specified array.
Specification
- Applies to:
Any field type
- Syntax:
{ <fieldName> : { $in: [<value1>, <value2>, ... etc] } }
Example
// find documents where the "name"
// field is one of the specified
// in the array
db.find({ name: { $in: ["ali", "john", "dina"] } });
$nin
The $nin
operator (opposite of $in
) selects the documents where the value of a field doesn't equals any value in the specified array.
Specification
- Applies to:
Any field type
- Syntax:
{ <fieldName> : { $nin: [<value1>, <value2>, ... etc] } }
Example
// find documents where the "name"
// field is one of the specified
// in the array
db.find({ name: { $nin: ["ali", "john", "dina"] } });
2.2 Element operators
$exists
When <boolean>
is passed and is true
, $exists
matches the documents that contain the field, including documents where the field value is null. If <boolean>
is false
, the query returns only the documents that do not contain the field.
Specification
- Applies to:
Any field type
- Syntax:
{ <fieldName> : { $exists: <boolean> } }
Example
// select documents where the "name"
// field is defined, even if it is null
db.find({ name: { $exists: true } });
// select documents where the "name"
// field is not defined
db.find({ name: { $exists: false } });
$type
$type
selects documents where the value of the field is an instance of the specified type. Type specification can be one of the following:
"string"
"number"
"boolean"
"undefined"
"array"
"null"
"date"
"object"
Specification
- Applies to:
Any field type
- Syntax:
{ <fieldName> : { $type: <spec> } }
Example
// find documents where the "name" field
// is a string.
db.find({ name: { $type: "string" } });
2.3 Evaluation operators
$mod
Select documents where the value of a field divided by a divisor has the specified remainder (i.e. perform a modulo operation to select documents).
Specification
- Applies to:
number
&Date
fields - Syntax:
{ <fieldName> : { $mod: [divisor, remainder] } }
Example
// select documents where the "years" field
// is an even number
db.find({
years: {
$mod: [2, 0],
},
});
// select documents where the "years" field
// is an odd number
db.find({
years: {
$mod: [2, 1],
},
});
$regex
Selects documents which tests true for a given regular expression.
Specification
- Applies to:
string
fields - Syntax:
{ <fieldName> : { $regex: <RegExp> } }
Example
// select documents where the "name"
// field starts with either "a" or "A".
db.find({ name: { $regex: /^a/i } });
2.4 Array operators
$all
The $all
operator selects the documents where the value of a field is an array that contains all the specified elements.
Specification
- Applies to:
array
fields - Syntax:
{ <fieldName> : { $all: [<value1>, <value2>,...etc] } }
Example
// select documents where the "tags" field
// is an array that has "music" & "art"
db.find({ tags: { $all: ["music", "art"] } });
$elemMatch
The $elemMatch
operator matches documents that contain an array field with at least one element that matches all the specified query criteria.
Specification
- Applies to:
array
fields - Syntax:
{{<fieldName>:{$elemMatch:{<query1>,<query2>,...etc}}}
Example
// select documents where the "price" field
// is an array field that has an element
// matching the following criteria
// has an even number
// less than 8
// and greater than 0
db.find({
price: {
$elemMatch: {
$mod: [2, 0],
$lt: 8,
$gt: 0,
},
},
});
$size
The $size
operator matches any array with the number of elements (length of the array) specified by the argument.
Specification
- Applies to:
array
fields - Syntax:
{ <fieldName> : { $size: number } }
Example
// select documents where the "tags"
// field is an array that has 10 elements.
db.find({ tags: { $size: 10 } });
Other operators behavior on arrays
The array fields has the operators $all
, $elemMatch
and $size
specific for them. However, all the operators mentioned earlier can also be applied to arrays, and they will return true if any element in the array matches the specified condition.
Here is a summary of how the operators work when applied to arrays:
$eq
: Matches an array if it contains an element equal to the value specified by the operator.$ne
: Matches an array if it contains an element different from the value specified by the operator.$gt
: Matches an array if it contains a number greater than the value specified by the operator.$lt
: Matches an array if it contains a number less than the value specified by the operator.$gte
: Matches an array if it contains a number greater than or equal to the value specified by the operator.$lte
: Matches an array if it contains a number less than or equal to the value specified by the operator.$in
: Matches an array if it contains any of the values specified by the operator.$nin
: Matches an array if it contains none of the values specified by the operator.$mod
: Matches an array if it contains a number that, when divided by the divisor specified by the operator, yields the remainder specified by the operator.$regex
: Matches an array if it contains a string that matches the regular expression specified by the operator.$exists
: Matches any given array.$type
: Matches an array if the array itself is of the type "array" as specified by the operator.
These operators provide flexibility for querying and filtering arrays based on various conditions.
2.5 Negation operator
All the above operators can be negated using the $not
operator.
// find all documents
// where they have "tags" that is not of length 10
db.find({ tags: { $not: { $size: 10 } } });
// similar to $ne
db.find({ name: { $not: { $eq: "ali" } } });
// find documents where the "name" field
// is a not a string
db.find({ name: { $not: { $type: "string" } } });
// select documents where the "tags"
// field is an array that doesn't have "music" & "art"
db.find({ tags: { $not: { $all: ["music", "art"] } } });
// select documents where the "years" field
// is an even number
db.find({
years: {
$not: {
$mod: [2, 1],
},
},
});
// ...etc
3. Top-level operators
$and
$and
performs a logical AND operation on an array of two or more expressions (e.g. <field level query 1>
, <field level query 2>
, etc.) and selects the documents that satisfy all the expressions in the array. The $and
operator uses short-circuit evaluation. If the first expression (e.g. <field level query 1>
) evaluates to false, XWebDB will not evaluate the remaining expressions.
Specification
Syntax
{
$and: [
<query1>,
<query2>,
<query3>,
...etc
]
}
Example
/**
* Select a document where the name
* isn't equal to "ali" and the property exists
*/
db.find({
$and: [{ $name: { $ne: "ali" } }, { $name: { $exists: true } }],
});
$nor
$nor
performs a logical NOR operation on an array of one or more query expression and selects the documents that fail all the query expressions in the array.
Specification
Syntax
{
$nor: [
<query1>,
<query2>,
<query3>,
...etc
]
}
Example
/**
* Select a document where the "name" is not "alex"
* and the age is not 13
*/
db.find({
$nor: [{ $name: "alex" }, { $age: 13 }],
});
$or
The $or
operator performs a logical OR operation on an array of two or more expressions and selects the documents that satisfy at least one of the expressions.
Specification
Syntax
{
$or: [
<query1>,
<query2>,
<query3>,
...etc
]
}
Example
/**
* Select a document where the "name" is not "ali"
* or the age is not 13
*/
db.find({
$or: [{ name: "ali" }, { $age: 13 }],
});
$where
Matches the documents that when evaluated by the given function would return true.
!WARNING The
$where
provides greatest flexibility, but requires that the database processes the JavaScript expression or function for each document in the collection. It's highly advisable to avoid using the$where
operator and instead use indexed getters as explained in the object mapping section of this documentation.
Specification
Syntax
{
$where: (this: Model) => boolean;
}
Example
/**
* Select a document where the "name"
* is 5 characters long and ends with "x"
*/
db.find({
$where: function () {
// DO NOT use arrow function here
return this.name.length === 5 && this.name.endsWith("x");
},
});
$deep
The $deep
operator is the only operator in XWebDB that doesn't exist in MongoDB. It has been introduced as an alternative to the dot notation to match deep fields in sub-documents.
Take the following document for example:
{
item: "box",
dimensions: {
height: 100,
width: 50
}
}
The following queries will behave as follows:
db.find({ dimensions: { height: 100 } });
// will not match, because field-level literal equality
// requires the query object to exactly equal the document object
db.find({ $deep: { dimensions: { height: 100 } } });
// the above query will match the document
The reason that the $deep
operator has been added is to keep strict typing even when querying deeply nested objects. Since it is not possible (in typescript) to define strict typings for the dot notation.
Specification
Syntax
{
$deep: <query>
}
Example
Basic example:
// document
{
item: "box",
dimensions: {
height: 100,
width: 50
}
}
db.find({ $deep: { dimensions: { height: 100 } } });
You can specify multiple deep fields:
//documents
{
name: {
first: "ali",
last: "saleem",
}
}
{
name: {
first: "john",
last: "cena",
}
}
db.find({
$deep: {
name: {
first: { $in: ["ali", "john"] },
last: { $in: ["saleem", "cena"] },
},
}
});
You can use the $deep
operator even in array elements by defining their index:
// document:
{
name: "ali",
children: [
{
name: "keko",
age: 2,
},
{
name: "kika",
age: 1,
}
]
}
db.find({
$deep: {
children: {
0: {
age: { $gt : 1 }
}
}
}
})
Counting
To count the number of documents matching a specific query use the count
method.
Count method always returns a promise that resolves to a number.
// count number of documents
// that has "ali" in the "name" field
db.count({ name: "ali" });
// count number of documents
// that has age greater than 20
db.count({ age: { $gt: 20 } });
The query API in the count
method is the same as the query API in read
method, which in turn very similar to MongoDB query language.
Aggregation
Aggregation is process of combining multiple rows of data into a single value or summary. It involves performing mathematical or statistical operations on a set of data to generate meaningful insights or summaries. Aggregation is commonly used to calculate various metrics, such as sums, averages, counts, maximums, minimums, or other statistical calculations, over a group of rows.
Aggregation In XWebDB uses the method chaining syntax, and the following aggregation methods are supported: $sort
, $limit
, $skip
, $project
, $match
, $addFields
, $group
, $unwind
let aggregate = await db.aggregate(
//optional starting query
{
name: "ali",
}
);
aggregate
.$sort({
age: -1, // sorting descending
city: 1, // sorting ascending
})
.$skip(1)
.$limit(50)
.$project({ children: 1 })
.$addFields((doc) => ({
// adds fields to each document
// based on calculation done on it
numberOfChildren: doc.children.length,
}))
// filter the aggregate based on a specific query
// this can take the same syntax of `db.find`
.$match({ numberOfChildren: { $gt: 1 } })
// deconstruct an array field within a document
// creating a new document for each element in the array
.$unwind("children")
// adds a field
.$addFields((doc) => ({ numberOfBrothers: doc.numberOfChildren - 1 }))
// group documents together based on a specified key
// and perform various transformations on the grouped data
// using a reducer function
.$group({
_id: "children",
reducer: (group) => ({
kidsNamed: group[0].children
count: group.length
}),
});
You can use the aggregation methods in any order and as much as you want, until you get the target meaningful data.
The current implementation of aggregation in XWebDB is a starting point, especially the $group
method.
Planned updates on next versions:
- Live aggregations
- $lookup method (for joining two databases)
- $group operators (such as
$sum
,$avg
,$max
, and$min
)
Updating
To update documents matching a specific query use the update
method.
db.update(
// first argument is the find query
// matching target documents
// to be updated
{
name: "ali",
},
// second argument is the update
// must be supplied with one of the update operators
{
$set: {
age: 32,
},
},
// third argument is a boolean
// whether to update multiple documents
// or to update the first match only
// defaults to "false"
// i.e. update first match only
false
);
The first argument of the update method takes a query with the same syntax as the find
& count
methods as explained above. The second argument must be supplied with the update operators.
!INFO Although it is possible in MongoDB to use the direct field updates (no operator,
{age:5}
), this is not supported in XWebDB to enforce a more strict type declaration.
The insert
method will return a promise that resolves to the number of updated documents, a boolean of whether the update was an upsert or not, and an array of the updated documents. the following is an example:
{
"number": 2,
"upsert": false,
"docs": [
{
"_id": "ad9436a8-ef8f-4f4c-b051-aa7c7d26a20e",
"name": "ali",
"age": 0
},
{
"_id": "38ae1bbd-60a7-4980-bbe9-fce3ffaec51c",
"name": "dina",
"age": 0
}
]
}
The update operators are:
- Field operators
- Mathematical operators
- Array operators
1. Field update Operators
$set
Sets the value of a field in a document to a specified value.
Specification
- Applies to:
any field type
- Syntax
{
$set: {
<fieldName1>: <value1>,
<fieldName2>: <value2>,
...etc
}
}
Example
/**
* update the name "ali" to "dina"
*/
db.update(
{
name: "ali",
},
{
$set: {
name: "dina",
},
}
);
$unset
Sets the value of a field in a document to undefined
.
Specification
- Applies to:
any field type
- Syntax
{
$unset: {
<fieldName1>: "",
<fieldName2>: "",
...etc
}
}
Example
/**
* setting the name "ali" to undefined
*/
db.update(
{
name: "ali",
},
{
$unset: {
name: "",
},
}
);
$currentDate
Sets the value of a field in a document to a Date
object or a timestamp number
representing the current date.
Specification
- Applies to:
Date
&number
fields - Syntax
{
$currentDate: {
// date object (applies to Date fields)
<fieldName1>: true,
// date object (alternative syntax)
<fieldName2>: { $type: "date" },
// timestamp (applies to number fields)
<fieldName3>: { $type: "timestamp" },
...etc
}
}
Example
/**
* setting the name "ali" to undefined
*/
db.update(
{
name: "ali",
},
{
$currentDate: {
lastLogin: true,
lastLoginTimestamp: {
$type: "timestamp",
},
},
}
);
$rename
Renames a property name to a different one while keeping the value the same.
Specification
- Applies to:
any field type
- Syntax
{
$rename: {
<fieldName1>: <newName1>,
<fieldName2>: <newName2>,
...etc
}
}
Example
/**
* setting the property name "name" to "firstName"
*/
db.update(
{
name: "ali",
},
{
$rename: {
name: "firstName",
},
}
);
2. Mathematical update Operators
$inc
increments the value of a field in a document to by a specific value.
Specification
- Applies to:
number
fields - Syntax
{
$inc: {
<fieldName1>: <number>,
<fieldName2>: <number>,
...etc
}
}
Example
/**
* increment the age field by 2, and the months by 24
*/
db.update(
{
name: "ali",
},
{
$inc: {
age: 2,
months: 24,
},
}
);
You can also pass a negative value to decrement
/**
* decrement the age field by 2, and the months by 24
*/
db.update(
{
name: "ali",
},
{
$inc: {
age: -2,
months: -24,
},
}
);
$mul
multiplies the value of a field in a document to by a specific value.
Specification
- Applies to:
number
fields - Syntax
{
$mul: {
<fieldName1>: <number>,
<fieldName2>: <number>,
...etc
}
}
Example
/**
* multiplies the age field by 2, and the months by 24
*/
db.update(
{
name: "ali",
},
{
$mul: {
age: 2,
months: 24,
},
}
);
You can also use the $mul
operator to do a division mathematical operation:
/**
* divides the age field by 2, and the months by 24
*/
db.update(
{
name: "ali",
},
{
$mul: {
age: 1 / 2,
months: 1 / 24,
},
}
);
$max
Updates the field value to a new value only if the specified value is greater than the existing value.
Specification
- Applies to:
number
&Date
fields - Syntax
{
$max: {
<fieldName1>: <number|Date>,
<fieldName2>: <number|Date>,
...etc
}
}
Example
/**
* sets the age field to 10 if it's less than 10
*/
db.update(
{
name: "ali",
},
{
$max: {
age: 10,
},
}
);
$min
Updates the field value to a new value only if the specified value is less than the existing value.
Specification
- Applies to:
number
&Date
fields - Syntax
{
$min: {
<fieldName1>: <number|Date>,
<fieldName2>: <number|Date>,
...etc
}
}
Example
/**
* sets the age field to 10 if it's greater than 10
*/
db.update(
{
name: "ali",
},
{
$min: {
age: 10,
},
}
);
3. Array update Operators
$addToSet
Adds elements to an array only if they do not already exist in it.
Specification
- Applies to:
Array
fields - Syntax
{
$addToSet: {
// adds a single value
<fieldName1>: <value>,
// adds multiple values
<fieldName2>: {
$each: [
<value1>,
<value2>,
<value3>,
... etc
]
}
}
}
Example
/**
* Update a document where "name" is "ali"
* by adding the skill "javascript"
* and adding two projects
*/
db.update(
{ name: "ali" },
{
$addToSet: {
skills: "javascript",
projects: {
$each: ["projectA.js", "projectB.js"],
},
},
}
);
$pop
removes the first or last element of an array. Pass $pop a value of -1 to remove the first element of an array and 1 to remove the last element in an array.
Specification
- Applies to:
Array
fields - Syntax
{
$pop: {
// removes first element
<fieldName1>: -1,
// removes last element
<fieldName2>: 1
}
}
Example
/**
* Update a document where name is "ali"
* removing last element of "skills"
* and removes first element of "projects"
*/
db.update(
{ name: "ali" },
{
$pop: {
skills: 1,
projects: -1,
},
}
);
$pull
Removes all array elements that match a specified query or value.
Specification
- Applies to:
Array
fields - Syntax
{
$pull: {
// remove by value
<fieldName1>: <value>,
// or remove by query
<fieldName2>: <query>
}
}
Example
/**
* Update a document where name is "ali"
* removing skill "javascript"
* and removing any project that ends with .js
*/
db.update(
{ name: "ali" },
{
$pop: {
projects: {
$regex: /\.js$/i,
},
skills: "javascript",
},
}
);
$pullAll
The $pullAll operator removes all instances of the specified values from an existing array.
Unlike the $pull operator that removes elements by specifying a query, $pullAll removes elements that match the listed values.
Specification
- Applies to:
Array
fields - Syntax
{
$pullAll: {
<fieldName1>: [<value1>, <value2> ... etc],
<fieldName2>: [<value1>, <value2> ... etc],
... etc
}
}
Example
/**
* Update a document where name is "ali"
* removing skills: "javascript" and "php"
* and removes "new", "x-client"
*/
db.update(
{ name: "ali" },
{
$pullAll: {
skills: ["php", "javascript"],
},
}
);
$push
The $push operator appends a specified value to an array.
Specification
- Applies to:
Array
fields - Syntax
{
$push: {
<fieldName1>: <value>,
// or
<fieldName2>: {
// multiple fields
$each: [<value1>, <value2>, ...etc]
// with modifiers
// discussed below
$slice: <number>,
$position: <number>,
$sort: <sort-specification>
},
... etc
}
}
Example
/**
* Update a document where name is "ali"
* removing skills: "javascript" and "php"
* and removes "new", "x-client"
*/
db.update(
{ name: "ali" },
{
$push: {
skills: "javascript",
projects: {
$each: ["projectA.js", "projectB.js"],
},
},
}
);
$push
with $each
modifier
Modifies the $push
operators to append multiple items for array updates. It can also be used with $addToSet
operator.
db.update(
{ name: "ali" },
{
$push: {
projects: {
$each: ["projectA.js", "projectB.js"],
},
},
}
);
$push
with $position
modifier
$position modifier must be supplied with a number. It specifies the location in the array at which the $push operator insert elements. Without the $position
modifier, the$push
operator inserts elements to the end of the array.
Must be used with $each
modifier.
db.update(
{ name: "ali" },
{
$push: {
projects: {
$each: ["projectA.js", "projectB.js"],
$position: 1,
// the array items above will be pushed
// starting from position 1
},
},
}
);
If the $position
modifier is supplied with a number larger than the length of the array the items will be pushed at the end without leaving empty or null slots.
$push
with $slice
modifier
Modifies the $push
operator to limit the size of updated arrays.
Must be used with $each
modifier. Otherwise it will throw. You can pass an empty array ([ ]
) to the $each
modifier such that only the $slice
modifier has an effect.
/**
* Update a document where name is "ali"
* by appending "a" & "b" to "tags"
* unless the size of the array goes
* over 6 elements
*/
db.update(
{ name: "ali" },
{
$push: {
tags: {
$each: ["a", "b"],
$slice: 6,
},
},
}
);
$push
with $sort
modifier
he $sort
modifier orders the elements of an array during a $push
operation. Pass 1
to sort ascending and -1
to sort descending.
Must be used with $each
modifier. Otherwise it will throw. You can pass an empty array ([]
) to the $each
modifier such that only the $sort
modifier has an effect.
Here's an elaborate example explaining all the modifiers that can be used with $push
operator:
/**
* Update a document where name is "john"
* by appending a new child to "children"
* at the start of the array
* then sorting the children by
* the "age" / ascending
* and the "name" / descending
* then allow only 10 children in the array
* by removing the last elements that goes
* over 10
*/
db.update(
{ name: "john" },
{
$push: {
children: {
$each: [{ name: "tim", age: 3 }],
$position: 0,
$sort: {
age: 1,
name: -1,
},
$slice: 10,
},
},
}
);
!INFO You can use the
$slice
and$sort
modifiers with empty$each
array, so they have an effect without actually pushing items to the array, i.e. only sorting (for$sort
modifier) or only slicing (for$slice
modifier).
Upserting
Upserting is a combination of updating & inserting. If the document you want to update already exists, it will be updated with new information. But if the document doesn't exist, a completely new document will be created with the specified information.
To do upsert operations use the upsert
method. The upsert
method has the same API as the update
method but requires the $setOnInsert
operator.
In short, The upsert
method behaves exactly the same as the update
method, with one exception: if no target document matched the query in the first argument, a new document will be created and the contents of the new document will be based on $setOnInsert
operator.
import { Database, Doc } from "xwebdb";
class Person extends Doc {
name: string = "";
age: number = 0;
}
let db = new Database<Person>({
ref: "database",
model: Person,
});
db.upsert(
// for documents matching the following query:
{
name: "ali",
},
{
// update the name to "dina"
$set: {
name: "dina",
},
// if no documents matches
// create a new document with the name "dina"
$setOnInsert: Person.new({
name: "dina",
}),
}
);
The upsert
method will return a promise that resolves to the number of updated (or inserted) documents, a boolean of whether the update was an upsert or not, and an array of the updated (or inserted) documents. the following is an example:
{
"number": 1,
"upsert": true,
"docs": [
{
"_id": "ad9436a8-ef8f-4f4c-b051-aa7c7d26a20e",
"name": "ali",
"age": 0
}
]
}
Deleting
To delete documents matching a specific query use the delete
(alias: remove
) method.
The delete method takes two arguments, the first one is the query that the target documents to be deleted needs to match, and the second one is a boolean of whether to delete multiple documents or the first one matching the query.
// delete a document with "name" field that equals "ali"
db.delete(
// query
{
name: {
$eq: "ali",
},
},
// multi-delete:
// true: delete all documents matching
// false: delete the first match only
false // default: false
);
// delete all documents with "age" field greater than 45
db.delete(
// query
{
age: {
$gte: 45,
},
},
// multi-delete:
// true: delete all documents matching
// false: delete the first match only
true // default: false
);
Indexing
Database indexing is a powerful technique that optimizes the performance of database queries by creating a separate data structure known as an index. When an index is created for a specific field, it acts as a reference point for the database engine, enabling fast retrieval of data when querying that field. By eliminating the need for full table scans, indexing significantly improves search and retrieval operations, especially when dealing with large amounts of data. However, it's important to note that indexing does introduce additional overhead during write operations. Therefore, thoughtful consideration should be given to the selection and design of indexes to strike a balance between read and write performance.
In short, creating an index for a specific field is beneficial when you expect that field to be frequently used in queries. XWebDB provides complete support for indexes on any field in the database. By default, all databases have an index on the _id
field, and you may add additional indexes to support important queries and operations.
To create an index use the method createIndex
, It takes an object as an argument with the following properties:
fieldName
(required): Specifies the name of the field that you want to index. You can use dot notation to index a field within a nested document.unique
(optional, defaults tofalse
): This option enforces field uniqueness. When a unique index is created, attempting to index two documents with the same field value will result in an error.sparse
(optional, defaults tofalse
): When set totrue
, this option ensures that documents without a defined value for the indexed field are not indexed. This can be useful in scenarios where you want to allow multiple documents without the field, while still enforcing uniqueness.
db.createIndex({
// field name to be indexed
fieldName: "email",
// enforce uniqueness for this field
unique: true,
// don't index documents with 'undefined' value for this field
// when this option is set to "true"
// even if the index unique, multiple documents that has
// the field value as "undefined" can be inserted in the
// database and not cause a rejection.
sparse: true,
});
db.createIndex({
// index on a field that is in a sub-document
fieldName: "address.city",
unique: false,
sparse: true,
});
To remove an index, use the removeIndex
method
db.removeIndex("email");
The createIndex
and removeIndex
methods return a promise that resolves to an object specifying the affected index:
{
"affectedIndex": "email"
}
To create indexes on database initialization use the indexes
configuration parameter:
import { Database, Doc } from "xwebdb";
class Person extends Doc {
name: string = "";
email: string = "";
age: number = 0;
}
let db = new Database<Person>({
ref: "myDB",
model: Person,
indexes: ["email"],
});
However, the indexes created on the database initialization are non-unique, to make them unique you have to recreate them using the createIndex
method.
!NOTE
createIndex
can be called when you want, even after some data was inserted, though it's best to call it at application startup.
Loading and reloading
When you initialize a database using new Database, the existing documents stored in the persistence layer (i.e. IndexedDB) will be read asynchronously. The database object will have loaded
property, which represents a promise. This promise will be resolved once all the previous documents have finished loading. In other words, you can use the load property to determine when the database has finished loading the existing data and is ready for use.
import { Database, Doc } from "xwebdb";
class Person extends Doc {
name: string = "";
email: string = "";
age: number = 0;
}
let db = new Database<Person>({
ref: "myDB",
model: Person,
indexes: ["email"],
});
db.loaded.then(() => {
// do whatever
});
However, all database operations (such as inserting, reading, updating, and deleting) will wait for database initial loading if it hasn't occurred. So you can safely perform such operations without checking the loaded
property.
If for any reason you want to reload the database from IndexedDB you can use the reload
method:
db.reload();
Reasons that you may want to reload a database:
- Multiple browser tabs on the same database, doing different operations.
- Multiple database instances on the same indexedDB reference.
Synchronizing
XWebDB synchronization empowers seamless data replication and consistency across multiple instances, following a multi-master model. Synchronization can be backed by various remote cloud databases like Cloudflare KV, S3 cloud file system, CosmosSB, DynamoDB, and more using synchronization adapters. Additionally you can write your own synchronization adapter to add support any remote Cloud database.
Synchronization protocol in XWebDB utilizes a unique revision ID methodology, ensuring a reliable and conflict-aware data replication. Each instance generates a revision ID for every data modification, enabling effective comparison during synchronization.
Conflict resolution in XWebDB follows the "latest-win" principle, prioritizing the most recent modification to automatically resolve conflicts. While this is an over-simplification of the real-world conflicts, its sufficient for the expected use-cases, avoiding needless complexity and manual intervention.
Synchronizing data
To synchronize data, the database must be instantiated with a sync adapter:
import { Database } from "xwebdb";
import { kvAdapter } from "xwebdb-kvadapter";
const db = new Database({
sync: {
// define remote sync adapter
syncToRemote: kvAdapter("YOUR_ENDPOINT", "YOUR_TOKEN"),
// define an interval at which the database will
// automatically sync with the remote database
// defaults to "0" (will not sync on interval) only manually
syncInterval: 500,
},
/// rest of database configuration
/// ref: ...
/// model: ...
/// ...etc: ...
});
When a database is configured for synchronization, you can use the sync
method to manually synchronize the database with remote database. The method will wait for any database operation, or sync operation to complete before starting the sync progress.
db.sync();
However, if you want a more forceful synchronization (i.e. doesn't wait for other operation or checkpoints matching) you can use the method forceSync
.
db.forceSync();
Both methods will return a promise that resolves to an object indicating the number of documents sent, number of documents received, and whether the sync found a difference or not.