Speedgoose NPM

About The Project

NPM download/month

This project is a next-level mongoose caching library that is fully written in typescript. It's caching on two levels. Shared - with Redis. And local inside memory. Supports all mongoose operations like find,findOne, count, aggregate... and others. Also supports lean queries. Why it is different?

It supports caching not only JSON objects in Redis but also the whole Mongoose. Document instances in local memory to speed up code, and prevent unnecessary hydrations. Also supports full in-memory caching, without Redis.
It has an auto-clearing ability based on mongoose events. So if the query was cached for some records, and in the meantime, those records change, all cached-related results will be cleared.
It supports deep hydration, for caching not only the root document instances but also those that are populated.
Supports custom eventing. For example, if you want to remove given results from cache, but removal logic is not based on removing documents from DB but rather field based (like deleted: true), then you can apply a wasRecordDeleted callback as an option for the plugin.
Supports multitenancy by clearing cached results only for a related tenant.

Release Note:

For now the latests version is on top of mongoose 7.4.1 and is tagged as beta. If you're facing any issues please use version 1.2.24.

Getting Started

This is an example of how you may give instructions on setting up your project locally. To get a local copy up and running follow these simple example steps.

Installation

$ npm install speedgoose
# or
$ yarn add speedgoose

Simple wrap your mongoose with the library (required)

import { applySpeedGooseCacheLayer } from 'speedgoose';
import mongoose from 'mongoose';

applySpeedGooseCacheLayer(mongoose, {
    redisUri: process.env.REDIS_URI,
});

To enable auto-clearing for a given schema, just add the plugin to it (required)

import { SpeedGooseCacheAutoCleaner } from 'speedgoose';

Schema.plugin(SpeedGooseCacheAutoCleaner);
//additionally you can pass options for example callback for setting the record as deleted
Schema.plugin(SpeedGooseCacheAutoCleaner, { wasRecordDeletedCallback });

Usage

With find, count, etc...

// with findOne
const result  = await model<SomeModelType>.findOne({}).cacheQuery()

// with count
const result  = await model<SomeModelType>.count({age: {$gt : 25}}).cacheQuery()

// with sorting query
const result  = await model<SomeModelType>.find({}).sort({fieldA : 1}).cacheQuery()

// with lean query
const result  = await model<SomeModelType>.find({}).lean().cacheQuery()

With aggregation

const result = await model.aggregate<AggregationResultType>([]).cachePipeline()

Checking if key was set under the key.

const isQueryCached = await model<SomeModelType>.find({}).sort({fieldA : 1}).isCached()
const isPipelineCached = await model.aggregate<AggregationResultType>([]).isCached()

:briefcase: Multitenancy

For enabling multitenancy, you have to pass multitenantKey into wrapper config, so it will look like

applySpeedGooseCacheLayer(mongoose, {
    redisUri: process.env.REDIS_URI,
    multitenancyConfig: {
        multitenantKey: 'tenantId',
    },
});

SpeedGooseCacheAutoCleaner plugin clears the cache for a given model each time when new record appears, or some record was deleted. In multitenancy, we won't clear the cache for all of the clients - as the change appears only for one tenant. SpeedGoose will handle it for You! But to make it work, you have to follow the rules:

Tenant key must be in the root of mongo documents
You have to somehow pass tenant value while running cacheQuery() or cachePipeline().

In the case of cacheQuery() you can simply include tenant filtering condition in the root of query, so tenantValue will be automatically set from there example:

const result = await model<SomeModelType>.find({
   someCondition : {$gt: 123},
   tenantId: "someTenantUniqueValue",
   ... //rest of the query
}).cacheQuery()

In other cases for cacheQuery() and cachePipeline() you have to pass tenantValue manually by passing params

/// with cachePipeline()
const result = await model.aggregate<AggregationResultType>([]).cachePipeline({ multitenantValue: 'someTenantUniqueValue' });

/// with cacheQuery()
const result = await model<SomeModelType>.find({}).cacheQuery({multitenantValue : 'someTenantUniqueValue'})

:electric_plug: Auto-cleaner Plugin

This plugin works on mongoose Document and Query/Model operations events. In the case of Document events, we already know which of the document was changed - as it was the parent of the event. But records affected by Query/Model events are predicted according to query conditions and options. Currently supported Mongoose events:

Deleting operations -> findByIdAndRemove, findByIdAndDelete, findOneAndDelete, findOneAndRemove, deleteOne, deleteMany, remove
Saving operations -> updateOne, findOneAndUpdate, findByIdAndUpdate, updateMany, save, insertMany

If anything is missing and it's worth implementing - let me know!

:bug: Debugging

For enabling debug mode, you have to pass multitenantKey into wrapper config, so it will look like

applySpeedGooseCacheLayer(mongoose, {
  redisUri: process.env.REDIS_URI,
  debugConfig?: {
        enabled?: true,
        /** Optional: An array of mongoose models to debug, if not set then the debugger will log operations for all of the models */
        debugModels?: ['yourModelName'],
        /** Optional: An array of operations to debug, if not set then the debugger will log all operations */
        debugOperations?: SpeedGooseDebuggerOperations[],
    }
})

:wrench: Configuration and method options

applySpeedGooseCacheLayer(mongoose, speedgooseConfig)

    /** Connection string for Redis containing URL, credentials, and port. It's required to make cache sync working */
    redisUri?: string;
    /** Connection options for Redis. If redisOptions set, redisUri will be ignored */
    redisOptions?: string;
    /** Config for multitenancy. */
    multitenancyConfig?: {
        /** If set, then the cache will work for multitenancy. It has to be a multitenancy field indicator, that is set at the root of every MongoDB record. */
        multitenantKey: string;
    },
    /** You can pass the default TTL value for all operations, which will not have it passed as a parameter. Value is in seconds. By default is 60 seconds */
    defaultTtl?: number;
    /** If true then will perform TTL refreshing on every read. By default is disabled */
    refreshTtlOnRead?: boolean;
    /** Config for debugging mode supported with debug-js */
    debugConfig?: {
        /** When set to true, it will log all operations or operations only for enabled namespaces*/
        enabled?: boolean,
        /** An array of mongoose models to debug, if not set then the debugger will log operations for all of the models */
        debugModels?: string[],
        /** An array of operations to debug, if not set then the debugger will log all operations */
        debugOperations?: SpeedGooseDebuggerOperations[],
    /** Cache strategy for shared results, by default it is SharedCacheStrategies.REDIS
     * Available strategies: SharedCacheStrategies.REDIS and SharedCacheStrategies.IN_MEMORY */
    sharedCacheStrategy?: SharedCacheStrategies,
    /** Indicates if caching is enabled or disabled, by default is enabled */
    enabled?: boolean
    }

`cacheQuery(operationParams)` and `cachePipeline(operationParams)`

{
    /** It tells to speedgoose for how long a given query should exist in the cache. Value is in seconds. By default is 60 seconds. Set 0 to make it disabled. */
    TTL?: number;
    /** Useful only when using multitenancy. Could be set to distinguish cache keys between tenants.*/
    multitenantValue?: string;
    /** Your custom caching key.*/
    cacheKey?: string;
    /** It tells to speedgoose to refresh the ttl time when it reads from a cached results.*/
    refreshTtlOnRead?: boolean;
}

`SpeedGooseCacheAutoCleaner(...)`

{
    /**
     * Could be set to check if a given record was deleted. Useful when records are removed by setting some deletion indicator like "deleted" : true
     * @param {Document} record mongoose document for which event was triggered
    **/
    wasRecordDeletedCallback?: <T>(record: Document<T>) => boolean
}

`clearCacheForKeys(cacheKey)`

/**
 * Can be used for manually clearing the cache for a given cache key
 * @param {string} key cache key
*/
clearCacheForKeys(cacheKey: string) : Promise<void>

`clearCachedResultsForModel(modelName,tenantId)`

/**
 * Can be used for manually clearing the cache for given modelName.
 * @param {string} modelName name of registered mongoose model
 * @param {string} multitenantValue [optional] unique value of your tenant
*/
clearCachedResultsForModel(modelName: string, multitenantValue?: string) : Promise<void>

:dart: Roadmap

Separated documentation
Add more examples
Deep hydration for nested documents
Cache-based population
Manual cache clearing for custom keys
Refreshing TTL on read
Support for clustered servers
Flowchart of logic
Tests
- commonUtils
- debuggerUtils
- mongooseUtils
- queryUtils
- cacheClientUtils
- cacheKeyUtils
- hydrationUtils
- redisUtils
- extendAggregate
- extendQuery
- mongooseModelEvents
- wrapper
- inMemory caching strategy
- Redis caching strategy
Multitenancy (tenant field indicator) support
Debugging mode
Support for more cache storage
- In memory
- Memcached https://github.com/arqo123/speedgoose/issues/49

See the open issues for a full list of proposed features (and known issues).

:ticket: Contributing

Want to contribute? Great! Open a new issue or pull request with the solution for a given bug/feature. Any ideas for extending this library are more than welcome.