@rainbow-o23/n3 v1.0.59

o23/n3

o23/n3 provides the most basic pipeline steps.

Snippet Supporting

Almost all pipeline steps use dynamic snippets to some extent. In order to facilitate various operations on objects in the snippet, o23/n3 injects rich function support in advance when executing these scripts. All the function support provided by the pipeline steps can be directly obtained and used in scripts using the $helpers handle. For example:

const currentTime = $helpers.$date.now();

When using scripts, pay attention to the usage of variables. Typically:

• $factor represents the incoming data and can be used in most snippet definitions,
• $result represents the processed data and only appears in the toResponse snippet of Fragmentary,
• $request represents the original request data and can be used in almost all snippets, but it is not recommended,
• $helpers represents function supporting and can be used in all snippets,
• $options represents a set of data, usually in error handles.

Typescript support

In dynamic snippet, TypeScript syntax can also be used. Currently, o23/n3 is compiled using ES2022 syntax. It is important to note that dynamic script fragments are function bodies, so import/export syntax is not supported. Moreover, they are compiled in loose mode, and the compilation process does not report any errors. Additionally, for script security reasons, the following keywords or classes are also not supported.

• process
• global
• eval
• Function

Basic Steps

Fragmentary

Usually, when processing logic, we do not need all the memory contexts, but only need to extract certain fragments for processing and return the processing results to the context for subsequent logic to continue processing. Therefore, o23/n3 provides a relevant implementation, allowing pipeline steps to flexibly access the relevant memory data and write back the processed result data to the context in the required format. All pipeline steps should inherit from this implementation to obtain the same capabilities.

Constructor Parameters

Merge Request

Error Handles

Each type of error handle can be either a snippet or a set of pipeline steps. If it is defined as pipeline steps, the first step will receive the request data in the following format:

const RequestContent = {$code: errorCode, $error: error, $factor: fragment, $request: request};

Except for the first step, each subsequent step will receive request data that depends on the processing result of the previous step.

Get Property

Constructor Parameters

The data source of given property name is after fromRequest transformation.

Delete Property

Constructor Parameters

Multi-level property names is not supported, make sure the data source of given property name is after fromRequest transformation.

Snippet

Constructor Parameters

Step Sets

Constructor Parameters

Execute the given pipeline steps in order.

Conditional

Constructor Parameters

First, execute the check snippet and return true. Then execute the given pipeline steps in order. Otherwise, execute the given otherwise pipeline steps in order.

Routes (Switch)

Constructor Parameters

The conditional step is defined as follows:

export interface RoutesConditionalStepOptions {
	check: ScriptFuncOrBody<ConditionCheckFunc>,
	steps?: Array<PipelineStepBuilder>;
}

Execute each given condition check in order. If a condition is true, execute the corresponding pipeline steps in order. If none of the conditions are true, execute the given otherwise pipeline steps in order.

For Each

Constructor Parameters

const RequestContent = {$code: errorCode, $error: error, $factor: fragment, $request: request};

The specified set of pipeline steps will be applied to each element of the array, and all the execution results will be gathered into an array and returned. It is important to note that the error handles of the For Each pipeline steps will be used for each iteration of executing individual elements, rather than being applied to the execution of the array as a whole. If the array is empty, the given pipeline steps will not be executed. The request data obtained in the first step of each element's execution will appear in the following format:

const RequestContent = {$content: content, $item: element, $semaphore};

If you need to terminate the loop prematurely, you just need to return the $semaphore signal from the request data. The loop will automatically end, and the collected processing results will be gathered and returned as an array.

Parallel

Constructor Parameters

The specified set of pipeline steps will be executed parallel, and

• All the execution results will be gathered into an array and returned when race is false,
• Returns the first settled result when race is true,

It is important to note that

• Each sub pipeline step will use the same request data, they share the same content memory address. So please be very careful NOT to attempt to modify the request data in the sub pipeline steps. Alternatively, you can use cloneData to create a copy of the request data for each sub pipeline steps, so that request data operations can be modified in certain sub pipeline steps without affecting other sub pipeline steps.
• The error handles of the For Each pipeline step will be used for each iteration of executing individual elements, rather than being applied to the execution of the array as a whole.

Trigger Pipeline

Constructor Parameters

Execute the specified pipeline.

Trigger Pipeline Step

Constructor Parameters

Execute the specified pipeline step.

Async

Constructor Parameters

Execute the given pipeline steps in order, but asynchronous. Therefore, no result returned.

Database (TypeOrm) Steps

Environment Parameters

DB represents database name.

• For mysql, default kept on global is true,
• For Better-SQLite3, default kept on global is false.

MySQL

When typeorm.DB.type=mysql:

Mysql driver read DateTime column to javascript string.

PostgreSQL

When typeorm.DB.type=pgsql:

PgSQL driver read DateTime column to javascript Date.

MSSQL

When typeorm.DB.type=mssql:

MSSQL of TypeORM for more details.

MSSQL driver read DateTime column to javascript Date.

Oracle

When typeorm.DB.type=oracle:

Oracle driver read DateTime column to javascript Date.

Better SQLite3

When typeorm.DB.type=better-sqlite3:

SQLite save DateTime column as javascript string.

NEVER use it in production.

Constructor Parameters

Autonomous transactions take precedence over the transaction name, meaning that if an autonomous transaction is enabled, the transaction specified by the transaction name will be ignored. If you need to use the transaction name, you must nest the pipeline steps within transactional step sets, and ensure that the datasource name and transaction name remain the same.

By SQL

Environment Parameters

Parsed SQL will not be cached when there is one-of syntax.

Constructor Parameters

Native SQL Support & Enhancement

SQL supports native database syntax. At the same time, o23/n3 enhances SQL syntax, allowing the use of the $property syntax to retrieve corresponding data from data objects, also supports multi-level property names, connected by .. For example, $person.name represents that person is an object and name is a property under person. The following are the supported syntax features:

• IN ($...names): one-of, names should be an array,
• LIKE $name%: starts-with,
• LIKE $%name: ends-with,
• LIKE $%name%: contains.

Name mapping is case-sensitive.
LIKE is case-sensitive.

Since different databases have varying degrees of support for dialects, o23/n3 also provides appropriate enhanced support for this:

• For pagination, $.limit($offset, $limit) will be translated and executed in the appropriate dialect. For example, - MySQL uses LIMIT $offset, $limit, - PostgreSQL uses OFFSET $offset LIMIT $limit. - MSSQL and Oracle use OFFSET $offset ROWS FETCH NEXT $limit ROWS ONLY, - MSSQL requires an ORDER BY clause for pagination SQL. If there is no ORDER BY clause, will use ORDER BY 1 OFFSET $offset ROWS FETCH NEXT $limit ROWS ONLY.
• For JSON column, because some databases (such as MSSQL) do not have a JSON column type, they cannot automatically replace strings in the result set with JSON objects, - Use config as "config.@json" to explicitly indicate that the config column is of JSON data type. - Use $config.@json to explicitly indicate that the config property of given parameters is of JSON data type.
• For boolean column which use numeric(int/smallint/tinyint) as storage type, because some databases (such as PostgreSQL) cannot automatically convert boolean values in memory to numeric 0 or 1 in the database, - Use enabled as "enabled.@bool" to explicitly indicate that the enabled column is of boolean in-memory and numeric in database data type. - Use $enabled.@bool to explicitly indicate that the enabled property of given parameters is of boolean in-memory and numeric in database data type.
• For datetime (MySQL, MSSQL) / timestamp (Oracle, PostgreSQL) column, - Use created_at as "createdAt.@ts" to explicitly indicate that the createdAt column is of string in-memory and timestamp in database data type. - Use $createdAt.@ts to explicitly indicate that the createdAt property of given parameters is of string in-memory and timestamp in database data type.

We recommend that if you need to consider support for multiple database dialects, using enhanced syntax will make it easier to write SQL. If you only need to support a specific database, then using its standard syntax is sufficient.

It is important to note that some databases (such as PostgreSQL) do not differentiate column names by case. This can affect the property names of the returned objects in the result set (usually recommended in camel case). Therefore, even though it is not a syntax enhancement, it is strongly recommended to use aliases to standardize the column names in the returned result set, for example, PERSON_NAME AS "personName", please pay attention to the use of quotation marks to correctly preserve the case.

Load One by SQL

Request and Response

// request
export interface TypeOrmLoadBasis extends TypeOrmBasis {
	params?: Array<TypeOrmEntityValue> | TypeOrmEntityToSave;
}

// response
export type TypeOrmEntityToLoad = Undefinable<DeepPartial<ObjectLiteral>>;

Load Many by SQL

Request and Response

// request
export interface TypeOrmLoadBasis extends TypeOrmBasis {
	params?: Array<TypeOrmEntityValue> | TypeOrmEntityToSave;
}

// response
Array<TypeOrmEntityToLoad>;

Load Many by SQL, Use Cursor

Environment Parameters

Constructor Parameters

Request and Response

// request
export interface TypeOrmLoadBasis extends TypeOrmBasis {
	params?: Array<TypeOrmEntityValue> | TypeOrmEntityToSave;
}

// response
Array<any>;

By specifying fetchSize, each batch of data retrieved will execute sub-steps. Before executing the sub-steps, the data to be passed to it will be calculated using the streamTo function. If streamTo is not specified, the batch of data retrieved itself will be passed to the sub-steps. If the sub-steps is not specified, all retrieved data will be merged and returned.

Therefore, the number of times the sub-step is executed is related to the quantity of data and the fetchSize. Meanwhile, each time the sub-step is invoked, the context will include a $$typeOrmCursorRound variable indicating the current batch (starting from 0), and a $typeOrmCursorEnd variable indicating whether it is the last batch.

Save by SQL

Request and Response

// request
export interface TypeOrmSaveBasis extends TypeOrmBasis {
	values?: Array<TypeOrmEntityValue> | TypeOrmEntityToSave;
}

// response
export type TypeOrmIdOfInserted = TypeOrmIdType;
export type TypeOrmCountOfAffected = number;
export type TypeOrmWrittenResult = TypeOrmIdOfInserted | TypeOrmCountOfAffected;

Bulk Save by SQL

Request and Response

// request
export interface TypeOrmBulkSaveBasis extends TypeOrmBasis {
	items?: Array<Array<TypeOrmEntityValue> | TypeOrmEntityToSave>;
}

// response
export type TypeOrmIdsOfInserted = Array<TypeOrmIdOfInserted>;
export type TypeOrmCountsOfAffected = Array<TypeOrmCountOfAffected>;
export type TypeOrmBulkWrittenResult = TypeOrmIdsOfInserted | TypeOrmCountsOfAffected;

By Snippet

Constructor Parameters

A TypeOrm Query Runner instance, $runner, will be passed to the snippet, and the snippet can use this instance to perform any operation on the database.

You do not need to manually start a transaction, whether you are using autonomous transaction or if it is nested within transaction step sets. The $runner instance passed to the snippet will automatically start a transaction.

Transactional

Transactional step sets are a wrapper for a set of pipeline steps that require the same transaction. It means that all the sub-steps inside a transactional step set can be executed within a single transaction. However, not all sub-steps within the set necessarily have to be transactional. Only the ones that need to be executed within the same transaction need to define the same transaction name as the step set. Additionally, nested transactions are also supported, which means Transactional Step Sets can be nested as well.

Steps with same datasource name and transaction name should be within same transaction.

Constructor Parameters

Http Steps

Fetch

Environment Parameters

SYSTEM represents endpoint system, ENDPOINT represents endpoint url. For example:

CFG_ENDPOINTS_ORDER_PURCHASE_URL=https://order.com/purchase
CFG_ENDPOINTS_ORDER_PAYMENT_URL=https://order.com/payment

Constructor Parameters

• transparentHeaderNames and omittedTransparentHeaderNames: Use transparentHeaderNames to specify the names of request headers whose values need to be transparently passed from the input parameters to the upstream service. Separate the names with ;. The names support using . for connection so that values from multi-level objects can be directly retrieved. For example, account.name will retrieve the value of the name property from the account property of the input object. When writing the values into the header values, the following rules apply: - If the value is an array, use , to connect the elements. null and empty strings will be filtered out. - If the value is an object, use the object's keys to generate multiple headers. null and empty strings will be filtered out. - For other values, convert them to strings. null and empty strings will be filtered out. - Note that an empty string does not include blank strings, and no automatic trimming will be performed.

  If the transparentHeaderNames at the step level is not defined, use the definition in endpoints.SYSTEM.ENDPOINT.headers.transparent. If it is also not defined at the endpoint level, use the definition in endpoints.SYSTEM.global.headers.transparent.

  After obtaining the transparently passed request headers, check the definition of omittedTransparentHeaderNames. If it is defined, remove the corresponding headers from the headers. omittedTransparentHeaderNames is case-insensitive. Similarly, if the omittedTransparentHeaderNames at the step level is not defined, use the definition in endpoints.SYSTEM.ENDPOINT.headers.transparent.omitted. If it is also not defined at the endpoint level, use the definition in endpoints.SYSTEM.global.headers.transparent.omitted.

  For example: If the input data contains {account: {name: 'John', token: '******'}} and transparentHeaderNames is defined as account, then two transparently passed headers will be obtained: name=John and token=******. At this time, if omittedTransparentHeaderNames is defined as name, the headers that will be finally transparently passed to the upstream service are token=******, and name will be ignored.

Request headers

There are three ways to transmit request headers to upstream services. In order of priority from high to low, they are headersGenerate, headers.transparent, and headers. If a header appears in a higher-priority method, the header with the same name generated by a lower-priority method will be ignored. Note that the matching is case-sensitive.

Normally, if you need to transparently pass the request headers from the client to the upstream service, you should use headers: true in the pipeline definition. Then you can directly use transparentHeaderNames: headers to obtain all the request headers, and then use omittedTransparentHeaderNames for necessary filtering.

It should be noted that since the fetch step initiates a new request to the upstream service, its request structure and data will be modified or reset according to requirements. Therefore, even if you need to transparently pass the request headers, some of the headers are still not applicable. So by default, the two headers content-encoding and content-length will be filtered out. No matter how the request headers are generated in the above process, these two headers are always automatically generated by node-fetch.

Tracing in upstream service groups

Use endpoints.SYSTEM.ENDPOINT.trace.header.name or endpoints.SYSTEM.global.trace.header.name to define the trace request header name for the upstream system. The value will be attempted to be retrieved from the response and stored in the execution context to be passed to the upstream system within the same SYSTEM.ENDPOINT or SYSTEM.

Installation

Note since nestjs only support commonjs module, which means node-fetch 3 series cannot be imported since it is built on esm mode.