@cmmv/normalizer v0.0.7

Description

The @cmmv/normalizer module provides tools for parsing, transforming, and normalizing data from large files in various formats, such as JSON, XML, and YAML. Built for performance and scalability, it uses stream-based parsing to handle large datasets with minimal memory usage. The module integrates seamlessly with CMMV models and contracts, ensuring data consistency and validation.

Additionally, @cmmv/normalizer includes tokenization capabilities for handling sensitive data, leveraging @cmmv/encryptor for AES256 encryption and elliptic curve key generation.

Features

• Stream-Based Parsing: Efficiently processes large files with low memory consumption.
• Multi-Format Support: JSON, XML, YAML, and customizable parsers.
• Data Normalization: Transform and validate data using configurable schemas.
• Integration: Works seamlessly with CMMV contracts and models.
• Tokenization: Secure sensitive data with AES256 and elliptic curves.
• Customizable Pipelines: Create custom transformers and validations.

Installation

Install the @cmmv/normalizer package via npm:

$ pnpm add @cmmv/normalizer

Quick Start

Below is an example of parsing and normalizing a JSON file with a custom schema:

import { 
    JSONParser, AbstractParserSchema, 
    ToLowerCase, Tokenizer, ToDate, ToObjectId 
} from '@cmmv/normalizer';

import { ECKeys } from '@cmmv/encryptor';

const keys = ECKeys.generateKeys();

const tokenizer = Tokenizer({
    publicKey: ECKeys.getPublicKey(keys),
});

class CustomerSchema extends AbstractParserSchema {
    public field = {
        id: {
            to: 'id',
            transform: [ToObjectId],
        },
        name: { to: 'name' },
        email: {
            to: 'email',
            validation: /^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}$/,
            transform: [ToLowerCase],
        },
        phoneNumber: {
            to: 'phone',
            transform: [tokenizer],
        },
        registrationDate: {
            to: 'createdAt',
            transform: [ToDate],
        },
    };
}

new JSONParser({
    input: './sample/large-customers.json',
    schema: CustomerSchema,
})
.pipe(data => console.log('Processed Data:', data))
.once('end', () => console.log('Parsing Complete'))
.once('error', error => console.error('Parsing Error:', error))
.start();

Supported File Formats

The @cmmv/normalizer module includes built-in parsers for the following formats:

For custom formats, you can create your own parser by extending the AbstractParser.

API Reference

AbstractParser

All parsers extend the AbstractParser class, providing a consistent interface for processing data:

Built-in Normalizers

The module provides several built-in normalizers for transforming data fields:

Custom Parser

You can create a custom parser by extending AbstractParser. Here’s an example for a CSV parser:

import * as fs from 'node:fs';
import * as path from 'node:path';
import * as readline from 'node:readline';
import { AbstractParser, ParserOptions } from '@cmmv/normalizer';

export class CSVParser extends AbstractParser {
    constructor(options?: ParserOptions) {
        super();
        this.options = options;
    }

    public override start() {
        const inputPath = path.resolve(this.options.input);

        if (fs.existsSync(inputPath)) {
            const readStream = fs.createReadStream(inputPath);

            const rl = readline.createInterface({
                input: readStream,
                crlfDelay: Infinity, // Handle all newlines (Windows, Unix)
            });

            let headers: string[] | null = null;

            rl.on('line', (line) => {
                if (!headers) {
                    // First line contains headers
                    headers = line.split(',');
                    return;
                }

                // Map CSV columns to header fields
                const values = line.split(',');
                const record = headers.reduce((acc, header, index) => {
                    acc[header.trim()] = values[index]?.trim();
                    return acc;
                }, {});

                // Process each record
                this.processData.call(this, record);
            });

            rl.on('close', this.finalize.bind(this));
            rl.on('error', (error) => this.error.bind(this, error));
        } else {
            console.error(`File not found: ${inputPath}`);
        }
    }
}

The @cmmv/normalizer module provides a powerful and extensible framework for parsing and normalizing large datasets in various formats. Its tight integration with CMMV contracts and models ensures consistency and scalability, making it an essential tool for building robust Node.js applications.