Embeddings-splitter NPM

Getting started

Installation

npm i embeddings-splitter

Usage

Split long strings

split makes sure your string will are short enough to be embedded. (default split size is 500 tokens, but you OpenAI embeddings allow you to go up to 8191)

import { split } from 'embeddings-splitter';

const chunks = split('somVeryLongText...');

// example with biggest chunk size
const chunks = split('someVeryLongText', 8191)

// now you can send these chunks to be embedded

Merge chunks into single string

This is useful when you want to do generative search.

import { merge } from 'embeddings-splitter';

const chunks = ['i am a text', 'that needs to be interpreted as one ', 'for a prompt to make sense'];
const context = merge(chunks);

// e.g. of what to do with merged array
const question = 'what is this text about?"

const prompt = Answer the question based on the context below, and if the question can't be answered based on the context, say "I don't know"\n\nContext: ${context}\n\n---\n\nQuestion: ${question}\nAnswer:

createCompletion(prompt)