@or-sdk/lookup v1.14.0

Lookup SDK

The @or-sdk/lookup package provides a client for working with the Question and Answer and Text Search system in OneReach.ai a.k.a Lookup. With this client, you can perform operations such as loading documents, creating collections, searching in collections, asking questions to generate answers based on the loaded documents, and managing properties of collections.

Concepts

Collection

A collection is a group of searchable items, also known as passages. It allows you to organize and manage related documents and their passages together under a single entity. When you perform a search or ask a question, the Lookup system will search through the passages within the specified collection.

Document

Documents are used to load a bulk of data into a collection. They can be any type of text-based file such as PDFs, text files, or web pages. When you provide a URL for a document, the Lookup system downloads the content, splits it into smaller chunks (passages), and adds them to the collection.

Passage

A passage is an atomic piece of data that represents a portion of a document. It is the smallest searchable unit within a collection. When you search or ask a question, the Lookup system uses vector similarity to find the most relevant passages from all the documents in the collection.

The search process involves comparing the vector representation of the input query (search term or question) with the vector representations of the passages in the collection. The results are ranked based on the similarity between the query and passages, with the most similar passages being returned as the search results.

Property

A property is an additional piece of information that can be added to a collection to store custom metadata about the collection. Properties can have a name, datatype, and an optional description. You can manage properties by adding, updating, or deleting them from a collection.

Installation

To install the package, run the following command:

$ npm install @or-sdk/lookup

Usage

To use the Lookup client, you need to create an instance of the class with the appropriate configuration options. You can either provide the direct API URL or the service discovery URL. Here is an example:

import { Lookup } from '@or-sdk/lookup';

// with direct api url
const lookup = new Lookup({
  token: 'my-account-token-string',
  serviceUrl: 'http://example.lookup/endpoint',
  feature: 'master', // service feature name
});

// with service discovery(slower)
const lookup = new Lookup({
  token: 'my-account-token-string',
  discoveryUrl: 'http://example.lookup/endpoint',
  feature: 'master', // service feature name
});

Once you have an instance of the Lookup client, you can use its methods to interact with the OneReach.ai Lookup system. The available methods are:

Example

This example demonstrates how to abort the loadDocument request by providing an AbortSignal.

const controller = new AbortController();
const signal = controller.signal;

setTimeout(() => controller.abort(), 5000);

try {
  const collectionId = 'a1b2c3d4-uuid';
  const document = await lookup.loadDocument(collectionId, {
    name: 'Sample Document',
    description: 'A sample document',
    url: 'http://example.com/sample-document.pdf',
    defaultProperties: {
      author: 'John Doe',
      publishDate: '2020-12-31'
    }
  }, { signal });
} catch (err) {
  if (err.name === 'AbortError') {
    console.log('loadDocument request was aborted');
  } else {
    console.error(err);
  }
}

In the above example, if the loadDocument call does not complete within 5 seconds, the AbortController's abort method is called, which signals that the request should be aborted. The loadDocument call then throws an AbortError, which is caught and logged. This is useful for putting a time limit on requests that might otherwise hang indefinitely due to network issues.

Note: The abort signal is available for all the methods of the Lookup SDK.

// Example response { id: 'e5f6g7h8-uuid', accountId: 'i9j0k1l2-uuid', collection: 'a1b2c3d4-uuid', name: 'Sample Document', description: 'A sample document', status: 'LOADING' }

After the document loading process, the status will update according to the result:

```typescript
// Example response when the document loading is successful
{
  id: 'e5f6g7h8-uuid',
  accountId: 'i9j0k1l2-uuid',
  collection: 'a1b2c3d4-uuid',
  name: 'Sample Document',
  description: 'A sample document',
  status: 'READY'
}

// Example response when there is an error during the document loading
{
  id: 'e5f6g7h8-uuid',
  accountId: 'i9j0k1l2-uuid',
  collection: 'a1b2c3d4-uuid',
  name: 'Sample Document',
  description: 'A sample document',
  status: 'Could not load the document due to an error...'
}

To handle cases where you need to wait for the document to fully load, you can use polling. Here's an example:

import { sleep } from 'utils';

async function waitForDocumentLoading(collectionId: string, documentId: string) {
  let document;
  let isLoading = true;

  while (isLoading) {
    document = await lookup.getDocument(collectionId, documentId);

    if (document.status === 'READY' || typeof document.status === 'string') {
      isLoading = false;
    } else {
      await sleep(5000); // Wait for 5 seconds before checking the status again
    }
  }

  return document;
}

async function loadDocumentAndWait(collectionId: string, documentParams: LoadDocument) {
  const document = await lookup.loadDocument(collectionId, documentParams);
  const loadedDocument = await waitForDocumentLoading(collectionId, document.id);
  return loadedDocument;
}

// Example usage
const collectionId = 'a1b2c3d4-uuid';
const documentParams = {
  name: 'Sample Document',
  description: 'A sample document',
  url: 'http://example.com/sample-document.pdf',
  defaultProperties: {
    author: 'John Doe',
    publishDate: '2020-12-31'
  }
};

const loadedDocument = await loadDocumentAndWait(collectionId, documentParams);
console.log('Loaded document:', loadedDocument);

In this example, the loadDocumentAndWait function first calls the loadDocument method to start loading a document and then uses the waitForDocumentLoading function to poll for the document's status until it's either READY or an error message is provided. The sleep function is used to introduce a delay before checking the status again.

createCollection

Creates a new collection with optional properties. A collection is a group of related documents that are searchable together. Additional custom properties can be defined to store metadata associated with the documents within the collection.

Params

• createParams: CreateCollection - An object containing the properties of the collection:
  • name: string - The name of the collection. It can start only with a letter and it must be at least 3 characters long, and cannot exceed 100 characters in length.
  • description: string (optional) - A brief description of the collection. If present, it must be at least 3 characters and no more than 500 characters in length.
  • properties: Property[] (optional) - An array of custom properties to add to the collection. Each property should include a name and datatype. Please see list of supported datatypes.
  • imageUrl: string (optional) - The URL of the thumbnail image for the collection. It must be at least 3 characters and it cannot exceed 2048 characters in length.
  • answerInstruction: string (optional) - An instruction used to generate a correct answer. If present, it cannot be more than 1000 characters long.
  • questionInstruction: string (optional) - An instruction used to generate a proper search query. If present, it cannot be more than 1000 characters long.
  • greetingInstruction: string (optional) - An instruction used to generate a correct first message. If present, it cannot be more than 1000 characters long.
  • temperature: number (optional) - Sampling temperature to use. If present, it must be a number between 0 and 2.
  • maxTokens: number (optional) - The maximum output length from the Large Language Model (LLM). If present, it must be a number between 128 and 2048.
  • frequencyPenalty: number (optional) - Value to penalize new tokens based on their frequency in existing text. If present, it must be a number between -2 and 2.
  • presencePenalty: number (optional) - Value to penalize new tokens based on their presence in existing text. If present, it must be a number between -2 and 2.
  • maxDistance: number (optional) - Value to filter out passages that are further than this value from the question or context. If present, it must be a number between 0 and 1.
  • modelName: string (optional) - Name of the large language model to be used. It can't be more than 64 characters long.
  • type: CollectionType (optional) - by default DEFAULT collection with content property. QNA_V1 collection with question and answer properties and allow to post process loaded content for create question and answer pairs. IDW_V1 skills with skills properties sectionAbbreviation, skillAbbreviation, skillDescription, skillId, skillName, skillNumbering, skillQuestion, skillSection, skillUrl, starterIconColor, starterIconUrl.

Example

Creating a collection with a name, description, and additional instructions

const collection = await lookup.createCollection({
  name: 'Sample Collection',
  description: 'A sample collection for Lookup',
  properties: [
    { name: 'author', dataType: 'string' },
    { name: 'year', dataType: 'int' },
  ],
  imageUrl: 'https://example.com/sample-collection.jpg',
  answerInstruction: 'Provide relevant answer from the text',
  questionInstruction: 'Generate a question based on understanding of the text',
  greetingInstruction: 'Initiate conversation providing context about the document',
  temperature: 0.7,
  maxTokens: 200,
  frequencyPenalty: 0.8,
  presencePenalty: 1,
  maxDistance: 0.5,
  modelName: 'text-davinci-002',
  type: 'DEFAULT'
});

console.log(collection);

This will print collection information with its ID, name, and other parameters. It will also include any custom properties. Creating a collection with additional properties

const collectionWithProperties = await lookup.createCollection({
  name: 'Collection With Properties',
  description: 'A collection with properties',
  properties: [
    {
      name: 'author',
      dataType: 'string',
      description: 'Author of the document',
    },
    {
      name: 'publishDate',
      dataType: 'date',
      description: 'Date when the document was published',
    },
  ],
});

// Example response
{
  id: 'x1y2z3w4-uuid',
  accountId: 'i9j0k1l2-uuid',
  name: 'Collection With Properties',
  description: 'A collection with properties',
  properties: [
    {
      name: 'author',
      dataType: 'string',
      description: 'Author of the document',
    },
    {
      name: 'publishDate',
      dataType: 'date',
      description: 'Date when the document was published',
    },
  ],
}

Note: The custom properties defined for the collection can be used for filtering, sorting, and selecting specific properties when performing search or ask operations.

addProperty

Add a custom property to an existing collection. Custom properties can be used to store additional metadata about the collection or its passages. The available data types for properties can be found at https://weaviate.io/developers/weaviate/config-refs/datatypes. Note that there are reserved property names 'accountId', 'collection', 'document', 'content', 'loaderMetadata', 'sourceUrl'

Params

• collectionId (string): The ID of the collection to which the property will be added.
• property (Property): The property to be added, which includes:
  • name (string): The name of the property, e.g. nameOfTheAuthor. Name should not include any space.
  • dataType (string): The data type of the property. See https://weaviate.io/developers/weaviate/config-refs/schema#datatypes for the available types
  • description (string, optional): An optional description of the property.
  • vectorize (boolean, optional): Whether the property should be vectorized. Default is false. It will use the default vectorization method for the data type. Allowed only for text dataType.

Example

const collectionId = 'a1b2c3d4-uuid';
const property = {
  name: 'author',
  dataType: 'text',
  description: 'Author of the document',
  vectorize: true,
};

await lookup.addProperty(collectionId, property);

// No direct response, the property is added to the collection

You can verify the added property by retrieving the collection:

const collection = await lookup.getCollection(collectionId);

// Example response
{
  id: 'a1b2c3d4-uuid',
  accountId: 'i9j0k1l2-uuid',
  name: 'Sample Collection',
  description: 'A sample collection',
  countPassages: 42,
  properties: [
    {
      name: 'author',
      dataType: 'text',
      description: 'Author of the document',
      vectorize: true,
    },
  ],
}

deleteCollection

Delete a collection by its ID. This operation will permanently remove the collection along with all the documents, passages, and custom properties associated with the collection.

Params

• collectionId: string - The ID of the collection to delete

Example

const collectionId = 'a1b2c3d4-uuid';
await lookup.deleteCollection(collectionId);

Note: Use this operation with caution, as deleting a collection is an irreversible action. Once a collection is deleted, all data associated with it will be lost and cannot be recovered. Always make sure to backup any important data before deleting a collection.

search

Search for documents in a collection based on a query. This method accepts several parameters that can be used to filter, sort, and limit the results.

Params

• term: string - The search term, query, or keywords
• limit: number (optional) - Limit the number of search results
• where: Record<string, unknown> (optional) - Weaviate filter object for advanced filtering. For more details, visit https://weaviate.io/developers/weaviate/api/graphql/filters
• select: string[] (optional) - An array of custom properties to include in the results
• maxDistance: number (optional) - Filters out passages that are further than this value from the question or context. Measured by cosine metric. Default is 0. Range accepted is 0 to 1

Example

const collectionId = 'a1b2c3d4-uuid';

const searchResults = await lookup.search(collectionId, {
  term: 'search term',
  limit: 10,
  where: {
    operator: 'And',
    operands: [
      {
        operator: 'GreaterThan',
        path: ['publishDate'],
        valueDate: '2020-01-01T00:00:00Z',
      },
      {
        operator: 'LessThan',
        path: ['publishDate'],
        valueDate: '2022-01-01T00:00:00Z',
      },
    ],
  },
  select: ['author', 'publishDate'],
});

// Example response
[
  {
    id: 'e5f6g7h8-uuid',
    distance: '0.1234',
    content: 'Found passage content',
    sourceUrl: 'http://example.com/source-url',
    loaderMetadata: {
      custom: 'metadata'
    },
    document: {
      id: 'i9j0k1l2-uuid',
      name: 'Document Name'
    },
    select: ['author', 'publishDate']
  }
]

In this example, the search method is called with advanced filtering and property selection options. The where parameter is used to filter the results based on the publishDate property, and the select parameter is used to include the author and publishDate properties of the selected passages in the response. As a result, the search returns a list of passages that meet the specified conditions and include the custom property values. Along with these properties, the result also includes new properties like sourceUrl, loaderMetadata and document.

ask

Asks a question and generates an answer based on the documents in a collection. This method accepts a set of parameters to configure the question-answering process and to fine-tune the generated outputs.

The ask method supports two modes: 'ask' and 'conversation'. Here are the details of each mode:

1. 'ask' mode:

   This mode allows you to ask a direct, standalone question. In this mode, the processing of the question and the generation of the answer are not influenced by any prior context or conversation history.

   This mode is suitable for scenarios where a single question is to be answered independently, such as a search query or a factual question.

2. 'conversation' mode:

   This mode allows you to ask questions in the context of a conversation. This means that prior messages from the conversation will be taken into account while processing the question and generating the answer.

   This mode is suitable for scenarios where a dialogue is happening, and the generation of an answer is dependent on the previous exchanges in the conversation. In this mode, you can supply previous conversation messages in the messages parameter, which accepts an array of message objects, each with a sender role ('assistant' or 'user') and the content of the message.

Both modes use a language model to generate an answer given a question and a collection of documents. How the prior messages are used to influence the retrieval of information and the generation of responses differs based on the mode used.

Params

• question: string - The question to ask
• messages: CompletionRequestMessage[] (optional) - An array of previous chat messages with roles 'user' and 'assistant'. Default is an empty array
• limit: number (optional) - Limit the number of search results. Default is 5. Accepts values between 1 and 100
• where: Record<string, unknown> (optional) - Weaviate filter object for advanced filtering. For more details, visit https://weaviate.io/developers/weaviate/api/graphql/filters
• select: string[] (optional) - Array of the custom properties to select
• mode: AskMode (optional) - Question mode, can be either 'ask' or 'conversation'. Default is 'conversation'
• answerInstruction: string | null (optional) - A summarization instruction used to generate correct answer. Maximum Length: 1000 characters
• questionInstruction: string | null (optional) - A summarization instruction used to generate correct search query. Maximum Length: 1000 characters
• temperature: number (optional) - Adjusts the randomness in the model's output. Default is 1. Accepts values between 0 and 2
• maxTokens: number (optional) - Maximum output length from the language model. Default is 1024. Accepts values between 128 and 2048
• frequencyPenalty: number (optional) - Penalizes new tokens based on their existing frequency in the text so far. Default is 0. Range accepted is -2 to 2
• presencePenalty: number (optional) - Penalizes new tokens based on whether they appear in the text so far. Default is 0. Range accepted is -2 to 2
• maxDistance: number (optional) - Filters out passages that are further than this value from the question or context. Measured by cosine metric. Default is 0. Range accepted is 0 to 1
• promptProvider: string (optional) - Prompt provider. By default use collection's prompt provider value
• promptModel: string (optional) - Large language model name. By default use collection's model name value

Example

const collectionId = 'a1b2c3d4-uuid';

const askResults = await lookup.ask(collectionId, {
  question: 'What is the meaning of life?',
  messages: [
    { role: 'user', content: 'Hello' },
    { role: 'assistant', content: 'Hi there!' },
  ],
  limit: 10,
  where: {
    operator: 'And',
    operands: [
      {
        operator: 'GreaterThan',
        path: ['publishDate'],
        valueDate: '2020-01-01T00:00:00Z',
      },
      {
        operator: 'LessThan',
        path: ['publishDate'],
        valueDate: '2022-01-01T00:00:00Z',
      },
    ],
  },
  select: ['title', 'author', 'publishDate'],
  mode: 'conversation',
  answerInstruction: 'Generate a brief and concise answer.',
  questionInstruction: 'Find documents related to the meaning of life.',
  temperature: 0.8,
  maxTokens: 1500,
  frequencyPenalty: 0.5;
  presencePenalty: 0.5;
  maxDistance: 0.9,
});

// Example response
{
  result: { role: 'assistant', content: 'The meaning of life is 42.' },
  select: ['title', 'author', 'publishDate'],
  searchResult: [
    {
      id: 'e5f6g7h8-uuid',
      distance: '0.1234',
      content: 'Found passage content',
      sourceUrl: 'https://example.com/meaning-of-life',
      loaderMetadata: {...},
      document: {
        id: 'i9j0k1l2-uuid',
        name: 'On the Meaning of Life',
      },
      select: ['title', 'author', 'publishDate']
      title: 'On the Meaning of Life',
      author: 'John Doe',
      publishDate: '2020-12-31',
    },
    // ... More results if available
  ],
}

updateDocument

Update a document's description in a collection. This method allows you to modify the description of an existing document, providing an updated text to better explain or detail the document's content.

Params

• collectionId: string - The ID of the collection the document belongs to
• documentId: string - The ID of the document to update
• params: UpdateDocument - The update document parameters, which include:
• description: string - The updated description of the document

Example

Updating a document's description in a collection:

const collectionId = 'a1b2c3d4-uuid';
const documentId = 'e5f6g7h8-uuid';
const updatedDocument = await lookup.updateDocument(collectionId, documentId, {
  description: 'Updated document description',
});

// Example response
{
  id: 'e5f6g7h8-uuid',
  accountId: 'i9j0k1l2-uuid',
  collection: 'a1b2c3d4-uuid',
  name: 'Sample Document',
  description: 'Updated document description'
}

Note: Only the document description can be updated using this method. Other metadata such as the document name or URL cannot be modified once the document is created and added to the collection.

updateCollection

Update an existing collection's properties. The collection must be previously created using the createCollection method. The updated collection will preserve its existing properties, ID, and name, but its associated documents might be affected based on the new instructions.

Params

• collectionId: string - The ID of the collection you want to update
• updateParams: UpdateCollection - An object containing the properties you want to update:
  • description: string (optional) - The new description for the collection. Minimum length: 3, Maximum length: 500
  • imageUrl: string (optional) - A collection thumbnail image URL. Minimum length: 3, Maximum length: 2048
  • answerInstruction: string (optional) - A summarization instruction used to generate a correct answer. Maximum length: 1000
  • questionInstruction: string (optional) - A summarization instruction used to generate a correct search query. Maximum length: 1000
  • greetingInstruction: string (optional) - A summarization instruction used to generate a correct first message. Maximum length: 1000
  • temperature: number (optional) - What sampling temperature to use. Default value: 1 Range: 0, 2
  • maxTokens: number (optional) - Max output length from the Large Language Model (LLM). Range: 128, 2048
  • frequencyPenalty: number (optional) - Positive values penalize new tokens based on their existing frequency in the text. Range: -2, 2
  • presencePenalty: number (optional) - Penalize new tokens based on whether they appear in the text. Range: -2, 2
  • maxDistance: number (optional) - Filter out passages that are further then maxDistance from the question or context. Range: 0, 1
  • modelName: string (optional) - Name of the large language model to be used. Maximum length: 64

Example

Updating a collection's description and adding a new question instruction.

const collectionId = 'a1b2c3d4-uuid';
const updatedCollection = await lookup.updateCollection(collectionId, {
  description: 'Updated collection description',
  questionInstruction: 'Consider the main theme of the document for the query',
  maxTokens: 200,
  temperature: 1.5,
  modelName: 'gpt-3'
});

// Example response
{
  id: 'a1b2c3d4-uuid',
  accountId: 'i9j0k1l2-uuid',
  name: 'Sample Collection',
  description: 'Updated collection description',
  questionInstruction: 'Consider the main theme of the document for the query',
  maxTokens: 200,
  temperature: 1.5,
  modelName: 'gpt-3'
}

getDocument

Retrieve a single document from a collection by its ID. A document is a text-based file (e.g., PDF, TXT, or HTML) that contains content, which is split into passages for searching within a collection.

Params

• collectionId: string - The ID of the collection the document belongs to.
• documentId: string - The ID of the document to retrieve.

Example

const collectionId = 'a1b2c3d4-uuid';
const documentId = 'e5f6g7h8-uuid';
const document = await lookup.getDocument(collectionId, documentId);

// Example response
{
  id: 'e5f6g7h8-uuid',
  accountId: 'i9j0k1l2-uuid',
  collection: 'a1b2c3d4-uuid',
  name: 'Sample Document',
  description: 'A sample document',
  sourceUrl: 'http://example.com/sample-document.pdf',
  countPassages: 42,
}

getCollection

Retrieve a collection by its ID. This method returns a collection object including its ID, account ID, name, description, and any additional properties defined during the collection's creation.

Params

• collectionId: string - The ID of the collection to retrieve

Example

In this example, the getCollection method is called with a collectionId to retrieve the corresponding collection object. The response includes the collection's metadata, such as its ID, name, and description, as well as any custom properties defined for the collection (in this case, author and publishDate).

const collectionId = 'a1b2c3d4-uuid';
const collection = await lookup.getCollection(collectionId);

// Example response
{
  id: 'a1b2c3d4-uuid',
  accountId: 'i9j0k1l2-uuid',
  name: 'Sample Collection',
  description: 'A sample collection',
  countPassages: 42,
  properties: [
    {
      name: 'author',
      dataType: 'string',
      description: 'Author of the document',
    },
    {
      name: 'publishDate',
      dataType: 'date',
      description: 'Date when the document was published',
    },
  ],
}

listDocuments

List documents in a collection with optional pagination, query, and ordering.

Params

• collectionId: string - The ID of the collection to retrieve documents from.

• find: Find (optional) - Optional find parameters.

  • query: string (optional) - The search query to filter documents based on their name and/or description.
  • mode: SearchMode (optional) - The search mode, either 'bm25' for full-text search or 'vector' for vector similarity search. Default is 'bm25'
  • size: number (optional) - The number of documents to return per request. Defaults to 10.
  • from: number (optional) - The starting index for fetching the documents, used for pagination.
  • orderProperty: string (optional) - The property by which to order the documents, e.g., 'name' or 'createdAt'.
  • orderDirection: OrderDirection (optional) - The direction to order the documents, either 'asc' for ascending or 'desc' for descending.

Returns

• An array of documents with pagination info.

Example

const collectionId = 'a1b2c3d4-uuid';
const documents = await lookup.listDocuments(collectionId, {
  query: 'search query',
  mode: 'bm25',
  size: 10,
  from: 0,
  orderProperty: 'name',
  orderDirection: 'asc',
});

// Example response
{
  "items": [
    {
      "id": "e5f6g7h8-uuid",
      "accountId": "i9j0k1l2-uuid",
      "collection": "a1b2c3d4-uuid",
      "name": "Sample Document",
      "description": "A sample document",
      "createdAt": "2022-01-02T03:04:05Z"
    }
  ],
  "total": 1
}

listCollections

List collections with optional pagination, ordering, and query.

Params

• find: Find - Optional find parameters.

  • query: string (optional) - The search query to filter collections based on their name and/or description.
  • mode: SearchMode (optional) - The search mode, either 'bm25' for full-text search or 'vector' for vector similarity search. Default is 'bm25'
  • size: number (optional) - The number of collections to return per request. Defaults to 10.
  • from: number (optional) - The starting index for fetching the collections, used for pagination.
  • orderProperty: string (optional) - The property by which to order the collections, e.g., 'name' or 'createdAt'.
  • orderDirection: OrderDirection (optional) - The direction to order the collections, either 'asc' for ascending or 'desc' for descending.

Returns

• An array of collections with pagination info.

Example

const collections = await lookup.listCollections({
  query: 'search query',
  mode: 'bm25',
  size: 10,
  from: 0,
  orderProperty: 'name',
  orderDirection: 'asc',
});

// Example response
{
  "items": [
    {
      "id": "a1b2c3d4-uuid",
      "accountId": "i9j0k1l2-uuid",
      "name": "Sample Collection",
      "description": "A sample collection",
      "createdAt": "2022-01-02T03:04:05Z"
    }
  ],
  "total": 1
}

deleteDocument

Delete a document from a collection by its ID. This will remove the document and all its associated passages from the collection, and the removed data will no longer be searchable.

Params

• collectionId: string - The ID of the collection containing the document
• documentId: string - The ID of the document to delete

Example

const collectionId = 'a1b2c3d4-uuid';
const documentId = 'e5f6g7h8-uuid';
await lookup.deleteDocument(collectionId, documentId);

// No direct response, the document is deleted from the collection

Note: Deleting a document is a permanent operation, and the removed data cannot be recovered. Be cautious when using this method and ensure you have backups of your data if necessary.

createPassage

Create a passage and add it to a specific document within a collection. A passage is an atomic piece of data that represents a portion of a document. It is the smallest searchable unit within a collection.

Params

• collectionId: string - The ID of the collection the document belongs to
• params: CreatePassage - An object containing parameters required to create a passage
• content (optional): The content of the passage for DEFAULT collection type
• question (optional): The question of the passage for QNA_V1 collection type
• answer (optional): The answer of the passage for QNA_V1 collection type
• documentId (string): The ID of the document the passage will be associated with
• property: value (optional): Custom properties of the passage. Note that these properties should be listed in the collection properties
• vector?: number[] (optional): Custom vector representation of the passage.

Example

const collectionId = 'a1b2c3d4-uuid';
const params: CreatePassage = {
  content: 'This is the content of the passage.',
  documentId: 'e5f6g7h8-uuid',
};
const passage = await lookup.createPassage(collectionId, params);

// Example response
{
  id: 'x1y2z3w4-uuid',
  content: 'This is the content of the passage.',
  // ...more properties
}

Note: When creating a passage, you can also include additional custom properties if they are defined in the collection schema. These custom properties can later be used for filtering, sorting, and selecting specific properties when performing search or ask operations.

createManyPassages

Create a batch of passages and add it to a specific document within a collection. A passage is an atomic piece of data that represents a portion of a document. It is the smallest searchable unit within a collection.

Params

• collectionId: string - The ID of the collection the document belongs to
• passages: CreatePassage[] - An array of objects containing parameters required to create each passage
• content (optional): The content of the passage for DEFAULT collection type
• question (optional): The question of the passage for QNA_V1 collection type
• answer (optional): The answer of the passage for QNA_V1 collection type
• documentId (string): The ID of the document the passage will be associated with
• property: value (optional): Custom properties of the passage. Note that these properties should be listed in the collection properties

Example

const collectionId = 'a1b2c3d4-uuid';
const passages: CreatePassage[] = [
  {
    content: 'This is the content of the first passage.',
    documentId: 'e5f6g7h8-uuid',
  },
  {
    content: 'This is the content of the second passage.',
    documentId: 'e5f6g7h8-uuid',
  },
];
const passage = await lookup.createPassage(collectionId, params);

// No direct response, the batch of passages are added to the collection

Note: The number of the passages are not limited but the total amount of the data to create in a single batch is limited to 1MB.

listPassages

List all passages in a collection with optional pagination, search query filtering, and search mode. This method allows you to retrieve passages from a collection and optionally filter them based on a search query, page size, and offset.

Params

• collectionId: string - The ID of the collection to retrieve passages from.
• params: FindPassages (optional) - An object containing the following optional properties:
  • query: string (optional): Search query for filtering passages.
  • from: number (optional): Pagination offset (the index of the first item on the page).
  • size: number (optional): Maximum number of passages to retrieve (page size).
  • mode: SearchMode (optional) - The search mode, either 'bm25' for full-text search or 'vector' for vector similarity search. Default is 'bm25'
  • where: Record<string, unknown> (optional): A weaviate where filter. See Weaviate Filters for more information.

Example

Listing passages with pagination, query filter, and a specific search mode:

const collectionId = 'a1b2c3d4-uuid';
const params: FindPassages = {
  query: 'sample query',
  from: 0,
  size: 10,
  mode: 'vector',
};
const passages = await lookup.listPassages(collectionId, params);

// Example response
{
  items: [
    {
      id: 'x1y2z3w4-uuid',
      content: 'This is the content of the passage.',
      // ...other properties
    },
    // ...more passages
  ],
  total: 123
}

This will return a list of passages from the specified collection that match the filter query and are ordered according to their vector similarity. It also supports a 'where' query for advanced filtering.

listPassagesInDocument

List all passages in a document with optional pagination, query, and ordering.

Params

• collectionId: string - The ID of the collection containing the document.
• documentId: string - The ID of the document to retrieve passages from.

• find: Find (optional) - Optional find parameters.

  • query: string (optional) - The search query to filter passages based on their content.
  • mode: SearchMode (optional) - The search mode, either 'bm25' for full-text search or 'vector' for vector similarity search. Default is 'bm25'
  • size: number (optional) - The number of passages to return per request. Defaults to 10.
  • from: number (optional) - The starting index for fetching the passages, used for pagination.
  • orderProperty: string (optional) - The property by which to order the passages, e.g., 'createdAt'.
  • orderDirection: OrderDirection (optional) - The direction to order the passages, either 'asc' for ascending or 'desc' for descending.

Returns

• An array of passages with pagination info.

Example

const collectionId = 'a1b2c3d4-uuid';
const documentId = 'e5f6g7h8-uuid';
const passages = await lookup.listPassagesInDocument(collectionId, documentId, {
  query: 'search query',
  mode: 'bm25',
  size: 10,
  from: 0,
  orderProperty: 'createdAt',
  orderDirection: 'asc',
});

// Example response
{
  "items": [
    {
      "id": "m3n4o5p6-uuid",
      "accountId": "i9j0k1l2-uuid",
      "collection": "a1b2c3d4-uuid",
      "documentId": "e5f6g7h8-uuid",
      "content": "This is a sample passage in the document.",
      "createdAt": "2022-01-02T03:04:05Z"
    }
  ],
  "total": 1
}

getPassage

Retrieve a single passage from a collection by its ID. The passage object will include any custom properties specified in the collection schema.

Params

• collectionId: string - The ID of the collection the passage belongs to
• passageId: string - The ID of the passage to retrieve
• options: { signal?: AbortSignal; params?: { vector?: boolean } } - Include the vector of the passage in the response. Default is false.

Example

const collectionId = 'a1b2c3d4-uuid';
const passageId = 'x1y2z3w4-uuid';
const passage = await lookup.getPassage(collectionId, passageId);

// Example response
{
  id: 'x1y2z3w4-uuid',
  content: 'This is the content of the passage.',
  // ...more properties based on the collection schema
}

const collectionId = 'a1b2c3d4-uuid';
const passageId = 'x1y2z3w4-uuid';
const passage = await lookup.getPassage(collectionId, passageId, { params: { vector: true } });

// Example response
{
  id: 'x1y2z3w4-uuid',
  content: 'This is the content of the passage.',
  vector: [0.123, 0.456, 0.789,..., 0.012],
  // ...more properties based on the collection schema
}

Returns:

A Passage object representing the retrieved passage:

• id (string): The ID of the passage
• content (string): The content of the passage
• The rest of the properties specified in the collection schema

updatePassage

Update a passage in a collection. This method allows you to modify properties of a specific passage, including both content and custom properties.

Params

• collectionId: string - The ID of the collection the passage belongs to
• passageId: string - The ID of the passage to update
• params: UpdatePassage - The update passage parameters, which can include:
• content (string): Updated content of the passage
• You can also include additional properties to update if they exist in the collection schema and have a text datatype

Example

Updating a passage with new content and a custom property

const collectionId = 'a1b2c3d4-uuid';
const passageId = 'x1y2z3w4-uuid';
const params: UpdatePassage = {
  content: 'Updated content of the passage',
  customProperty: 'Updated custom property value',
};
const updatedPassage = await lookup.updatePassage(collectionId, passageId, params);

// Example response
{
  id: 'x1y2z3w4-uuid',
  content: 'Updated content of the passage',
  customProperty: 'Updated custom property value',
  // ...more properties if available
}

Returns:

A Passage object representing the updated passage:

• id (string): The ID of the updated passage
• The updated properties, including the new content and any other properties specified in the update parameters

Note that updating text datatype properties will update the passage vector.

deletePassage

Delete a passage from a collection by its ID. This method removes a specific passage from the given collection, based on the passage's ID.

Params

• collectionId: string - The ID of the collection the passage belongs to
• passageId: string - The ID of the passage to delete

Example

In this example, we provide the collectionId and passageId as input parameters to the deletePassage method. The specified passage will be deleted from the collection, with no direct response returned.

const collectionId = 'a1b2c3d4-uuid';
const passageId = 'x1y2z3w4-uuid';
await lookup.deletePassage(collectionId, passageId);

// There's no direct response, the passage is deleted from the collection

Note: Deleting a passage is a permanent operation, and the removed data cannot be recovered. Be cautious when using this method and ensure you have backups of your data if necessary.

backupCollection

Start the backup process for an existing collection. The backup operation takes a snapshot of the current state of a collection and stores it with a given prefix for future restore.

Params

• collectionId (string): The ID of the collection to backup.
• backup (Backup): The backup parameters, which includes:
  • prefix (string): The prefix in files the backup files to put into. Needs to be between 3 and 500 characters long.

Example

const collectionId = 'a1b2c3d4-uuid';
const backup = {
  prefix: 'backup-prefix',
};

await sdk.backupCollection(collectionId, backup);

// No direct response, the collection backed up is started

restoreCollection

Start the restore process for a collection from a previous backup. The given key is used to find the backup files.

Params

• collectionId (string): The ID of the collection to restore.
• restore (Restore): The restore parameters, which include:
  • key (string): The key in files for backup files. Needs to be between 3 and 500 characters long.
  • isPublic (boolean): File privacy.

Example

const collectionId = 'e5f6g7h8-uuid';
const restore = {
  key: 'backup-prefix',
  isPublic: false,
};

await sdk.restoreCollection(collectionId, restore);
// No direct response, the collection restore up is started

loadCrawlerDocument

Start the job which will get all links matched by prefix and create a document for each link.

Params

• collectionId (string): The ID of the collection to restore.
• params (CrawlDocuments): The crawling parameters, which include:
  • url (string): The first page which will be crawled.
  • prefix (string): Define which links we should check.
  • limit (number): The max. amounts of documents which may be created.
  • defaultProperties (object): A default properties for all documents.

Example

const collectionId = 'e5f6g7h8-uuid';
const params = {
  url: 'https://tailwindcss.com/docs/installation',
  prefix: 'docs/',
  limit: 10,
  defaultProperties: {}
};

await sdk.loadCrawlerDocument(collectionId, params);
// No direct response, the crawler is started

updateManyPassages

Update properties of a list of passages

Params

• collectionId (string): The ID of the collection.
• documentId (string): The ID of the document.
• ids (string[]): List of passages ids which will be updated.
• params (object): Object with properties:
  • key (any): key value with properties to be updated.

Example

const collectionId = 'e5f6g7h8-uuid';
const documentId = 'w5f6g7h8-uuid';
const ids = ['123-uuid', '1234-uuid'];
const params = { "registered": true };

await sdk.updateManyPassages(collectionId, documentId, ids, params);
// Returns a list of updated passages

deleteManyPassages

Removes a list of passages

Params

• collectionId (string): The ID of the collection.
• documentId (string): The ID of the document.
• ids (string[]): List of passages ids which will be removed.

Example

const collectionId = 'e5f6g7h8-uuid';
const documentId = 'w5f6g7h8-uuid';
const ids = ['123-uuid', '1234-uuid'];

await sdk.deleteManyPassages(collectionId, documentId, ids);
// Returns a list of ids of removed passages

deleteManyDocuments

Removes a list of documents

Params

• collectionId (string): The ID of the collection.
• ids (string[]): List of documents ids which will be removed.

Example

const collectionId = 'e5f6g7h8-uuid';
const ids = ['123-uuid', '1234-uuid'];

await sdk.deleteManyDocuments(collectionId, ids);
// Returns ids of removed documents and ids of removed passages

getAiProviders

This method is responsible for fetching the AI models provided by different vendors. It returns a promise that resolves with an array of AI.

Usage

const providers = await sdk.getAiProviders();