Ozymandias_osiris NPM

Osiris - Document Ingestion Pipeline

Osiris is a powerful document ingestion pipeline designed to process content into vector embeddings and store them in a Qdrant vector database. It's built to work seamlessly with the Ibis chat application.

Prerequisites

Node.js (v18 or higher)
npm or yarn
A Qdrant instance (local or cloud)
OpenAI API key

Installation

Clone the repository:
```
git clone <repository-url>
cd osiris
```
Install dependencies:
```
npm install
```
Create a .env file in the root directory:

OpenAI API key for embeddings generation

OPENAI_API_KEY=your_openai_api_key

Qdrant settings

QDRANT_URL=your_qdrant_url # e.g., http://localhost:6333 or your cloud URL QDRANT_API_KEY=your_qdrant_api_key # Optional for local, required for cloud

Preparing Your Data

Create JSON files containing your documents. Each JSON file should follow this structure:

{
  "title": "Document Title",
  "content": "Document content goes here...",
  "metadata": {
    "source": "optional source information",
    "author": "optional author information",
    "date": "optional date information"
  }
}

Store your JSON files in a directory (e.g., ./data).

Usage

Build the project:
```
npm run build
```

Run the ingestion pipeline:

node dist/index.js ingest <directory> [options]

Options:

--collection - Collection name (default: 'website_content')
--batch-size - Batch size for processing (default: 100)
--max-retries - Max retries for failed operations (default: 3)
--max-concurrent - Max concurrent operations (default: 5)

Example

node dist/index.js ingest ./data --collection documents --batch-size 50

or add this in .zshrc

# Osiris data ingestion function
osiris() {
  # Show help if no arguments provided
  if [ -z "$1" ] || [ "$1" = "-h" ] || [ "$1" = "--help" ]; then
    echo "Usage: osiris <command> [options]"
    echo ""
    echo "Commands:"
    echo "  ingest <directory> -c <collection> -g <group-id>  # Ingest content"
    echo "  health                                           # Check system health"
    echo "  clean <collection>                               # Clean collection"
    echo "  delete-by-group -c <collection> -g <group-id>    # Delete by group"
    echo ""
    echo "Examples:"
    echo "  osiris ingest ./content -c my-collection -g client1"
    echo "  osiris health"
    echo "  osiris clean my-collection"
    echo "  osiris delete-by-group -c my-collection -g client1"
    return 1
  fi

  # Get the directory of the script
  local OSIRIS_PATH="/users/ivan/sites/ozymandias/osiris"

  # If the first argument looks like a path and not a command, insert 'ingest'
  if [[ "$1" != "health" && "$1" != "clean" && "$1" != "delete-by-group" ]]; then
    set -- "ingest" "$@"
  fi

  # Run the command using tsx instead of node
  cd "$OSIRIS_PATH" && npx tsx src/index.ts "$@"
}

and then run

# Ingest content
osiris ./content -c collection-name -g client1

# Check health
osiris health

# Clean collection
osiris clean my-collection

# Delete by group
osiris delete-by-group -c my-collection -g client1