@e-llm-studio/streaming-response NPM

eLLM Studio Chat Stream Package

Welcome to the eLLM Studio Chat Stream package! 🎉 This package enables streaming chat functionality with your AI assistant in eLLM Studio via WebSocket and GraphQL. It's designed for both frontend and backend implementations.

🚀 Features

Real-time Streaming: Receive messages in a streaming fashion.
AI Assistant Integration: Seamlessly connects with the AI models deployed within your organization.
Customizable: Pass custom prompts or previous messages to enhance conversations.
Error Handling: Catch and manage errors using callbacks.
Image Upload: Upload images to the server.
Image Deletion: Delete previously uploaded images from the server.

📦 Installation

npm i @e-llm-studio/streaming-response

🛠️ Streaming Usage

Here’s how you can use the startChatStream function to set up the AI chat stream:

import { startChatStream } from '@e-llm-studio/streaming-response';

const params = {
    WEBSOCKET_URL: 'wss://your-ellm-deployment/graphql', // Required: WebSocket URL for your deployment
    organizationName: 'TechOrg',                         // Required: Organization name where the assistant is created
    chatAssistantName: 'MyAssistant',                    // Required: Assistant name
    selectedAIModel: 'gpt-4',                            // Required: AI model selected for the assistant
    replyMessage: '',                                    // Optional: Pass the previous response from AI.
    userName: 'John Doe',                                // Required: Username of the person using the assistant
    userEmailId: 'john@example.com',                     // Required: User's email
    userId: 'user-123',                                  // Required: Unique identifier for the user
    query: 'What is the weather like today?',            // Required: The user's question or prompt
    requestId: `requestId-${uuid()}`,                    // Required: Unique ID for the request
    customPrompt: 'Personalized prompt here',            // Optional: Add custom context to your prompt
    enableForBackend: false,                             // Optional: Set to true if you're using it on the backend (Node.js)
    images: { 'image1': 'ImageName1.jpg', 'image2': 'ImageName2.jpg' } // Optional: Pass image data if relevant.
    guestSessionId: `organizationName-chatAssistantName-${uuid()}` // Optional: Custom unique id for storing chat history
    chatPreviewId:  `${uuid()}`                          //Optional: Custom unique Id for tracking chat sessions
    aiModel: encodeParam(selectedAIModel),
    isDocumentRetrieval:true,                            // To bypass the document retrieval.
    onStreamEvent: (data) => console.log('Stream event:', data),  // Required: Callback for handling stream data
    onStreamEnd: (data) => console.log('Stream ended:', data),    // Optional: Callback for when the stream ends
    onError: (error) => console.error('Stream error:', error),    // Optional: Callback for handling errors
    customMetaData: ["test1", "test2"] || {""}           // Optional: Add custom metadata for tracking or context purposes
    multiAgentToggle: boolean                            // Optional: Toggle key for orchestrator by pass
    selectedAgent: String                                // Optional: selected agent for handling the query 
    runtimeAgents:[JSON]                                 //Optional: Agents array which can be utilized at runtime
};

startChatStream(params);

🔑 Parameters

Required Parameters:

WEBSOCKET_URL: WebSocket URL of your eLLM deployment. Example: wss://dev-egpt.techo.camp/graphql.
organizationName: Name of the organization where the assistant is created.
chatAssistantName: Name of the assistant you're interacting with.
selectedAIModel: The AI model used (e.g., GPT-4).
userName: The name of the user interacting with the assistant.
userEmailId: Email ID of the user.
userId: Unique user ID.
query: The question or prompt you want to send to AI.
requestId: Unique request ID, e.g., requestId-${uuid()}.
onStreamEvent: Callback function to capture incoming stream events.

Optional Parameters:

replyMessage: If you want to include a previous response with the new query, pass it here. Leave empty for normal chat scenarios.
customPrompt: Use this to add additional context to the prompt sent to the AI.
images: Pass image data if relevant.
isDocumentRetrieval: To bypass the document retrieval.
Annotations & Citations: Dynamically display annotations with tooltip-based citations.
enableForBackend: Set to true if you're using this package in backend e.g. NodeJs. Defaults to false, which is suitable for frontend use e.g. React/Next.js.
onStreamEnd: Callback for when the stream ends. Useful for handling final events or cleanup.
onError: Callback for capturing any errors during the stream.
guestSessionId: Custom unique id for storing chat history
chatPreviewId: Custom unique Id for tracking chat sessions
runtimeAgents: Agents array which needs to be utilized at runtime

How to Use Specific Agents on Runtime

When calling the startChatStream function, one can include a runtimeAgents parameter in the params object:

import { startChatStream } from '@e-llm-studio/streaming-response';


// Include runtime agents in the params
const params = {
  WEBSOCKET_URL: `${process.env.NEXT_PUBLIC_WEBSOCKET_URL}`,
  organizationName: 'yourOrganization',
  chatAssistantName: chatAssistantName,
  selectedAIModel: 'bestai',
  // ... other parameters ...
  runtimeAgents: runtimeAgents,
  onStreamEvent: (data) => {
    // Handle streaming data
  },
  onStreamEnd: () => setIsStreaming(false),
  onError: (error) => console.error('Stream error:', error),
};

// Start the streaming process
startChatStream(params);

Runtime Agents Structure

The runtimeAgents parameter should be an Agents array which can be utilized at runtime

_id: A unique identifier for the agent
title: The name/title of the agent

Example:

const runtimeAgents = [
  { _id: 'ObjectId', title: 'RAGAgent' }, 
  { _id: 'ObjectId2', title: 'SQLAgent' }
];

🛠️ Stop Streaming

The stopStreaming function allows you to gracefully stop an ongoing chat stream and clean up WebSocket connections.

Function Signature

export const stopStreaming = async (
  reqid: string,
  WEBSOCKET_URL: string,
  enableForBackend?: boolean
): Promise<{ success: boolean, message?: string, error?: any }>;

Parameters

reqid (string): The request ID used in startChatStream
WEBSOCKET_URL (string): The WebSocket URL of your eLLM deployment
enableForBackend (boolean, optional): Set to true when using in Node.js environment (default: false)

Return Value

Returns a Promise that resolves to an object with:

success (boolean): Indicates whether the operation was successful
message (string, optional): Success or error message
error (any, optional): Error details if operation fails

Example Usage

import { startChatStream, stopStreaming } from '@e-llm-studio/streaming-response';

const handleChat = async () => {
  const requestId = `request-${uuid()}`;
  
  // Start chat stream
  startChatStream({
    WEBSOCKET_URL: 'wss://your-ellm-deployment/graphql',
    requestId,
    // ... other required params
  });

  // Stop chat stream when needed
  const handleStop = async () => {
    try {
      const result = await stopStreaming(
        requestId,
        'wss://your-ellm-deployment/graphql',
        false 
      );
      
      if (result.success) {
        console.log("Stream stopped successfully:", result.message);
      } else {
        console.error("Failed to stop stream:", result.error);
      }
    } catch (error) {
      console.error("Error stopping stream:", error);
    }
  };
};

Important Notes

Connection Cleanup: The function automatically cleans up both the original streaming connection and the stop request connection.
Timeout: Includes a built-in timeout (1 second) to prevent hanging connections.
Backend Usage: Set enableForBackend: true when using in Node.js environment.
Error Handling: Properly handles and reports errors during the stopping process.
WebSocket Management: Ensures all WebSocket connections are properly disposed.

Best Practices

Always store the requestId when starting a stream to use it for stopping.
Use the same WEBSOCKET_URL for both starting and stopping.
Handle errors appropriately in your application.
Clean up any UI states after stopping the stream.
Implement proper error boundaries in your React components.

🛠️ Image Upload

You can upload an image to the server using the uploadImage function:

import { uploadImage } from "./index";

const imageFile = document.querySelector('input[type="file"]').files[0];

uploadImage(baseURL, imageFile)
  .then((message) => console.log(message))
  .catch((error) => console.error(error));

🛠️ Image Deletion

To delete an image from the server, use the deleteImage function:

import { baseURL, name, deleteImage } from "./index";

deleteImage("User Name", "image.jpg")
  .then((message) => console.log(message))
  .catch((error) => console.error(error));

🛠️ Consultative Mode

The Consultative Mode allows you to toggle a specialized conversation style with the AI assistant. This mode ensures the AI follows a specific set of guidelines to better understand user requirements, ask clarifying questions, and provide tailored responses.

Default Prompt: If no custom prompt is provided, the following default prompt is used:

"You are responsible for understanding the user’s needs by making sure that you gather enough relevant information before responding to achieve the goal described above. To achieve this, you will always ask clarifying follow-up questions, making sure to ask only one question at a time to maintain a focused and organized conversation. The follow-up questions should be short and crisp within 2 lines only. You should always follow a CARE framework which is about Curious:- being curious and interested to know about user needs, Acknowledge the responses given by the user and Respond with Empathy. Do not ask too many unnecessary question

🛠️ Delete Message

This API allows you to delete a specific message from a chat.

📌 Endpoint DELETE /api/chat/delete-message

🔹 Request Headers Header Type Description Authorization Bearer Token Required authentication token

🛠️ Usage:

Example:

import { ChatbotService } from "@e-llm-studio/streaming-response";

const chatbotService = new ChatbotService(
  "https://your-api-url.com", // API base URL
  "username", // Your username
  "password" // Your password
);

const toggleConsultativeMode = async () => {
  try {
    const chatBotId = "chatbot-id-123"; // Chatbot ID
    const assistantName = "MyAssistant"; // Assistant Name
    const organizationName = "TechOrg"; // Organization Name
    const currentMode = false; // Current consultative mode state
    const isFetch = false; // Whether to fetch or toggle
    const customPrompt = "Custom consultative mode prompt here"; // Optional

    const isModeEnabled = await chatbotService.toggleConsultativeMode(
      chatBotId,
      customPrompt,
      assistantName,
      organizationName,
      isFetch,
      currentMode
    );

    console.log("Consultative Mode:", isModeEnabled ? "Enabled" : "Disabled");
  } catch (error) {
    console.error("Error toggling consultative mode:", error);
  }
};

toggleConsultativeMode();

# Parallel Tool Streaming Response Handling

getAssistantObj(data, accordianMessageArray)
Purpose
The getAssistantObj function processes streaming responses from the orchestrator and agents. It dynamically manages messages in an accordion-style structure, supporting:

Orchestrator Messages:
Handles normal messages, thinking breaks, and final responses.
Agent Messages:
Supports parallel tool responses.
Manages explainability IDs for tracking responses.
Parameters
Parameter	Type	Description
data	any	Incoming streaming response data.
accordianMessageArray	any	Array to store processed messages for UI rendering.
Functionality Breakdown
Orchestrator Response Handling

Direct messages are added normally.
Messages with thinking breaks are stored in accordianMessageArray with finalResponseStatus: false.
Final orchestrator messages are processed separately.
Agent Response Handling

Messages are collapsed by default.
Supports explainability tracking using explainabilityMap to ensure responses are updated in the right place.
If a message corresponds to an existing explainability ID, it updates the previous entry.
Otherwise, it appends a new response.
Explainability Tracking

Uses a Map<string, { responded_by: string; index: number }> to track agent responses.
Ensures correct messages are updated instead of duplicated.
import { getAssistantObj } from "@e-llm-studio/llmstudioui";

const data = {
  chatStream: {
    continueChatResult: {
      message: { content: "Hello from AI" },
      is_orchestrator: true,
      isEnd: false,
      thinking_break: true,
      responded_by: "AI Assistant",
      header_message: "Processing...",
      explainability_id: "12345",
    },
  },
};

const accordianMessages = [];
const response = getAssistantObj(data, accordianMessages);

console.log(response);

 const  orchestratorAndAgentsMessages = [
    {
      "responsedata": "Hello from AI",
      "responseFrom": "AI Assistant",
      "header_message": "Processing...",
      "finalResponseStatus": false,
      "explainability_id": "12345",
      "isParallelToolResponse": false
    }
  ]

🛠️ Get Agent Settings

This function allows to retrieve agent settings for a given assistant on the basis of assistant name and organization name. Assuming you have intiaized ChatbotService, you can directly use the function from the same class.

🛠️ Usage:

Example:

import { ChatbotService } from "@e-llm-studio/streaming-response";

const chatbotService = new ChatbotService(
  "https://your-api-url.com", // API base URL
  "username", // Your username
  "password" // Your password
);

const getAgentSettings = async () => {
  try {
    const assistantName='assistantName';
    const organizationName='techolution'

    const agentSettings = await chatbotService.getAgentSettings(assistantName, organizationName);

    console.log("Agent settings:", agentSettings);
  } catch (error) {
    console.error("Error in retrieving agent settings:", error);
  }
};

🛠️ Get Models

This function allows to retrieve models supported by LLM Studio. The response would be a api response with message field and other fields which would contain models in an array

🛠️ Usage:

Example:

import { ChatbotService } from "@e-llm-studio/streaming-response";

const chatbotService = new ChatbotService(
  "https://your-api-url.com", // API base URL
  "username", // Your username
  "password" // Your password
);

const getModels = async () => {
  try {
    const models = await chatbotService.getModels();
    console.log("Models: ", models);
  } catch (error) {
    console.error("Error in retrieving models:", error);
  }
};

🧠 getChatbotData

This function fetches detailed chatbot configuration and metadata for a given assistant in a specified organization from the LLM Studio backend. It returns a full chatbot configuration object including settings, capabilities, and preferences associated with the assistant.

🛠️ Usage:

Example:

import { ChatbotService } from "@e-llm-studio/streaming-response";

const chatbotService = new ChatbotService(
  "https://your-api-url.com", // API base URL
  "username", // Your username
  "password" // Your password
);

import { ChatbotService } from "@e-llm-studio/streaming-response";

const chatbotService = new ChatbotService(
  "https://your-api-url.com", // API base URL
  "username", // Your username
  "password" // Your password
);

const getChatbotConfig = async () => {
  try {
    const chatbotData = await chatbotService.getChatbotData("assistantName", "organizationName");
    console.log("Chatbot Configuration: ", chatbotData);
  } catch (error) {
    console.error("Error in retrieving chatbot data:", error);
  }
};
 📌 Parameters
| Name               | Type     | Description                        |
| ------------------ | -------- | ---------------------------------- |
| `assistantName`    | `string` | The name of the assistant chatbot. |
| `organizationName` | `string` | The name of the organization.      |

📦 Response

Returns a JSON object containing the chatbot’s configuration and metadata including model, voice agent settings, transcription preferences, etc.

🔐 getAccessToken

This function performs user authentication by sending login credentials to the LLM Studio backend. Upon successful authentication, it retrieves and stores a bearer access token for future API requests.

🛠️ Usage

import { ChatbotService } from "@e-llm-studio/streaming-response";

const chatbotService = new ChatbotService(
  "https://your-api-url.com", // API base URL
  "username", // Your username
  "password" // Your password
);

const login = async () => {
  try {
    const accessToken = await chatbotService.getAccessToken("your-username", "your-password");
    console.log("Access Token: ", accessToken);
  } catch (error) {
    console.error("Login failed:", error);
  }
};