@convai/web-sdk

@convai/web-sdk

Real-time conversational AI characters for the web.

TypeScript-first SDK for embedding Convai AI characters into React and vanilla JS applications. Voice, text, lipsync, emotions, video, and screen share — all in one package.

---

Features

• React & vanilla JS — useConvaiClient hook, ConvaiWidget, and a framework-agnostic core
• Real-time audio/video — full-duplex WebRTC with echo cancellation, camera, and screen share
• Lipsync — ARKit and MetaHuman blendshape streams for facial animation
• Emotions — per-turn emotion detection with intensity scale
• Dynamic context — inject game state, user preferences, or scene data mid-session
• Long-term memory — persistent cross-session memories scoped to each end user
• File upload — send images to the character during a live session
• WebSocket transport — opt-in alternative to WebRTC for constrained networks
• Auth tokens — server-side token exchange for production deployments

---

Installation

npm install @convai/web-sdk
# or
pnpm add @convai/web-sdk
# or
yarn add @convai/web-sdk

React peer dependencies: react and react-dom ^18 || ^19

Runtime requirement: secure context (https:// or http://localhost) for microphone/camera access.

---

Quick start

React

import { useConvaiClient, ConvaiWidget } from '@convai/web-sdk';

export function App() {
  const client = useConvaiClient({
    apiKey: import.meta.env.VITE_CONVAI_API_KEY,
    characterId: import.meta.env.VITE_CONVAI_CHARACTER_ID,
  });

  return <ConvaiWidget convaiClient={client} />;
}

Vanilla TypeScript

import { ConvaiClient, createConvaiWidget } from '@convai/web-sdk/vanilla';

const client = new ConvaiClient({
  apiKey: import.meta.env.VITE_CONVAI_API_KEY,
  characterId: import.meta.env.VITE_CONVAI_CHARACTER_ID,
});

const widget = createConvaiWidget(document.body, { convaiClient: client });

window.addEventListener('beforeunload', () => {
  widget.destroy();
  void client.disconnect();
});

---

Documentation

Full documentation is at docs.convai.com.

Guide	Description
Quick Start	First working integration in under 5 minutes
Configuration	All ConvaiConfig options
React Integration	useConvaiClient, ConvaiWidget, and React-specific patterns
Vanilla JS	ConvaiClient, createConvaiWidget, and audio setup
Events	Full event reference — botReady, stateChange, messagesChange, interactionCreated, and more
Context Management	Dynamic context, updateContext, file upload, session management
Emotions	Per-turn emotion detection with provider options
Lipsync	ARKit / MetaHuman blendshape streams and BlendshapeQueue API
Actions	Trigger character behaviors and scene actions
Memory	Long-term memory scoped to end users
Audio & Video	Microphone, camera, screen share controls
Error Handling	error, disconnect, serverResponse, retry patterns
Auth Tokens	Server-side token exchange for production
WebSocket Transport	Alternative transport for WebRTC-constrained environments

---

Package entry points

Import path	Contents
@convai/web-sdk	React hooks, components, and re-exported core types
@convai/web-sdk/react	Same as default (React-explicit alias)
@convai/web-sdk/vanilla	ConvaiClient, createConvaiWidget, AudioRenderer
@convai/web-sdk/core	Framework-agnostic ConvaiClient, managers, and all types
@convai/web-sdk/lipsync-helpers	Blendshape format utilities and queue helpers
@convai/web-sdk/vanilla/websocket	Opt-in. Registers the WebSocket transport. Import alongside /vanilla when using transport: "websocket".

---

WebSocket transport

The default transport is WebRTC (LiveKit). A WebSocket-based transport is available for environments where WebRTC is unavailable or undesirable.

// Add this import once (e.g. in your app entry file)
import '@convai/web-sdk/vanilla/websocket';
import { ConvaiClient } from '@convai/web-sdk/vanilla';

const client = new ConvaiClient({
  apiKey: '...',
  characterId: '...',
  transport: 'websocket',
});

The WebSocket packages (@pipecat-ai/client-js, @pipecat-ai/websocket-transport) are excluded from your bundle unless you import the /vanilla/websocket subpath — so LiveKit-only apps pay no bundle cost.

See the WebSocket Transport guide for the full feature comparison.

---

Example app

A full React Three Fiber integration demo is in examples/react-three-fiber. It demonstrates lipsync on a 3D character, emotion display, dynamic context, and file upload.

---

License

Licensed under the Apache License 2.0. Copyright 2025 Convai.