@cephable/cephable-web NPM

Cephable Web SDK

The Cephable Web SDK is a powerful suite of tools designed for developers looking to integrate advanced, AI-driven capabilities into their web applications. This SDK provides services, each catering to specific functionalities such as camera control, voice commands, device profile management, and more. The SDK is exclusively available to users with a valid Cephable license, ensuring that your applications are powered by Cephable’s cutting-edge technologies.

Services

1. `CephableService`

The CephableService is the central service that integrates various Cephable services into a web application. It greatly simplifies integrations.

Key Features:

Initialize and manage all other services including voice, camera, auth and profiles.
Automatic profile switching based on the active application.
Support for both regular and guest user authentication.

Example Usage:

const cephableService = new CephableService({
    authenticationConfiguration: {
        clientId: 'your-client-id',
        clientSecret: 'your-client-secret',
        redirectUri: location.href,
        autoRefresh: true
    },
    deviceName: 'your-device-or-app-name',
    deviceTypeId: 'your-device-type-id',
    locale: 'en-US',
    includeDefaultControls: true // enabled default commands for clicking things, scrolling, zooming, typing, etc.
    customControls: [] // add custom controls if you want more than just the defaults
    enableRemoteControls: false // set to true to enable remote commands via device hub (only for authenticated users)
    deviceHubConfiguration: {} // set device hub details if using remote controls
});

// alternatively sign in a real cephable user
// pass null for voice config or camera config if you only want one input. Or none if you only want remote controls
await cephableService.initializeWithGuestUser({
    locale: 'en-US',
    onPartialResult: (result) => {
        // do something to show speech if you want
    },
    onFinalResult: (result) => {
        // do something to show speech if you want
    },

}, {
    isDrawingEnabled: true,
    videoElementId: "video",
    canvasElementId: "canvas",
    faceLinesColor: "red",
    baselineFaceLinesColor: "blue"
});

2. `VoiceService`

The VoiceService implements voice command capabilities, allowing your application to respond to spoken commands. This service integrates with Cephable’s voice recognition models for accurate and reliable performance.

Key Features:

Real-time speech recognition using the Vosk engine.
Event handling for partial and final speech recognition results.
Control over voice recognition operations, including start, pause, resume, and stop.

Example Usage:

const voiceService = new VoiceService({
    locale: "en-US",
    modelPath: "path-to-hosted-model",
    onPartialResult: (result) => {
            console.log("Partial Result: ", result);
    },
    onFinalResult: (result) => {
            console.log("Final Result: ", result);
    }, 
});

voiceService.startVoiceControls([]);

3. `CameraService`

The CameraService is designed to handle camera operations, including capturing video from a webcam and performing face detection. This service provides methods to start, stop, pause, and resume the camera feed and process the video frames to detect and analyze faces in real-time.

Key Features:

Initialize Service: Loads and caches models for offline use
Start/Stop Camera: Begins, pauses, resumes, and stops capturing video from the webcam and processes each frame to detect faces.
Face Detection: Detects faces in the video feed using the BlazeFace ONNX model.

Example Usage:

const cameraService = new CameraService({
    modelDirectoryPath: "path-to-remote-models",
    isDrawingEnabled: true,
    videoElementId: "video",
    canvasElementId: "canvas",
    faceLinesColor: "red",
    baselineFaceLinesColor: "blue",
    onGesturesRecognized: (gestures: string[]) => {
        // do something with the gestures
    },
    onFaceProcessed: (face: FaceExpression) => {
        // do something with the face data
    }
});

cameraService.startCameraControls();

Along with the javascript to initialize it, you'll also need a <video> and <canvas> element with distinct IDs to read the video frames from and to draw the frames over.

<video id="video" style="display: none"></video>
<canvas id="canvas"></canvas>

4. `DOMInteractiveService`

The DOMInteractiveService comes packed with easy to use functions for DOM interactions on behalf of the user such as scrolling, typing, navigation, focusing, and more.

Key Features:

Use tab order for navigating forward/backward through the DOM
Fuzzy search for elements on the page and click, scroll to, or focus them
Scroll to specific locations on the page

Example Usage:

const domService = new DomInteractionService();
// these are just sample values, but you Cephable will search the DOM to find an element that matches the display value
// these methods will return an IResult<HTMLElement> with an "Ok" resultType if the element is found and an "Invalid" result type if not
const scrollResult = domService.scrollToElementByDisplayValue("Button one");
const focusResult = domService.focusElementByDisplayValue("Button two");
const clickResult = domService.clickElementByDisplayValue("Button three");

5. `AuthenticationService`

The AuthenticationService manages user authentication, including OAuth flows and guest users.

Key Features

User Authentication: Supports OAuth-based authentication for registered users. Guest Authentication: Allows the creation and management of guest users.
Token Management: Manages OAuth tokens, including automatic refreshing of access tokens. Secure Storage: Stores authentication state securely using an encrypted storage provider.
Automatic Token Refreshing: Automatically refreshes tokens before they expire, ensuring continuous authentication without user intervention.

Example Usage:

    const authService = new AuthenticationService({
        clientId: "your-client-id",
        clientSecret: "your-client-secret", // you should not store this in your code
        redirectUri: location.href, // a redirect uri that can handle the auth code
        autoRefresh: true // leave on if you want Cephable SDK to keep user signed in
    })
    authService.startUserAuth(false); 
    // false value is for existing users, true is for new users. 
    // Either way a user can create a new account from sign in

    //...
    // then on a page to handle the callback from oauth:
    // if there is a code in the query string, exchange it with the auth service and remove the query string
    const urlParams = new URLSearchParams(window.location.search);
    const code = urlParams.get('code');
    const state = urlParams.get('state');
    if (code && state) {
        await authService.authenticateFromCode(code, state);
        window.history.replaceState({}, document.title, window.location.pathname);
    }

6. `DeviceProfileService`

The DeviceProfileService manages user device profiles and handles the current profile to use for controls from the CephableService. Profiles can include settings such as macros, keybindings, and application associations.

Key Features:

Manage user device profiles.
Store and retrieve profiles from secure storage.

Example Usage:

 const authService = new AuthenticationService({
    clientId: "your-client-id",
    clientSecret: "your-client-secret", // you should not store this in your code
    redirectUri: location.href, // a redirect uri that can handle the auth code
});
const profileService = new DeviceProfileService(authService);
await profileService.loadProfiles();

profileService.currentProfile = {
    name: 'test',
    configuration: {
        macros: [], // configure macros if you want
        keybindings: [],
        audioEvents: [],
        dictationCommands: ['type'],
    }
};

Getting Started

Installation

Ensure you have a valid Cephable license: Before using the SDK, you must have an active Cephable license
Install the SDK via npm: You can easily add the Cephable WebSDK to your project using the NuGet Package Manager Console:
```
npm install @cephable/cephable-web
```
Configure Your Application:

const cephableService = new CephableService({
    authenticationConfiguration: {
        clientId: 'your-client-id',
        clientSecret: 'your-client-secret',
        redirectUri: location.href,
        autoRefresh: true
    },
    deviceName: 'your-device-or-app-name',
    deviceTypeId: 'your-device-type-id',
    locale: 'en-US',
    includeDefaultControls: true, // enables default commands for clicking things, scrolling, zooming, typing, etc.
    enableAutoDomActions: true, // enables cephable to take phrases from a user and automatically search for interactable elements to focus and click on without needing a prefix command to be said like 'click' or 'focus'
    customControls: [], // add custom controls if you want more than just the defaults
    enableRemoteControls: false, // set to true to enable remote commands via device hub (only for authenticated users)
    deviceHubConfiguration: {} // set device hub details if using remote controls
});

// alternatively sign in a real cephable user
// pass null for voice config or camera config if you only want one input. Or none if you only want remote controls
await cephableService.initializeWithGuestUser({
    locale: 'en-US',
    onPartialResult: (result) => {
        // do something to show speech if you want
    },
    onFinalResult: (result) => {
        // do something to show speech if you want
    },

}, {
    isDrawingEnabled: true,
    videoElementId: "video",
    canvasElementId: "canvas",
    faceLinesColor: "red",
    baselineFaceLinesColor: "blue"
});

Telling Cephable how to interact with text inputs and voice

Cephable can default allow for the "type" command in any text field the user is focused in to type into that field. You can also add data-cephable- attributes to specific elements to inform how typing should work including:

data-cephable-no-spell: Tells Cephable not to activate spelling mode in that text field
data-cephable-sentence-case: Tells Cephable to automatically sentence case the text inputs
data-cephable-title-case: Tells Cephable to automatically capitalize the first letter of each word. Common for name fields
data-cephable-optimistic: Tells Cephable to automatically enable optimistic mode when in that text field
data-cephable-alt: String value telling Cephable other ways to search for this element when interacting
data-cephable-ignore-auto: true/false value telling Cephable to ignore that element when doing automatic dom interactions. These values will only be selectable/interactable via an explicit user action like "click, focus, tap" commands.

Start Using the Services: Once configured, you can start utilizing the various services provided by the SDK.

License

The Cephable WebSDK is only available to licensed users. Ensure that your application complies with Cephable’s licensing terms. For more details on obtaining a license, please contact Cephable Sales or online at contact cephable.com.