# @personifyinc/psyseg-sdk > The Personify Segmentation JavaScript Software Development Kit (PsySeg JS SDK) is a library that utilizes our state-of-the-art machine learning models and algorithms to perform user segmentation and apply special effects (blur and replacement) on the back Latest version **5.6.0** (published 2024-01-10) · ISC license · 0 weekly downloads ## Install ```sh npm install @personifyinc/psyseg-sdk pnpm add @personifyinc/psyseg-sdk yarn add @personifyinc/psyseg-sdk bun add @personifyinc/psyseg-sdk ``` ## Health **Score 20/100 (F)** — status: abandoned. Positive: has types; no vulnerabilities. Warnings: low downloads; no esm support. Negative: abandoned; low maintenance score. ## Facts | | | |---|---| | Version | 5.6.0 | | Published | 2024-01-10 | | First published | 2022-07-18 | | Weekly downloads | 0 | | License | ISC | | TypeScript types | bundled | | Module format | CommonJS | | Dependencies | 0 | | Unpacked size | 180.4 KB | | Known vulnerabilities | 0 | | Install scripts | no | | Author | PersonifyInc | | Maintainers | personifydev | ## Links - npm: https://www.npmjs.com/package/@personifyinc/psyseg-sdk - Repository: https://github.com/PersonifyInc/PsySegSDKJS - Homepage: https://github.com/PersonifyInc/PsySegSDKJS#readme - Issues: https://github.com/PersonifyInc/PsySegSDKJS/issues - npm.io page: https://npm.io/package/@personifyinc/psyseg-sdk ## Recent versions - 5.6.0 (latest) — 2024-01-10 - 5.5.1 — 2023-11-30 - 5.5.0 — 2023-10-18 - 5.4.0 — 2023-05-08 - 5.3.0 — 2023-01-19 - 5.2.0 — 2022-12-05 - 5.2.0-beta.1 — 2022-11-24 - 5.0.0 — 2022-10-28 - 5.0.0-beta.1 — 2022-10-17 - 4.0.0 — 2022-08-30 - 3.0.0 — 2022-07-18 ## README # PsySeg SDK for JavaScript The Personify Segmentation JavaScript Software Development Kit (PsySeg JS SDK) is a library that utilizes our state-of-the-art machine learning models and algorithms to perform user segmentation and apply special effects on the background. ## Prerequisites This project requires NodeJS and NPM. [Node](http://nodejs.org/) and [NPM](https://npmjs.org/) are really easy to install. To make sure you have them available on your machine, try running the following command. ```sh $ npm -v && node -v 6.4.1 v8.16.0 ``` ## Getting Started ## Installation To install and set up the latest sdk, run: ```sh $ npm install @personifyinc/psyseg-sdk ``` To install and set up specific sdk version, run: ```sh $ npm install @personifyinc/psyseg-sdk@x.x.x ``` ## Usage To use the SDK interfaces, simply import @personifyinc/psyseg-sdk as you normally would. ```javascript import PsySeg, { IPsySegResponses, IPsySegEffectParams, IPsySegProps } from '@personifyinc/psyseg-sdk'; ``` To use the SDK class, simply import @personifyinc/psyseg-sdk as you normally would. ```javascript import PsySeg from '@personifyinc/psyseg-sdk'; const psySegInstance = new PsySeg() ``` ## Interfaces IPsySegResponses ```typescript export interface IPsySegResponses { status: boolean licenseControl?: string data?: ImageData modelSize?: { width: number height: number } } ``` IPsySegEffectParams ```typescript export interface IPsySegEffectParams { mode?: string erode?: number enhanceFrame?: boolean gamma?: number isSkipAlpha?: boolean blur?: number dstSize?: { width: number, height: number } overlaySize?: number overlayPosition?: number shape?: string isRemoveBlob?: boolean } ``` IPsySegProps ```typescript export interface IPsySegProps { psySegWasmURL?: string tfliteWasmURL?: string tfliteSimdWasmURL?: string segmentationModelURL?: string license?: string accessToken?: string accessKey?: string } ``` ## APIs - [version](#version) - [init](#init) - [replaceBackground](#replacebackground) - [blurBackground](#blurbackground) - [removeBackground](#removebackground) - [runAlpha](#runalpha) - [toMask](#tomask) - [shapeForeground](#shapeForeground) - [maskShapeBackground](#maskShapeBackground) ### version Get the current version of the installed SDK. ```javascript const version = psySegInstance.version() //=> v3.0.0 ``` ### init **It is required** because it initializes the wasm file and validates the license. ```javascript const psySegProps: IPsySegProps = { psySegWasmURL: '', tfliteWasmURL: '', tfliteSimdWasmURL: '', segmentationModelURL: '', license: '', accessToken: '', accessKey: '' } await psySegInstance.init(psySegProps) ``` | Attribute | Type | Required | Default | Description | |----------------------|--------|----------|---------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | psySegWasmURL | string | no | empty | Where to host the wasm file, if this attribute empty it should be hosted from our CDN `https://psyjs-cdn.personifyinc.com/core/wasm/psyseg-x.x.x.wasm` | | tfliteWasmURL | string | no | empty | Where to host the wasm file, if this attribute empty it should be hosted from our CDN `https://psyjs-cdn.personifyinc.com/core/tflite/wasm/tflite-20221011.wasm` | | tfliteSimdWasmURL | string | no | empty | Where to host the wasm file, if this attribute empty it should be hosted from our CDN `https://psyjs-cdn.personifyinc.com/core/tflite/wasm/tflite-simd-20221011.wasm` | | segmentationModelURL | string | no | empty | Where to host the model tflite file, if this attribute empty it should be hosted from our CDN `https://psyjs-cdn.personifyinc.com/core/tflite/model/psysegmentation-112x192-20230419_e36.asset` | | license | string | no | empty | For getting more information, please contact [PersonifyInc](https://personifyinc.com/contact) | | accessToken | string | no | empty | For getting more information, please contact [PersonifyInc](https://personifyinc.com/contact) | | accessKey | string | no | empty | For getting more information, please contact [PersonifyInc](https://personifyinc.com/contact) | ### replaceBackground Extract user from background and replace it with desired ones. ```javascript const inputColor: ImageData = inputImageData const inputBackground: ImageData = backgroundImageData const effectParams: IPsySegEffectParams = { mode: 'full', erode: 1, enhanceFrame: false, gamma: 0, isSkipAlpha: false, overlaySize: 1, overlayPosition: 0.5, isRemoveBlob: true, } const result: IPsySegResponses = await psySegInstance.replaceBackground( inputColor, inputBackground, effectParams ) ``` | Attribute | Type | Required | Default | Description | |------------------------------|---------------------|----------|---------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | inputColor | ImageData | yes | | The ImageData of Video Frame or Image Element | | inputBackground | ImageData | yes | | The ImageData of Image Element | | effectParams | IPsySegEffectParams | no | {} | The advanced configuration | | effectParams.mode | string | no | full | Setting mode for PsySeg to reach the best performance or quality based on different purposes of customer.By default, full mode is selected. This option provides the best quality, but it has a tradeoff for performance | | effectParams.erode | number | no | 1 | Erode level for edge smoothing | | effectParams.enhanceFrame | boolean | no | false | Enable lighting feature | | effectParams.gamma | number | no | 0 | Gamma value for lighting feature | | effectParams.isSkipAlpha | boolean | no | false | Enable skip alpha feature | | effectParams.overlaySize | number | no | 1 | Overlay size of user extraction, 0 < overlaySize <= 1, step is 0.1 | | effectParams.overlayPosition | number | no | 0.5 | Overlay position from left to right for user extraction, 0 < overlayPosition <= 1, step is 0.1 | | effectParams.isRemoveBlob | boolean | no | true | Remove the blobs surrounding the user | ### blurBackground Extracting user from background and making a background blurred to hidden unnecessary object. ```javascript const inputColor: ImageData = inputImageData const effectParams: IPsySegEffectParams = { mode: 'full', erode: 1, enhanceFrame: false, gamma: 0, isSkipAlpha: false, blur: 10, } const result: IPsySegResponses = await psySegInstance.blurBackground( inputColor, effectParams ) ``` | Attribute | Type | Required | Default | Description | |---------------------------|---------------------|----------|---------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | inputColor | ImageData | yes | | The ImageData of Video Frame or Image Element | | effectParams | IPsySegEffectParams | no | {} | The advanced configuration | | effectParams.mode | string | no | full | Setting mode for PsySeg to reach the best performance or quality based on different purposes of customer.By default, full mode is selected. This option provides the best quality, but it has a tradeoff for performance | | effectParams.erode | number | no | 1 | Erode level for edge smoothing | | effectParams.enhanceFrame | boolean | no | false | Enable lighting feature | | effectParams.gamma | number | no | 0 | Gamma value for lighting feature | | effectParams.isSkipAlpha | boolean | no | false | Enable skip alpha feature | | effectParams.blur | number | no | 10 | Blur level | ### removeBackground Extract only user from background. ```javascript const inputColor: ImageData = inputImageData const effectParams: IPsySegEffectParams = { mode: 'full', erode: 1, enhanceFrame: false, gamma: 0, isSkipAlpha: false, } const result: IPsySegResponses = await psySegInstance.removeBackground( inputColor, effectParams ) ``` | Attribute | Type | Required | Default | Description | |---------------------------|---------------------|----------|---------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | inputColor | ImageData | yes | | The ImageData of Video Frame or Image Element | | effectParams | IPsySegEffectParams | no | {} | The advanced configuration | | effectParams.mode | string | no | full | Setting mode for PsySeg to reach the best performance or quality based on different purposes of customer.By default, full mode is selected. This option provides the best quality, but it has a tradeoff for performance | | effectParams.erode | number | no | 1 | Erode level for edge smoothing | | effectParams.enhanceFrame | boolean | no | false | Enable lighting feature | | effectParams.gamma | number | no | 0 | Gamma value for lighting feature | | effectParams.isSkipAlpha | boolean | no | false | Enable skip alpha feature | ### runAlpha Extract alpha mask that separate persona(transparent) and background(black). ```javascript const inputColor: ImageData = inputImageData const effectParams: IPsySegEffectParams = { isSkipAlpha: false, } const result: IPsySegResponses = await psySegInstance.runAlpha( inputColor, effectParams ) ``` | Attribute | Type | Required | Default | Description | |--------------------------|---------------------|----------|---------|-----------------------------------------------| | inputColor | ImageData | yes | | The ImageData of Video Frame or Image Element | | effectParams | IPsySegEffectParams | no | {} | The advanced configuration | | effectParams.isSkipAlpha | boolean | no | false | Enable skip alpha feature | ### toMask Extract alpha mask that separate persona(white) and background(transparent). ```javascript const inputColor: ImageData = inputImageData const effectParams: IPsySegEffectParams = { isSkipAlpha: false, dstSize: { width: 320, height: 240, }, erode: 1, } const result: IPsySegResponses = await psySegInstance.toMask( inputColor, effectParams ) ``` | Attribute | Type | Required | Default | Description | |--------------------------|---------------------|----------|----------------------------|-----------------------------------------------| | inputColor | ImageData | yes | | The ImageData of Video Frame or Image Element | | effectParams | IPsySegEffectParams | no | {} | The advanced configuration | | effectParams.isSkipAlpha | boolean | no | false | Enable skip alpha feature | | effectParams.dstSize | object | no | { width: 160, height: 96 } | Destination size of output | | effectParams.erode | number | no | 1 | Erode level for edge smoothing | ### shapeForeground Ability to present user in a circle/square shape and merge into a background image ```javascript const inputColor: ImageData = inputImageData const inputBackground: ImageData = backgroundImageData const effectParams: IPsySegEffectParams = { shape: 'circle' } const result: IPsySegResponses = await psySegInstance.shapeForeground( inputColor, inputBackground, effectParams ) ``` | Attribute | Type | Required | Default | Description | |--------------------|---------------------|----------|---------|----------------------------------------------------------------| | inputColor | ImageData | yes | | The ImageData of Video Frame or Image Element | | inputBackground | ImageData | yes | | The ImageData of Image Element | | effectParams | IPsySegEffectParams | no | {} | The advanced configuration | | effectParams.shape | string | no | circle | Present user in a border shape, it should be "circle"/"square" | ### maskShapeBackground Make a mask has circle/square shape into center of background image ```javascript const inputBackground: ImageData = backgroundImageData const effectParams: IPsySegEffectParams = { shape: 'circle' } const result: IPsySegResponses = await psySegInstance.maskShapeBackground( inputBackground, effectParams ) ``` | Attribute | Type | Required | Default | Description | |--------------------|---------------------|----------|---------|-------------------------------------------------------------| | inputBackground | ImageData | yes | | The ImageData of Image Element | | effectParams | IPsySegEffectParams | no | {} | The advanced configuration | | effectParams.shape | string | no | circle | Make mask in a border shape, it should be "circle"/"square" | ## Examples Create Image from URL ```typescript export const createImage = async ( imgSrc: string, resolution: _IResolution ): Promise => { return new Promise(resolve => { const img = new Image(resolution.width, resolution.height) img.crossOrigin = 'Anonymous' img.onload = () => { resolve(img) } img.src = imgSrc }) } ``` Get ImageData from Video Frame or Image ```typescript export const getImageData = ( sourceInput: HTMLVideoElement | HTMLImageElement | null, resolution: _IResolution ) => { const inputCanvas = document.createElement('canvas') inputCanvas.width = resolution.width inputCanvas.height = resolution.height const canvasContext = inputCanvas.getContext('2d') if (canvasContext && sourceInput) { canvasContext.drawImage( sourceInput, 0, 0, resolution.width, resolution.height ) return canvasContext.getImageData(0, 0, resolution.width, resolution.height) } } ``` Render the Image Data into the Canvas Output ```typescript export const drawCanvas = ( outputBuf: ImageData | undefined, ctxOutput: CanvasRenderingContext2D, options: _IDrawnCanvasOptions ) => { if (outputBuf && ctxOutput) { const {canvasWidth, canvasHeight} = options ctxOutput.clearRect(0, 0, canvasWidth, canvasHeight) ctxOutput.putImageData(outputBuf, 0, 0) } } ``` How to use PsySeg SDK ```typescript // Initialize PsySeg SDK instance const psySegInstance = new PsySeg() console.info('PsySeg Version: ', psySegInstance.version()) const psySegInstanceInit = await psySegInstance.init({ license, accessToken, accessKey, }) console.info('PsySeg Initiation: ', psySegInstanceInit) // Get Background Image Data const imageBackground = await createImage('https://psyjs-cdn.personifyinc.com/assets/psyseg2-js-demo-background.png', { width: 320, height: 240, }) const inputBackground = getImageData(imageBackground, { width: 320, height: 240, }) as ImageData // Get Video Element & Canvas Element const videoElement = document.getElementById('#video') const canvasOutput = document.getElementById('#canvas') canvasOutput.width = 320 canvasOutput.height = 240 const ctxOutput = canvasOutput.getContext( '2d' ) as CanvasRenderingContext2D // Running function const runEffect = async (): Promise => { try { const inputColor = getImageData( videoElement as HTMLVideoElement, {width: 320, height: 240} ) as ImageData const output = await psySegInstance.replaceBackground( inputColor, inputBackground as ImageData, { mode: 'full', erode: 1, enhanceFrame: false, gamma: 0, isSkipAlpha: false, overlaySize: 1, overlayPosition: 0.5, } ) if (output?.status) { drawCanvas( output.data, ctxOutput, { canvasWidth: 320, canvasHeight: 240, } ) } // Timeout will affect to the FPS and CPU usage const timeout = 1000 / 30 // Recursive call setTimeout(runEffect, timeout) } catch (e) { console.error(e) } } runEffect() ``` **_Notes: For Real-Time Messaging Protocol (RTMP), we should use insertable streams instead of setTimeout or requestAnimationFrame to keep web application working in inactive mode. Please take a look at https://developer.chrome.com/articles/mediastreamtrack-insertable-media-processing/ and https://mediastreamtrack.glitch.me/_** ## Demo Please take a look our [demo](https://jsdemo.personifyinc.com/segmentation/) ## Contact For getting more information, please contact [PersonifyInc](https://personifyinc.com/contact) ## Changelog Release npm package @personifyinc/psyseg-sdk - v5.6.0 - Main Features: - Add a parameter to toggle the removal of the box blob - Ensure license validation prior to invoking APIs - Add an erosion value to the toMask api for the ability to adjust the edges of the body Release npm package @personifyinc/psyseg-sdk - v5.5.1 - Bug Fixes: - Fix issue: `module parse failed: Unexpected character '#'` while integrating with react-script 4 Release npm package @personifyinc/psyseg-sdk - v5.5.0 - Main Features: - Present user in a circle/square shape and merge into a background image - Make a mask has circle/square shape into center of background image Release npm package @personifyinc/psyseg-sdk - v5.4.0 - Main Features: - Reducing the SDK startup time - Update api `replaceBackground` to scale and move UE - Improve Psyseg model - Return the model size for api init() Release npm package @personifyinc/psyseg-sdk - v5.3.0 - Main Features: - Re-provide the api runAlpha Release npm package @personifyinc/psyseg-sdk - v5.2.0 - Main Features: - Apply license validation to the api toMask - Remove online license validation - Remove api runAlpha - Reduce CPU and memory usage Release npm package @personifyinc/psyseg-sdk - v5.0.0 - Main Features: - Separate the tflite model from the wasm file to reduce capacity and easily develop other models with different sizes in the future - Update the tflite model to improve segmentation and reduce CPU & Memory usage - Remove unnecessary logs Release npm package @personifyinc/psyseg-sdk - v4.0.0 - Main Features: - Implement API toMask to extract alpha mask that separate persona(white) and background(transparent). - Use reject statement instead of resolve statement for catching error - Update API version() to return value v4.0.0 Release npm package @personifyinc/psyseg-sdk - v3.0.0 - Main Features: - Restructure the entire SDK to make it simpler and easier for developers to integrate. - Add an init() method to wait for the SDK to initialize. - Remove some useless packages to reduce the size of SDK. - Remove some redundant code as well as some unnecessary processing to reduce CPU usage. - Bug Fixes: - Fix issue: Bypass license control by blocking the api checkLicense --- _Source: https://npm.io/package/@personifyinc/psyseg-sdk · Machine-readable twin of the npm.io package page. Health data is recomputed on every publish._