@quarkworks-inc/avatar-webkit NPM

Avatar WebKit by Hallway

Avatar WebKit uses machine learning models to provide face motion capture blend shapes (aka. "morph targets" or "action units") for animation of human expressions, all in real time in the browser. This can be used in a variety of ways:

Real-time animation from webcam
Video calls, AR, VR, Metaverse apps
Recording high quality animations for movies and games
Bringing your NFT / personal avatar to life

Installation

# yarn
yarn add @quarkworks-inc/avatar-webkit

# npm
npm install @quarkworks-inc/avatar-webkit

First Steps

Get an API token
Start your predictor

import { AUPredictor } from '@quarkworks-inc/avatar-webkit'
// ...

let predictor = new AUPredictor({
  apiToken: <YOUR_API_TOKEN>,
  shouldMirrorOutput: true,
})

let stream = await navigator.mediaDevices.getUserMedia({
  audio: false,
  video: {
    width: { ideal: 640 },
    height: { ideal: 360 },
    facingMode: 'user'
  }
})

predictor.onPredict = (results => {
  console.log(results)
})

// or if you like RX
predictor.dataStream.subscribe(results => {
  console.log(results)
})

predictor.start({ stream })

Hot Reloading

To get hot reloading / npm link to work for the demo run these steps

cd ~/avatar-webkit
yarn link
cd ./demo
yarn link @quarkworks-inc/avatar-webkit
Delete node modules and yarn.lock in demo folder and in avatar-webkit folder
cd ../
yarn && yarn start
open a new tab cd ./demo
yarn && yarn dev

More Docs

https://docs.google.com/document/d/16c3qSYvMi_5l2zXdrsykb2xH6XneOqxTd2wwnEVawxY/edit#

Benchmarking

Macbook Pro

16" 2019 MBP, 2.4 GHz Intel Core i9, 16 GB RAM

Browser	Total FPS (Avg Frame)	face mesh (Avg Frame)	Dense Predictor (Avg Frame)
Chrome - (2560 × 1440)	52 FPS (19 ms)	96 FPS (9.4ms)	104 FPS (9.6 ms)
Chrome - Retina	50 FPS (20 ms)	98 FPS (10.2ms)	102 FPS (9.8 ms)
Brave - (2560 × 1440)	54 FPS (19 ms)	103 FPS (9.7ms)	107 FPS (9.3 ms)
Brave - Retina	50 FPS (20 ms)	95 FPS (10.5ms)	105 FPS (9.5 ms)
Safari - (2560 × 1440)	30 FPS (33 ms)	57 FPS (17.6 ms)	64 FPS (15.7 ms)
Safari - Retina	30 FPS (33 ms)	57 FPS (17.6 ms)	64 FPS (15.7 ms)
Firefox - (2560 × 1440)	35 FPS (29 ms)	71 FPS (14.1 ms)	63 FPS (15.9 ms)
Firefox - Retina	36 FPS (30 ms)	74 FPS (13.6ms)	61 FPS (16.4 ms)
Opera - (2560 × 1440)	48 FPS (21 ms)	91 FPS (11 ms)	100 FPS (10 ms)
Opera - Retina	45 FPS (22 ms)	85 FPS (11.8 ms)	98 FPS (10.2 ms)