@quarkworks-inc/avatar-webkit

Avatar WebKit by Hallway

Avatar WebKit uses machine learning models to provide face motion capture blend shapes (aka. "morph targets" or "action units") for animation of human expressions, all in real time in the browser. This can be used in a variety of ways:

• Real-time animation from webcam
• Video calls, AR, VR, Metaverse apps
• Recording high quality animations for movies and games
• Bringing your NFT / personal avatar to life

Installation

# yarn
yarn add @quarkworks-inc/avatar-webkit

# npm
npm install @quarkworks-inc/avatar-webkit

First Steps

1. Get an API token

2. Start your predictor

import { AUPredictor } from '@quarkworks-inc/avatar-webkit'
// ...

let predictor = new AUPredictor({
  apiToken: <YOUR_API_TOKEN>,
  shouldMirrorOutput: true,
})

let stream = await navigator.mediaDevices.getUserMedia({
  audio: false,
  video: {
    width: { ideal: 640 },
    height: { ideal: 360 },
    facingMode: 'user'
  }
})

predictor.onPredict = (results => {
  console.log(results)
})

// or if you like RX
predictor.dataStream.subscribe(results => {
  console.log(results)
})

predictor.start({ stream })

Hot Reloading

To get hot reloading / npm link to work for the demo run these steps

1. cd ~/avatar-webkit
2. yarn link
3. cd ./demo
4. yarn link @quarkworks-inc/avatar-webkit
5. Delete node modules and yarn.lock in demo folder and in avatar-webkit folder
6. cd ../
7. yarn && yarn start
8. open a new tab cd ./demo
9. yarn && yarn dev

More Docs

https://docs.google.com/document/d/16c3qSYvMi_5l2zXdrsykb2xH6XneOqxTd2wwnEVawxY/edit#

Benchmarking

Macbook Pro

16" 2019 MBP, 2.4 GHz Intel Core i9, 16 GB RAM

iPhone XR