@needle-tools/facefilter NPM

Needle Engine FaceFilter

Add face tracking to your Needle Engine projects with ease.

Features

Blendshape mesh face tracking
Texture face tracking with google/mediapipe canonical or procreate texture layouts (Use the FaceMeshTexture class)
Video face tracking: Play a video as a face texture (Use the FaceMeshVideo class)
Custom shader face meshes: Use custom materials on your face mesh (Use the FaceMeshCustomShader class)
Tracking for multiple faces at once (with smoothing)
Can be used with Unity to create filters, animations, materials...

Quickstart

Run npm i @needle-tools/facefilter in your web project

Then see the code or examples below:

Face Filter with HTML only

See full examples in /examples/html/

Supported attributes


`face-filter`	URL to either a image or model file. Supported formats: `glTF`, `GLB`, `FBX`, `OBJ` or 2D `jpeg`, `jpg`, `png`, `webp`
`face-filter-mask`	(optional, 2D only) URL to image file that will be used to mask out the face filter texture
`face-filter-layout`	(optional, 2D only) Either `procreate`, `mediapipe` or `canonical`. Default `mediapipe`
`face-filter-scale`	(optional, 3D only) Apply scale to the 3D face filter model (e.g. `face-filter-scale=".5"`)
`face-filter-offset`	(optional, 3D only) Offset the 3D face filter model (e.g. `face-filter-offset="0.0, 0.1, 0.1"`)
`face-filter-max-faces`	(optional) How many faces should be tracked automatically. Default: `1`
`face-filter-show-video`	(optional) Should the camera videofeed be rendered in the background? Default: `true`. Can be set to `0` to hide the videofeed in the background.
`face-filter-video-selector`	(optional) HTML selector for a HTMLVideoElement. Useful if you want to render provide your own video element on the website elsewhere. If none is provided a hidden video element will be created automatically.

HTML Example

<!DOCTYPE html>
<html>
  <head>
      <script type="importmap">
          {
            "imports": {
              "three": "https://cdn.jsdelivr.net/npm/@needle-tools/engine@4.4.0-alpha.5/dist/three.min.js",
              "@needle-tools/engine": "https://cdn.jsdelivr.net/npm/@needle-tools/engine@4.4.0-alpha.5/dist/needle-engine.min.js",
              "@needle-tools/facefilter": "https://cdn.jsdelivr.net/npm/@needle-tools/facefilter/dist/facefilter.min.js"
            }
          }
      </script>
      <script type="module" src="https://cdn.jsdelivr.net/npm/@needle-tools/facefilter/dist/facefilter.min.js"></script>
  </head>

  <body style="margin:0; padding:0;">
      <needle-engine
          background-color="#ffffdd"
          face-filter="https://cdn.needle.tools/static/facefilter/facemask-template-procreate.webp"
          face-filter-mask="https://cdn.needle.tools/static/facefilter/facemask-occlusion-procreate.webp"
          face-filter-layout="procreate"
          >
      </needle-engine>
  </body>
</html>

Open 2D Example – Open 3D Example

Face Mesh Texture Filter

import { onStart } from '@needle-tools/engine';
import { FaceMeshTexture, NeedleFaceFilterTrackingManager } from '@needle-tools/facefilter';

onStart(context => {
  const scene = context.scene;

  // Create a face filter tracking manager and add it to the scene
  const filtermanager = new NeedleFaceFilterTrackingManager();
  filtermanager.createMenuButton = true;
  scene.addComponent(filtermanager);

  // Creating a filter
  const filter = new FaceMeshTexture({
    layout: 'procreate', // we support both the google/mediapipe canonical layout and procreate/arkit layouts
    texture: {
      url: './assets/crocodile.webp', // provide a URL to the texture
      // texture: <your texture> // alternatively you can assign an existing texture directly
    },
  });
  // Activate one of your filters
  filtermanager.activateFilter(filter);
});

Open Example on Stackblitz

Face Mesh Blendshapes Filter

import { onStart } from '@needle-tools/engine';
import { FaceMeshTexture, NeedleFaceFilterTrackingManager } from '@needle-tools/facefilter';

onStart(context => {
  const scene = context.scene;

  // Create a face filter tracking manager and add it to the scene
  const filtermanager = new NeedleFaceFilterTrackingManager();
  filtermanager.createMenuButton = false;
  scene.addComponent(filtermanager);

  // Creating a filter using a GLB/glTF URL model that has blendshapes
  const filter = await FaceFilterRoot.create('https://cloud.needle.tools/-/assets/Z23hmXBZWllze-ZWllze/file', {
    scale: 0.5,
    offset: { x: 0, y: 0.01, z: 0 },
  });
  if (filter) filtermanager.activateFilter(filter);
});

Open Example on Stackblitz