@jspsych-contrib/extension-mediapipe-face-mesh v2.0.0

mediapipe-face-mesh extension

Overview

This extension provides online tracking of facial posture during trials using the MediaPipe Face Mesh library. It currently records the estimated 3D translation and rotation.

The primary purpose of this extension is to use it in plugins. For instance, it allows creation of interactive virtual mirror paradigms similar to the Open Virtual Mirror Framework. However, it can also be used as an online face tracker if recording of sensitive facial data can be avoided. Refer to trial parameters for more details.

Loading

In browser

<script src="https://unpkg.com/@jspsych-contrib/extension-mediapipe-face-mesh@2.0.0">

Via NPM

npm install @jspsych-contrib/extension-mediapipe-face-mesh

import jsPsychExtensionMediapipeFacemesh from '@jspsych-contrib/extension-mediapipe-face-mesh';

Compatibility

This extension was developed for, and tested with jsPsych v7.3.0.

Documentation

Initialization Parameters

Initialization parameters can be set when calling initJsPsych()

initJsPsych({
  extensions: [
    {type: jsPsychExtensionMediapipeFacemesh, params: {...}}
  ]
})

Trial Parameters

Trial parameters can be set when adding the extension to a trial object.

var trial = {
  type: jsPsych...,
  extensions: [
    {type: jsPsychExtensionMediapipeFacemesh, params: {...}}
  ]
}

Data Generated

In addition to the default data collected by all plugins, this plugin collects all parameter data described above and the following data for each trial.

Usage in other plugins

This extension allows creation of interactive experiments, e.g., virtual mirrors as in this project. Here is an example of the sage of the extension from another plugin:

  class MyInteractivePlugin {

    constructor(jsPsych) {
        this.jsPsych = jsPsych;
        this.mediapipe = this.jsPsych.extensions["mediapipe-face-mesh"];
    }

    trial(display_element, trial) {

        const callback = (trackingResult) => {
            console.log(trackingResult);
        };

        this.mediapipe.addTrackingResultCallback(callback);

        // Do some stuff

        this.jsPsych.pluginAPI.setTimeout(
            () => {
                this.mediapipe.removeTrackingResultCallback(callback);
                this.jsPsych.finishTrial();
            },
            trial["trial_duration"]);
    }