pa-egloff v0.2.9
PA.EmotionsTracker
The EmotionalAwareness-Tool consists of two parts: a tracker, where webcam images are processed and the estimates regarding valence, fatigue and engagement are calculated, and a client that visualizes the estimates.
This is the tracker project, the client can be found here
This project is published on npm
Documentation
Used tensorflow models
- faceAPI for facial expression and head orientation detection
- Face landmarks detection for explicit facial landmarks detection
- PoseNet for pose detection
Endpoints
Communication with the tracker is done over IPC. The following endpoints exist:
Incoming
start-tracker
to start the trackerpause-tracker
to pause the trackerset-low-performance-mode
withpayload: boolean
to activate/deactivate low performance mode
Outgoing
face-api-expression
withpayload: FaceApiExpression
face-api-head-orientation
withpayload: FaceApiHeadOrientation
face-api-aggregated-expression
withpayload: FaceApiAggregatedExpression
face-api-smiled-once
withpayload: FaceApiHasSmiled
face-api-raw-expressions
withpayload: string
facial-landmarks-data
withpayload: FacialLandmarksData
facial-landmarks-blink-count
withpayload: FacialLandmarksBlinkCount
pose
withpayload: PoseData
z-deviation
withpayload: ZDeviation
head-orientation-deviation
withpayload: HeadOrientationDeviation
metric-log
withpayload: MetricLog
estimated-dimensions
withpayload: EstimatedDimensions
update-tracker-status
withpayload: Status
to get the current state of the tracker
Main Components
The following diagram shows the main dataflow of the program. For the sake of clarity, the Renderer is not depicted. The Renderer handles incoming requests from the client, like starting/pausing or switching to low performance mode.
Open issues concerning coupling between tracker & client
Tracker: pa-shared interface is no longer used
Client: in package.json, we need to know exact folder structure of tracker
"html": "./node_modules/emotions-tracker/src/renderer/index.html", "js": "./node_modules/emotions-tracker/src/renderer/renderer.ts", "name": "tracker_window"
Client: in webpack.plugins.js, we have to copy the whole assets folder from the tracker to the client, because the tracker_window needs access to models / images
{ from: path.resolve(__dirname, "node_modules/emotions-tracker/src/renderer/assets"), to: path.resolve(__dirname, ".webpack/renderer/assets"), }