Face Tracking for Creative Coding

Overview

A growing collection of examples used to demonstrate basic concepts of the usage of face tracking with creative coding on the modern web. This repository is the basis of the workshop series called DataFaces. These events are focusing on data politics, creative coding, machine learning and critical discussion on cutting edge (surveillance based) technologies, the concepts are usually introduced in a playful & practical way using open source coding frameworks.

fig. 1: Feature points of tracked faces

fig. 2: Left- Source Code in Brackets editor, Middle- Indexed feature points, Right- Error message

Installation

All the examples are using P5JS for displaying graphics and playing back sounds. The face tracking is based on clmtrackr, except for 009_pose-basic & 010_pose_scene examples, see below for details. Each of these libraries can be found in the 'libs' folder. To launch the experiments, download the brackets editor. Open the folder of this repository and hit 'live preview' on the top right.

fig. 3: Scene example, where graphical elements are moved, based on head position

fig. 4: Sentiment Analysis

Folder Structure

Each example includes media files (sounds, animated gifs, images etc). The 'libs' folder contains the necessary js libraries and the pre-trained face traclking models. The examples are kept as simple as possible, so some html related manipulation needed to be wrapped into some utility functions. Check libs/worshop-utils.js on how these simplified, workshop specific functions operate (loading camera, tracker, etc.)

Workshop Schedule

See the wiki for detailed workshop schedule and the specific tasks we are dealing with during the session. Durations of the parts may vary depending on the knowledge and interest of the participants.

Notes

009_pose-basic & 010_pose_scene examples are using pose estimation from tensorflow.js together with p5js (we are using it for multiple head tracking on the workshop). This version of PoseNet is built with the ml5 javascript libary, that aims to make machine learning accessible to a broad audience of artists, creative coders, and students. The library provides access to machine learning algorithms and models in the browser, building on top of TensorFlow.js with no other external dependencies.

(c) 2018 Agoston Nagy / gpl v3