face tracking for virtual reality

Whilst studying for my masters I came accross various videos of the Nintendo wiimote being used for motion tracking to control 3d movement, and subsequently webcams being used for the same application. However there had been no work on how effective the webcam is at creating a 3d effect in comparison to traditional input methods such as the conventional mouse. Based on this I decided to design, build and test such a system for my masters thesis

The project can be considered in two discreate parts, the design, implementation and testing of the software and the testing of usefulness of the webcam in the system as an input device.

system structure

The development of the software can be divided into three discreet systems:

interfacing hardware and software

The system requires that images are captured from the webcam at high speed, to perform this OpenCV provides a simple cross platform interface which is able to grab images in all common formats from a video stream. DirectX based options were also explored for this task but they proved to be much more complex, requiring many more lines of code to achieve the same result.

face tracking

The success of a system such as this hinges on the ability of the system to be able to effectively track a face in the images. After extensive research it was decided to implement a simple library developed at Carnegie Mellon University, it provides a simple interface to locate faces in a stream of images based on training performed from the initial frame in the stream.

3d rendering

As this is only a system to demonstrate the plain 3d effect, not integrate this into any other kind of system, the 3d rendering system used was chosen to be as simple as possible to implement thus OpenGL was chosen.


Based on these decisions I implemented a multithreaded program in C++ which uses one thread to handle the capture of the images and location of the faces, this feeds a pair of coordinates to the other thread which updates the 3d display position dependant on the coordinates provided. A variation of the system was also produced which used the mouse for control instead of the webcam for comparative purposes.


To test the usability of the system user testing was performed comparing the mouse input with the webcam input. This showed that while the face tracking system wasn’t without its faults it did give a more realistic impression of the scene being three dimensional.


Download the report (pdf)

source code

coming soon

related links



Face tracking at Carnegie Mellon

Joe Faith - project supervisor