face tracking for virtual reality
23 Oct 2009Whilst studying for my masters I came accross various videos of the Nintendo wiimote being used for motion tracking to control 3d movement, and subsequently webcams being used for the same application. However there had been no work on how effective the webcam is at creating a 3d effect in comparison to traditional input methods such as the conventional mouse. Based on this I decided to design, build and test such a system for my masters thesis
The project can be considered in two discreate parts, the design, implementation and testing of the software and the testing of usefulness of the webcam in the system as an input device.
system structure
The development of the software can be divided into three discreet systems:
interfacing hardware and software
The system requires that images are captured from the webcam at high speed, to perform this OpenCV provides a simple cross platform interface which is able to grab images in all common formats from a video stream. DirectX based options were also explored for this task but they proved to be much more complex, requiring many more lines of code to achieve the same result.
face tracking
The success of a system such as this hinges on the ability of the system to be able to effectively track a face in the images. After extensive research it was decided to implement a simple library developed at Carnegie Mellon University, it provides a simple interface to locate faces in a stream of images based on training performed from the initial frame in the stream.
3d rendering
As this is only a system to demonstrate the plain 3d effect, not integrate this into any other kind of system, the 3d rendering system used was chosen to be as simple as possible to implement thus OpenGL was chosen.
development
Based on these decisions I implemented a multithreaded program in C++ which uses one thread to handle the capture of the images and location of the faces, this feeds a pair of coordinates to the other thread which updates the 3d display position dependant on the coordinates provided. A variation of the system was also produced which used the mouse for control instead of the webcam for comparative purposes.
testing
To test the usability of the system user testing was performed comparing the mouse input with the webcam input. This showed that while the face tracking system wasn’t without its faults it did give a more realistic impression of the scene being three dimensional.
report
source code
coming soon