Using a webcam with OpenCV to display video on an LED Matrix controlled by arduino

06 Dec 2009

Recently I’ve been looking for new things to do with my LED Matrix shield I made for arduino and as I have some experience with writing video caputure systems I thought I’d have a go at trying to capture webcam video frames and processing them and  then displaying them in real time on an 8x8 LED Matrix. Previously as part of my masters thesis I had worked with OpenCV which, amongst many other image processing and computer vision features, provides simple, cross platform video capture capability so this formed the starting point of my experiments.

In this post I will describe a method for the capture of video frames from a webcam using OpenCV, the subsequent processing required to convert to an 8x8 monochrome image and the process of sending the data via serial to arduino using libserial which is linux specific. Example source code is included at the end.

Having previously used a Logitech Quickcam Fusion with windows and OpenCV I started out on this project assuming it would work in linux, however the camera isn't fully supported by Linux due to the specific chipset used so I replaced my webcam and found the Microsoft Lifecam Cinema Webcam to be fully linux compatible and the highest quality webcam currently available at a sensible price (currently ~£45 on Amazon) and has proved to work very well in all respects producing very high quality images (for a webcam).

Video Capture and Processing

The rest of this post assumes that OpenCV has been correctly installed and is known to the compiler and will concentrate on the main functions required to perform the capture and image processing. OpenCV represents video streams as CvCapture objects, andOpenCV images such as video frames as IPL images, a standard type defined by Intel but basically an encapsulated BGR matrix of the pixels. So initially I define the variables which will hold the capture stream and the images required:

CvCapture * pCapture;    //new OpenCV capture stream
IplImage * pVideoFrame;           //new OpenCV image
IplImage * pVideoFrameBW;      //new OpenCV image for colour conversion

A video stream from a camera, assuming that the camera is correctly installed in the guess operating system, can be initialised in one line:

pCapture = cvCaptureFromCAM(0);    //choose camera for capture

where the parameter is the capture device, if you only have one webcam it's likely to be 0 otherwise you'll need to experiment to find the correct one. To display the image being transmitted a window can be created using the HighGUI library integrated with OpenCV:

cvNamedWindow( "video", CV_WINDOW_AUTOSIZE );

Now the main data structures have been defined we need to enter into a loop to perform the frame capture and processing. OpenCV includes a GUI library and this will be used to display the images on screen and also control behaviour of the program, in the example code extra GUI features control some parameters. To control the loop we will use the cvWaitKey(int time) function which returns -1 if no key press has been detected and the ASCII code of the character if a key is pressed within the time specified in milliseconds. Therefore the main loop will be of the form:

int key = -1;
while(key==-1)
{
capture and process
}
clean up and exit

Inside every cycle of the loop a new frame needs to be captured from the stream, this is achieved using the the cvQueryFrame(CvCapture pCapture) method:

pVideoFrame = cvQueryFrame(pCapture);

Next to reduce the number of colours in the image a threshold is needed of the image, again OpenCV includes a function to do this:

cvThreshold(pVideoFrame, pVideoFrame, thresh, 255, CV_THRESH_BINARY);

where thresh is the level to which thresholding is required, more information can be found here. As the data required for display on a basic LED matrix is not in colour it needs to be converted to black and white, in OpenCV this works by creating an empty image of the correct colour depth and then using the cvCvtColor() function:

pVideoFrameBW=cvCreateImage(cvGetSize(pVideoFrame),8,1);
cvCvtColor(pVideoFrame, pVideoFrameBW, CV_BGR2GRAY);

As currently the LED matrix I'm working with is an 8x8 type the image needs to be scaled down considerably to fit, in my case by about 60 time, this is done in a similar way to the conversion of the colour:

/*create small image*/
IplImage* out = cvCreateImage( cvSize(pVideoFrameBW->width/60,
pVideoFrameBW->height/60), pVideoFrameBW->depth,
pVideoFrameBW->nChannels );
cvResize(pVideoFrameBW,out,CV_INTER_NN); 

In order to display the image on screen at a sensible size the small image needs to be scaled back up again and then displayed:

cvResize(out,pVideoFrameBW,CV_INTER_NN);
cvShowImage( "video", pVideoFrameBW);

This produces a window which will look something like this:

Now that a small monochrome image has been formed the value of each pixel needs to be transmitted to the arduino if it is black or white. The value of a pixel can be extracted from the image using the cvGet2D() method and returns a CvScalar object containing three matrices ( for BGR respectively)  for ease I have taken just the blue values here, the two loops consider each column of pixels and store their value in p:

for(int y=0; y<8; y++)
{
  for(int x=0;x<8;x++)
  {
    CvScalar s;
    s=cvGet2D(out,y,x);
    /*save pixel value (blue)*/
    int p = s.val[0];
    /*print value to terminal for debugging*/
    printf("%i",p);
    if(p==255) //if the pixel is black
    {
    p=1; //turn on led
    }
}

Now that all processing and capture has taken place for this frame we can wait for a keypress to close the program and clean up any memory used during this iteration:

key = cvWaitKey(20);
/*release memory*/
cvReleaseImage(&pVideoFrameBW);
cvReleaseImage(&out);

Now the main while loop can be closed and final cleaning up carried out

cvReleaseImage( &pVideoFrame );
cvReleaseCapture ( &pCapture );
cvDestroyWindow( "video" );

That concludes the summary of what is required to capture images from a video stream and process it to a smaller monochrome image, next we will consider sending the data.

Serial Communication

I shall be fairly brief discussing serial communication as it's a topic well covered in other postings such as here. It can be achieved in linux quite simply using libSerial which allows a steam object to be created and anything which is added to the stream is forwarded to the serial port. In this system this means each pixel would be represented by a 1 or 0 depending on if the led is required to be on or off which is sent to the arduino and interpreted. In my example I used 115200 baud but had to add a delay in the loop using usleep to allow the arduino to have time to process the incoming data. This method of sending the data is however quite inefficient as each led only requires an on or off to be sent which could consist of one bit being either 0 or 1 however this system sends data using ASCII. ASCII uses several bits per led making the communication and processing much slower than it could be, shifting the process to a stream of bits should greatly improve the framerate and will be discussed in a later post when I finish writing the code!

Arduino Code

The the arduino takes data from the serial stream and displays it on the leds and takes no other part in the processing. It uses similar code to my twitterpop project however I will get around to posting the code here soon.

Example

An example class can be found here, it shows the includes which might not normally be there but it makes things slightly clearer, you'll need to do you're own header file.