Draw on webcam using OpenCV - c++

Question

I want to draw/paint on a webcam screen using OpenCV. Since I'm reading from a cam, the frames are constantly changing, so I'm trying to figure out a way to keep or save the drawing on the current frame and use it for the next frame. The code below allows you to draw on the screen but when it gets the next frame, the drawing is gone and it starts over.
Could someone please help me ... Thanks.
CvCapture *input;
input = cvCaptureFromCAM( 0 );
cvSetMouseCallback("Demo",&on_mouse, 0);
for(;;)
{
frame = cvQueryFrame(input);
if(!image)
{
image = cvCreateImage( cvSize(frame->width, frame->height), IPL_DEPTH_8U, 3);
screenBuffer = cvCreateImage( cvSize(frame->width, frame->height), IPL_DEPTH_8U, 3);
}
cvCopy(frame, image, 0);
if(drawing) //drawing is a global variable
{
cvCircle(image, cvPoint(last_x,last_y), 10,CV_RGB(red,green,blue), -1, CV_AA, 0);
cvCopy(image, screenBuffer, 0);
}
cvShowImage( "Demo", screenBuffer );
}
void on_mouse( int event, int x, int y, int flags, void* param )
{
last_x = x;
last_y = y;
if(event==CV_EVENT_LBUTTONDOWN)
{
drawing = 1;
}
}

Answer 1

Draw into a separate image and then cvAdd() that to the video image immediately before dispalying it

Answer 2

I will not go into all the details why your approach is bad, but keep in mind that creating 2 extra frames for drawing is a little bit too much.
It's important that you realize that all this kinky stuff is being done on the same thread used to capture new frames. This means what exactly? It means that the extra code you are adding inside the loop will slow the process of capturing and displaying new frames. In other words, you are sabotaging yourself by lowering the framerate of your application. If you don't care, it's ok. If you do, my tip for you is that you stack the captured frames into a buffer and have another thread read, process and display them.
Ok, so you REALLY want to draw over the window that's displaying the captured frames. Well, the obvious thing you can't do (and you discovered this yourself) is that the drawing cannot be made on the captured frame because the frame it's replaced with new data on every loop. So what do you do? You create a 2nd frame to do the drawing. Let's call it the drawing_frame.
The only thing that will be on the drawing_frame are the circles that will appear when the mouse moves over the window, when the LBUTTON of the mouse is clicked (a 2nd click switches between ON/OFF).
After the drawing of the circle occurs, the drawing_frame is overlayed on top on the frame captured by the camera. This process is a little expensive on the CPU, and since we are doing it in the main thread of the application, it will also decrease the framerate.
I strongly suggest that everyone interested in adding/merging/overlaying transparent frames with OpenCV take a look at Transparent image overlays in OpenCV.
By the way, I'm using cvCaptureFromCAM(-1) becouse I'm on Linux. You probably should change that to whatever works for you. According to your post it's cvCaptureFromCAM(0).
#include <stdio.h>
#include <cv.h>
#include <highgui.h>
int drawing = 0;
int last_x = 0;
int last_y = 0;
void on_mouse(int event, int x, int y, int flags, void* param)
{
last_x = x;
last_y = y;
if (event == CV_EVENT_LBUTTONDOWN)
{
// switches between On and Off
if (drawing)
drawing = 0;
else
drawing = 1;
}
}
int main()
{
CvCapture* capture = NULL;
if ((capture = cvCaptureFromCAM(-1)) == NULL)
{
fprintf(stderr, "ERROR: capture is NULL \n");
return -1;
}
cvNamedWindow("mywindow", CV_WINDOW_AUTOSIZE);
cvQueryFrame(capture); // Sometimes needed to get correct data
cvSetMouseCallback("mywindow",&on_mouse, 0);
IplImage* frame = NULL;
IplImage* drawing_frame = NULL;
while (1)
{
if ((frame = cvQueryFrame(capture)) == NULL)
{
fprintf( stderr, "ERROR: cvQueryFrame failed\n");
break;
}
if (frame == NULL)
{
fprintf( stderr, "WARNING: cvQueryFrame returned NULL, sleeping..\n");
usleep(100000);
continue;
}
if (!drawing_frame) // This frame is created only once
{
drawing_frame = cvCreateImage(cvSize(frame->width, frame->height), frame->depth, frame->nChannels);
cvZero(drawing_frame);
}
if (drawing)
{
cvCircle(drawing_frame, cvPoint(last_x,last_y), 10,CV_RGB(0, 255, 0), -1, CV_AA, 0);
// For overlaying (copying transparent images) in OpenCV
// http://www.aishack.in/2010/07/transparent-image-overlays-in-opencv/
for (int x = 0; x < frame->width; x++)
{
for (int y = 0; y < frame->height; y++)
{
CvScalar source = cvGet2D(frame, y, x);
CvScalar over = cvGet2D(drawing_frame, y, x);
CvScalar merged;
CvScalar S = { 1,1,1,1 };
CvScalar D = { 1,1,1,1 };
for(int i = 0; i < 4; i++)
merged.val[i] = (S.val[i] * source.val[i] + D.val[i] * over.val[i]);
cvSet2D(frame, y, x, merged);
}
}
}
cvShowImage("mywindow", frame);
int key = cvWaitKey(10);
if (key == 113) // q was pressed on the keyboard
break;
}
cvReleaseImage(&frame);
cvReleaseImage(&drawing_frame);
cvReleaseCapture(&capture);
cvDestroyWindow("mywindow");
return 0;
}

Answer 3

You usually will have problems of adding images (they will eventually saturate), so I guess thats why you start over. I see you have color images... if you use more powerful stuff like OpenGL for your drawing you could use the overlay for your drawings. Otherwise check this out:
http://aishack.in/tutorials/transparent-image-overlays-in-opencv/