I am trying to capture images from several cameras using the cameras driver,OpenCV and C++. My goal is to get as many FPS as possible, and to this end I have found saving the images in the hard drive to be the slowest operation. In order to speed up the process, I am doing each saving in separate threads. Problem is, I still have to wait for the saving to be complete to avoid the captured image being overwritten. Doing this provides good results, but for unknown reasons every 30-40 frames the speed is 10x higher.
I am addressing this by creating a ring buffer where I store the images, as these sudden drops in write speed are very short. I have obtained very good results using this approach, but unfortunately for more than 3 cameras the camera driver can't handle the stress and my program halts, waiting for the first image of the 4th camera to be saved. I checked and it's not the CPU, as 3 cameras + a thread writing random data in the disk works fine.
Now, seeing how using opencv reduced the stress on the camera driver, I would like to create a OpenCV mat buffer to hold the images while they are saved without my camera overwritting them (well, not until the buffer has done a whole lap, which I will make sure won't happen).
I know I can do
cv::Mat colorFrame(cv::Size(width, height),CV_8UC3,pointerToMemoryOfCamera);
to initialize a frame from the memory written by the camera. This does not solve my problem, as it will only point to the data, and the moment the camera overwrites it, it will corrupt the image saved.
How do I create a matrix with a given size and type, and then copy the contents of the memory to this matrix?
You need to create a deep copy. You can use clone:
cv::Mat colorFrame = cv::Mat(height, width, CV_8UC3, pointerToMemoryOfCamera).clone();
You can also speed up the process of saving the images using matwrite and matread functions.
Related
I want to create the vector of matrices to stores as many images as possible.
I know that,it is possible as written below:
vector<Mat> images1;
and during the image acquisition from the camera and i would save the images at 100fps with resolution of 1600*800 as below:
images1.push_back(InputImage.clone());
Where InputImage is the Mat and given by the camera. Since creating video during the acquisition process either leads to frame missing in the video or reduction in aquisition speed.
Later after stopping the image acquisition and before stopping the program, I would write the images into video as written below:
VideoWriter writer;
writer = Videowriter("video.avi",-1,100,frameSize(1600,800),false);
for (vector<Mat>::iterator iter = images1.begin(); ier != images1.end(); iter++)
writer.write(*iter);
Is it correct, since I am not sure the images1 can store the images around 1500 images without overflow.
You don't really have to worry about "overflow", whatever that means in your context.
The bigger problem is memory. A single frame takes (at 8 bits per color, with 3 colors) 3 * 1600 * 800 == 3.84Mb. At 100fps, One second of footage requires 0.384Gb of memory. 8GB of memory can only hold about 20 seconds of footage. You'll need almost 24GB of memory before you can hold a whole minute. There's a reason that the vast, vast, vast majority of Video Encoding Software only keeps a few frames of video data in memory at any given time, and dumps the rest to the hard drive (or discards it, depending on what purpose the software is serving).
What you should probably be doing (which is what programs like FRAPS do) is dumping frames to the hard drive as soon as soon as you receive them. Then, when recording finishes, you can either call it a day (if raw video footage is what you need) or you can begin a process of reading the file and encoding it into a more compressed format.
Pre-allocate your image vector in memory so that you just need to copy the frames without real-time allocation.
If you have memory problems, try dumping the frames to a file, the OS will hopefully be able to handle the I/O. If not try memory mapped files.
In order to accomplish some specific editing on some .avi files, I'd like to create an application (in C++) that is able to load, edit, and save those .avi files. But, what is the most efficient way? When first thinking about it, a simple 3D-Array containing a 2D-array of pixels for every frame seems the simplest solution; But then its size would be ENORMOUS. I mean, let's assume that a pixel only needs a color. One color would mean 3bytes (1char r, 1char b, 1char g). If I now have a 1920x1080 video format, this would mean 2MEGABYTES for only one frame! This data may or may not be smaller if using pointers for the colors, so that alreay used colors wont take more size - I don't really know, since I'm pretty new to C++ and the whole low-level stuff. (As a comparison: One of my AVI files recorded with Xvid codec is 40seconds long, 30fps, and only has 2MB.)
So how would you actually store the video data (Not even the audio, just the video) efficiently (while still being easily able to perform per-frame-changes on it)?
As you have realised, uncompressed video is enormous and it is not practical to store an entire video in this way.
Video compression is an extremely complex topic, but more-or-less, it works as follows: certain "key-frames" are compressed using fairly standard compression techniques similar or identical to still-photo compression such as JPEG. Frames following key-frames are compressed by comparing the frame with the previous one and looking for changes (such as moving blocks). Every now and again, a new key-frame is used.
You don't really have to worry much about that as you are not going to write your own video coder/decoder (codec). There are standard ones.
What will happen is that your program will decode the compressed video frame-by-frame and keep a certain number of frames in memory while you are working on them and then re-encode them when it is finished. In the uncompressed form, you will have access to the individual pixels and can work on them how you want.
You are probably not going to do that either by yourself - it is very hard. You probably need to use a framework, such as OpenCV. There are a huge number of standard filters and tools built in to these frameworks, and it may be that what you want to do is already implemented somewhere.
The OpenCV framework can return individual frames in a Mat object and you can then access the pixels. See this post Get Pixels from Mat
OpenCV
Tutorial page: Open CV Tutorial
I know the title is a bit vague but I'm not sure how else to describe it.
CentOS with ffmpeg + OpenCV 2.4.9. I'm working on a simple motion detection system which uses a stream from an IP camera (h264).
Once in a while the stream hiccups and throws in a "bad frame" (see pic-bad.png link below). The problem is, these frames vary largely from the previous frames and causes a "motion" event to get triggered even though no actual motion occured.
The pictures below will explain the problem.
Good frame (motion captured):
Bad frame (no motion, just a broken frame):
The bad frame gets caught randomly. I guess I can make a bad frame detector by analyzing (looping) through the pixels going down from a certain position to see if they are all the same, but I'm wondering if there is any other, more efficient, "by the book" approach to detecting these types of bad frames and just skipping over them.
Thank You!
EDIT UPDATE:
The frame is grabbed using a C++ motion detection program via cvQueryFrame(camera); so I do not directly interface with ffmpeg, OpenCV does it on the backend. I'm using the latest version of ffmpeg compiled from git source. All of the libraries are also up to date (h264, etc, all downloaded and compiled yesterday). The data is coming from an RTSP stream (ffserver). I've tested over multiple cameras (dahua 1 - 3 MP models) and the frame glitch is pretty persistent across all of them, although it doesn't happen continuously, just once on a while (ex: once every 10 minutes).
What comes to my mind in first approach is to check dissimilarity between example of valid frame and the one we are checking by counting the pixels that are not the same. Dividing this number by the area we get percentage which measures dissimilarity. I would guess above 0.5 we can say that tested frame is invalid because it differs too much from the example of valid one.
This assumption is only appropriate if you have a static camera (it does not move) and the objects which can move in front of it are not in the shortest distance (depends from focal length, but if you have e.g. wide lenses so objects should not appear less than 30 cm in front of camera to prevent situation that objects "jumps" into a frame from nowhere and has it size bigger that 50% of frame area).
Here you have opencv function which does what I said. In fact you can adjust dissimilarity coefficient more large if you think motion changes will be more rapid. Please notice that first parameter should be an example of valid frame.
bool IsBadFrame(const cv::Mat &goodFrame, const cv::Mat &nextFrame) {
// assert(goodFrame.size() == nextFrame.size())
cv::Mat g, g2;
cv::cvtColor(goodFrame, g, CV_BGR2GRAY);
cv::cvtColor(nextFrame, g2, CV_BGR2GRAY);
cv::Mat diff = g2 != g;
float similarity = (float)cv::countNonZero(diff) / (goodFrame.size().height * goodFrame.size().width);
return similarity > 0.5f;
}
You do not mention if you use ffmpeg command line or libraries, but in the latter case you can check the bad frame flag (I forgot its exact description) and simply ignore those frames.
remove waitKey(50) or change it to waitKey(1). I think opencv does not spawn a new thread to perform capture. so when there is a pause, it confuses the buffer management routines, causing bad frames..maybe?
I have dahua cameras and observed that with higher delay, bad frames are observed. And they go away completely with waitKey(1). The pause does not necessarily need to come from waitKey. Calling routines also cause such pauses and result in bad frames if they are taking long enough.
This means that there should be minimum pause between consecutive frame grabs.the solution would be to use two threads to perform capture and processing separately.
I am trying to Stitch an image from a live video camera (more like a panorama) using OpenCv. The stitching is working fine. My problem is, i want the stitching to be done in real time say around 30 mph but the processing of the stitching is slow.
I want to use Threads to improve the speed but in order to use them do i need to store my live video stream or is there any way to directly use threads for the live stream.
Here is a sample code:
SapAcqDevice *pAcq=new SapAcqDevice("Genie_HM1400_1", false);
SapBuffer *pBuffer = new SapBuffer(20,pAcq);
SapView *pView=new SapView(pBuffer,(HWND)-1);
SapAcqDeviceToBuf *pTransfer= new SapAcqDeviceToB(pAcq,pBuffer,XferCallback,pView);
pAcq->Create();
pBuffer->Create();
pView->Create();
pTransfer->Create();
pTransfer->Grab();
printf("Press any key to stop grab\n");
getch();
pTransfer->Freeze();
pTransfer->Wait(5000);
printf("Press any key to terminate\n");
getch();
This above code is used to capture the live stream. The XferCallback function is used to do the processing of the frames. In this function i call my stitch engine. Since the processing of the engine is slow i want to use threads.
Here is a sample code of the callback function:
SapView *pView = (SapView *) pInfo->GetContext();
SapBuffer *pBuffer;
pBuffer = pView->GetBuffer();
void *pData=NULL;
pBuffer->GetAddress(&pData);
int width=pBuffer->GetWidth();
int height=pBuffer->GetHeight();
int depth=pBuffer->GetPixelDepth();
IplImage *fram;
fram = cvCreateImage(cvSize(width,height),depth,1);
cvSetImageData(fram,pData,width);
stitching(frame_num , fram);
cvWaitKey(1);
frame_num++;
I want many threads working on the stitch engine.
If you think you can get the stitching fast enough using threads, then go for it.
do i need to store my live video stream or is there any way to
directly use threads for the live stream.
You might benefit from setting up a ring buffer with preallocated frames. You know the image size isn't going to change. So your Sapera acquisition callback simply pushes a frame into the buffer.
You then have another thread that sits there stitching as fast as it can and maintaining state information to help optimize the next stitch. You have not given much information about the stitching process, but presumably you can make it parallel with OpenMP. If that is fast enough to keep up with frame acquisition then you'll be fine. If not, then you will start dropping frames because your ring buffer is full.
As hinted above, you can probably predict where the stitching for the next frame ought to begin. This is on the basis that movement between one frame and the next should be reasonably small and/or smooth. This way you narrow your search and greatly improve the speed.
To track object on video frame, first of all I extract image frames from video and save those images to a folder. Then I am supposed to process those images to find an object. Actually I do not know if this is a practical thing, because all the algorithm did this for one step. Is this correct?
Well, your approach will consume a lot of space on your disk depending on the size of the video and the size of the frames, plus you will spend a considerable amount of time reading frames from the disk.
Have you tried to perform real-time video processing instead? If your algorithm is not too slow, there are some posts that show the things that you need to do:
This post demonstrates how to use the C interface of OpenCV to execute a function to convert frames captured by the webcam (on-the-fly) to grayscale and displays them on the screen;
This post shows a simple way to detect a square in an image using the C++ interface;
This post is a slight variation of the one above, and shows how to detect a paper sheet;
This thread shows several different ways to perform advanced square detection.
I trust you are capable of converting code from the C interface to the C++ interface.
There is no point in storing frames of a video if you're using OpenCV, as it has really handy methods for capturing frames from a camera/stored video real-time.
In this post you have an example code for capturing frames from a video.
Then, if you want to detect objects on those frames, you need to process each frame using a detection algorithm. OpenCV brings some sample code related to the topic. You can try to use SIFT algorithm, to detect a picture, for example.