I want to receive frames from UDP port and run face recognition algorithms on them with opencv cv::dnn framework. Tello drone is sending frames over UDP protocol.
/* load dnn model */
cv::dnn::Net net = cv::dnn::readNetFromCaffe("dnnmodel/deploy.prototxt.txt","dnnmodel/res10_300x300_ssd_iter_140000.caffemodel");
cv::VideoCapture cap("udp://#0.0.0.0:11111?overrun_nonfatal=1&fifo_size=50000000");
cv::Mat frame;
float confidenceThreshold = 0.2;
while(true)
{
if(!cap.read(frame))
break;
cv::Mat inputBlob = cv::dnn::blobFromImage(frame, 1, cv::Size(300, 300), cv::Scalar(104.0, 177.0, 123.0), false, false);
net.setInput(inputBlob, "data");
cv::Mat detection = net.forward("detection_out");
cv::Mat detectionMat(detection.size[2], detection.size[3], CV_32F, detection.ptr<float>());
cv::imshow("window", frame);
char key = cv::waitKey(10);
if (key == 27) // ESC
break;
}
Camera response time is very high like 10-20 seconds. When I move the camera, I get the new frame after 20 seconds.
But If I used my own laptop webcam instead of udp port in VideoCapture with this call ;
VideoCapture cap;
cap.open(0)
result is perfect. There is no delay when I am using the webcam.
What is the reason of this delay ?
With unreliable protocols like UDP, where the comms stack can, and will, discard data if not promptly taken out to user space, it's important to attach a high importance to reading data, even at the expense of added complexity in the recv code.
In this case, a separate thread can be used to extract datagrams as soon as available and queueing the buffers, (pointers to buffers, anyway), off to processing code that, otherwise, would result in excessive time use and dropped datagrams.
Hey - it worked!
Related
I have purchased an ELP-1MP2CAM001 which shows up as two webcam devices on Windows. If I open the Windwos default "Camera" app and Skype I can display the feeds from both the left and right camera at the same time. I don't think therefore it is a USB Bandwidth issue with two cameras coming into the same port
I'm using fairly standard code (shown below) to open both of these feeds and it works successfully if I use two standard Microsoft HD3000 webcams instead of the single stereo camera.
I've tried a range of numbers inside the cap2() arguments so I don't think it's hiding at number 10 or anything weird like that.
My questions are:
There must be some sort of on board hub for the ELP cameras, do I need to do something different in OpenCV?
Could it be that both frames are accessible through cap(0)? This seems unlikely to me.
This questions says I don't need to do anything special? but obviously I'm missing something.
Any help on this would be great.
Code:
#include <opencv2/opencv.hpp>
using namespace cv;
int main(int, char**)
{
VideoCapture cap(0); // open the default camera
VideoCapture cap2(1); // open the default camera
cap.set(CV_CAP_PROP_FRAME_WIDTH, 240);
cap.set(CV_CAP_PROP_FRAME_HEIGHT, 120);
cap2.set(CV_CAP_PROP_FRAME_WIDTH, 240);
cap2.set(CV_CAP_PROP_FRAME_HEIGHT, 120);
if (!cap.isOpened()) // check if we succeeded
return -1;
if (!cap2.isOpened()) // check if we succeeded
return -1;
Mat frame;
Mat frame2;
namedWindow("Frame", 1);
namedWindow("Frame2", 1);
for (;;)
{
Mat frame;
cap >> frame; // get a new frame from camera
imshow("Frame", frame);
Mat frame2;
cap2 >> frame2;
imshow("Frame2", frame2);
if (waitKey(30) >= 0) break; // Finish on "esc" key
}
// the camera will be deinitialized automatically in VideoCapture destructor
return 0;
}
I have the same camera as this, and I ran into the same problem before. Try change the order of your code as below:
VideoCapture cap(0);
cap.set(CV_CAP_PROP_FRAME_WIDTH, 240);
cap.set(CV_CAP_PROP_FRAME_HEIGHT, 120);
VideoCapture cap2(1);
cap2.set(CV_CAP_PROP_FRAME_WIDTH, 240);
cap2.set(CV_CAP_PROP_FRAME_HEIGHT, 120);
I think it's a problem with USB bandwidth. In your code, you opened two cameras in full resolution at the beginning, then you change the resolution of two cameras.
When you call VideoCapture cap(0); // open the default camera, cap has resolution 1280*720. cap already occupied the bandwidth. Thereby, VideoCapture cap2(1); won't open camera cap2 sucessfully.
Hope it helps.
According to VideoCapture documentation, there is a function called cv::VideoCapture::grab:
The primary use of the function is in multi-camera environments, especially when the cameras do not have hardware synchronization. That is, you call VideoCapture::grab() for each camera and after that call the slower method VideoCapture::retrieve() to decode and get frame from each camera
You can try that, with:
cap.grab();
cap.retrieve(...);
I am capturing video through a webcam which gives a mjpeg stream.
I did the video capture in a worker thread.
I start the capture like this:
const std::string videoStreamAddress = "http://192.168.1.173:80/live/0/mjpeg.jpg?x.mjpeg";
qDebug() << "start";
cap.open(videoStreamAddress);
qDebug() << "really started";
cap.set(CV_CAP_PROP_FRAME_WIDTH, 720);
cap.set(CV_CAP_PROP_FRAME_HEIGHT, 576);
the camera is feeding the stream at 20fps.
But if I did the reading in 20fps like this:
if (!cap.isOpened()) return;
Mat frame;
cap >> frame; // get a new frame from camera
mutex.lock();
m_imageFrame = frame;
mutex.unlock();
Then there is a 3+ seconds lag.
The reason is that the captured video is first stored in a buffer.When I first start the camera, the buffer is accumulated but I did not read the frames out. So If I read from the buffer it always gives me the old frames.
The only solutions I have now is to read the buffer at 30fps so it will clean the buffer quickly and there's no more serious lag.
Is there any other possible solution so that I could clean/flush the buffer manually each time I start the camera?
OpenCV Solution
According to this source, you can set the buffersize of a cv::VideoCapture object.
cv::VideoCapture cap;
cap.set(CV_CAP_PROP_BUFFERSIZE, 3); // internal buffer will now store only 3 frames
// rest of your code...
There is an important limitation however:
CV_CAP_PROP_BUFFERSIZE Amount of frames stored in internal buffer memory (note: only supported by DC1394 v 2.x backend currently)
Update from comments. In newer versions of OpenCV (3.4+), the limitation seems to be gone and the code uses scoped enumerations:
cv::VideoCapture cap;
cap.set(cv::CAP_PROP_BUFFERSIZE, 3);
Hackaround 1
If the solution does not work, take a look at this post that explains how to hack around the issue.
In a nutshell: the time needed to query a frame is measured; if it is too low, it means the frame was read from the buffer and can be discarded. Continue querying frames until the time measured exceeds a certain limit. When this happens, the buffer was empty and the returned frame is up to date.
(The answer on the linked post shows: returning a frame from the buffer takes about 1/8th the time of returning an up to date frame. Your mileage may vary, of course!)
Hackaround 2
A different solution, inspired by this post, is to create a third thread that grabs frames continuously at high speed to keep the buffer empty. This thread should use the cv::VideoCapture.grab() to avoid overhead.
You could use a simple spin-lock to synchronize reading frames between the real worker thread and the third thread.
Guys this is pretty stupid and nasty solution, but accepted answer didn't helped me for some reasons. (Code in python but the essence pretty clear)
# vcap.set(cv2.CAP_PROP_BUFFERSIZE, 1)
data = np.zeros((1140, 2560))
image = plt.imshow(data)
while True:
vcap = cv2.VideoCapture("rtsp://admin:#192.168.3.231")
ret, frame = vcap.read()
image.set_data(frame)
plt.pause(0.5) # any other consuming operation
vcap.release()
An implementation of Hackaround 2 in Maarten's answer using Python. It starts a thread and keeps the latest frame from camera.read() as a class attribute. A similar strategy can be done in c++
import threading
import cv2
# Define the thread that will continuously pull frames from the camera
class CameraBufferCleanerThread(threading.Thread):
def __init__(self, camera, name='camera-buffer-cleaner-thread'):
self.camera = camera
self.last_frame = None
super(CameraBufferCleanerThread, self).__init__(name=name)
self.start()
def run(self):
while True:
ret, self.last_frame = self.camera.read()
# Start the camera
camera = cv2.VideoCapture(0)
# Start the cleaning thread
cam_cleaner = CameraBufferCleanerThread(camera)
# Use the frame whenever you want
while True:
if cam_cleaner.last_frame is not None:
cv2.imshow('The last frame', cam_cleaner.last_frame)
cv2.waitKey(10)
You can make sure that grabbing the frame took a bit of time. It is quite simple to code, though a bit unreliable; potentially, this code could lead to a deadlock.
#include <chrono>
using clock = std::chrono::high_resolution_clock;
using duration_float = std::chrono::duration_cast<std::chrono::duration<float>>;
// ...
while (1) {
TimePoint time_start = clock::now();
camera.grab();
if (duration_float(clock::now() - time_start).count() * camera.get(cv::CAP_PROP_FPS) > 0.5) {
break;
}
}
camera.retrieve(dst_image);
The code uses C++11.
There is an option to drop old buffers if you use a GStreamer pipeline. appsink drop=true option "Drops old buffers when the buffer queue is filled". In my particular case, there is a delay (from time to time) during the live stream processing, so it's needed to get the latest frame each VideoCapture.read call.
#include <chrono>
#include <thread>
#include <opencv4/opencv2/highgui.hpp>
static constexpr const char * const WINDOW = "1";
void video_test() {
// It doesn't work properly without `drop=true` option
cv::VideoCapture video("v4l2src device=/dev/video0 ! videoconvert ! videoscale ! videorate ! video/x-raw,width=640 ! appsink drop=true", cv::CAP_GSTREAMER);
if(!video.isOpened()) {
return;
}
cv::namedWindow(
WINDOW,
cv::WINDOW_GUI_NORMAL | cv::WINDOW_NORMAL | cv::WINDOW_KEEPRATIO
);
cv::resizeWindow(WINDOW, 700, 700);
cv::Mat frame;
const std::chrono::seconds sec(1);
while(true) {
if(!video.read(frame)) {
break;
}
std::this_thread::sleep_for(sec);
cv::imshow(WINDOW, frame);
cv::waitKey(1);
}
}
If you know the framerate of your camera you may use this information (i.e. 30 frames per second) to grab the frames until you got a lower frame rate.
It works because if grab function become delayed (i.e. get more time to grab a frame than the standard frame rate), it means that you got every frame inside the buffer and opencv need to wait the next frame to come from camera.
while(True):
prev_time=time.time()
ref=vid.grab()
if (time.time()-prev_time)>0.030:#something around 33 FPS
break
ret,frame = vid.retrieve(ref)
I am taking input from the camera. to be more clear i added a photo:
2 cameras that connected on the same usb port
with OpenCV as the following :
#define CamLeft 2
#define CamRight 0
#define WIN_L "win_l"
#define WIN_R "win_r"
int main(int argc, const char * argv[])
{
VideoCapture capLeft(CamLeft);
bool opened = capLeft.isOpened();
if(!opened /*|| !capRight.isOpened()*/) // check if we succeeded
return -1;
Mat edges;
namedWindow(WIN_L,1);
for(;;)
{
Mat frameL;
Mat frameR;
capLeft >> frameL; // get a new frame from camera
cvtColor(frameL, edges, CV_RGB2RGBA);
imshow(WIN_L, edges);
if(waitKey(30) >= 0) break;
}
return 0;
}
So I am creating a window named "win_l" stands for window left and process video capture. It works well. Now I upgraded my code to support another camera like this:
int main(int argc, const char * argv[])
{
VideoCapture capLeft(CamLeft);
VideoCapture capRight(CamRight);
bool opened = capLeft.isOpened();
if(!opened /*|| !capRight.isOpened()*/) // check if we succeeded
return -1;
Mat edges;
namedWindow(WIN_L,1);
namedWindow(WIN_R,1);
for(;;)
{
Mat frameL;
Mat frameR;
capLeft >> frameL; // get a new frame from camera
cvtColor(frameL, edges, CV_RGB2RGBA);
imshow(WIN_L, edges);
imshow(WIN_R, edges);
if(waitKey(30) >= 0) break;
}
return 0;
}
But then I don't see the debugger hit this line: bool opened.... is it the correct way to take capture from 2 cameras?
I suspect USB bandwidth allocating problem. Not very sure if that is the cause though, but it is usually the case for 2 separate camera with individual USB cable each.
But still, do give it a shot. Some methods to overcome that problem: 1)put a Sleep(ms) in between the lines of your capture line. 2)Use lower resolution which would reduce the bandwidth used by each camera. 3)Use MJPEG format(compressed frames).
EDIT:
I came across this website, WEBCAM AND USB DRIVERS, thought you might like to read it.
Anyway, I am not sure if both cameras are able to run concurrently with each other only through one USB port and the reason is as mentioned(I actually forgotten about it. My bad):
USB: Universal Serial Bus, as the name suggest, it is a serial data transmission, meaning, that it can only send data from one device to/and from the computer.
What happening in your code is a deadlock, causing your program to not progress, even on one camera.
2 reasons that causes this:
1)I suspect that the driver for each camera is consistently overwriting each other, sending an I/O Request Packet (IRP), and the "pipe" keeps on receiving and processing each request, and the system keep deallocating each resource immediately after it is allocated. This goes on and on.
2)There is starvation from the 2nd camera, where if it is not allocated the resource, the program cannot progress. Hence the system is stuck in a loop where the 2nd camera keep requesting for the resource.
However, it would be weird if somebody invented this 2 webcam using a single serial bus when each camera needs a channel/pipe of it's own to communicate with the system. Maybe you would like to check with the manufacturer straight? Maybe they came up with a workaround, hence they invented this camera?
If not, the only purpose for this invention is pretty redundant. I can only think of a security purpose, one configured for night view and one configured for day view.
I'm building a skin-detection algorithm that takes constant, real-time feed with a webcam, converts it to a binary image (based on the skin color of the person's face), and filters out the noise by only showing focusing on the largest blobs (using CvBlobsLib). The output of my code, however, shows a lot of lag, and I'm not sure what to change to make it faster.
Here's (the important part of) my code:
Mat frame;
IplImage ipl, *res = new IplImage;
CBlobResult blobs;
CBlob *currentBlob;
cvNamedWindow("output");
for(;;){
cap >> frame; //get a new frame from camera
cvtColor(frame, lab, CV_BGR2Lab);//frame now in L*a*b*
inRange(lab, BW_MIN, BW_MAX, bw);//frame now only shows "skin values"...BW_MIN/BW_MAX determined earlier
ipl = bw; //IplImage header
blobs = CBlobResult(&ipl, NULL, 0);
blobs.Filter(blobs, B_EXCLUDE, CBlobGetArea(), B_LESS, 10000);
res = cvCreateImage(cvGetSize(&ipl), IPL_DEPTH_8U, 3);
cvMerge(&ipl, &ipl, &ipl, NULL, res);
cvShowImage("output", res);
if(waitKey(5) >= 0) break;
}
cvDestroyWindow("output");
I convert Mat to IplImage because CvBlobsLib only works with the IplImage type.
Does anyone see a way that I could make this faster? I've just recently heard other blob detection libraries do a better job with real-time video, but I'd be interested to see if there's something I'm simply overlooking in my code.
You can decrease the resolution of the camera capture using set method
set(CV_CAP_PROP_FRAME_WIDTH , double width)
and
set(CV_CAP_PROP_FRAME_HEIGHT , double height)
If your default capture resolution is too high, this can increase the detection speed considerably.
Hey guys,
I'm using OpenCV with the C++ API, and in order for my project to be more reliable I need a certain camera connection\disconnection handling.
I have searched for how-to's, but I could only find answers that require an ugly hack in order to do so.
Can you suggest a cleaner way to do it?
Thnx
Detecting camera connection/disconnection might require some tricks.
I suggest that you start another thread to check the success of cvCreateCameraCapture() in a loop, while your application is running.
Something like the following:
while (run_detection_thread) // global variable controlled by the main thread
{
CvCapture* capture = cvCreateCameraCapture(-1); //-1 or whatever number works for you
if (camera) //camera is connected
{
sleep(1);
}
else
{
// camera was disconnected
}
}
I think that I have a good workaround for this problem. I create an auxiliary Mat array with zeros with the same resolution like the output from camera. I assign it to Mat array to which just after is assign the frame captured from camera and at the end I check the norm of this array. If it is equal zero it means that there was no new frame captured from camera.
VideoCapture cap(0);
if(!cap.isOpened()) return -1;
Mat frame;
cap >> frame;
Mat emptyFrame = Mat::zeros(CV_CAP_PROP_FRAME_WIDTH, CV_CAP_PROP_FRAME_HEIGHT, CV_32F);
for(;;)
{
frame = emptyFrame;
cap >> frame;
if (norm(frame) == 0) break;
}