I have been working on a simple video-over-ip program, partly for use in a project and partly to teach myself some basics of networking using high-level interfaces. The trouble is that I can send the data from a cv::Mat over the network just fine, but once I attempt to decode the data, it appears to be missing much of the color data. The code is in this gist, which contains all the files necessary to build and run the project under Linux. Can anyone shine some light on this?
If you need any more information, let me know. You'll need a webcam to test, I'm afraid.
When you copy your data with memcpy(m.data(), frame.data, frame.rows * frame.cols);, you're only copying a third of the total data since your image is a 3-channel one.
Try to change it to memcpy(m.data(), frame.data, 3 * frame.rows * frame.cols); (and allocate enough space before).
Related
I am an amateur in video/image processing but I am trying to create an app for HD video calling. I hope someone would see where I may be doing wrong and guide me on the right path. Here is what I am doing and what I think I understand, please correct me if you know better.
I am using OpenCV currently to grab an image from my webcam in a DLL. (I will be using this image for other things later)
Currently, the image that opencv gives me is a Opencv::Mat. I resized this and converted to a byte array size of a 720p image, which is about 3 Megapixels.
I pass this ptr back to my C# code then I can now render this onto a texture.
Now I created a TCP socket and connect the server and client and start to transmit previously gotten image byte array. I am able to transmit the byte array over to the client then I use the GPU to render it to a texture.
Currently, there is a big delay of about 4-500ms delay. This is after I tried compressing the buffer with gzipstream for unity. It was able to compress the byte array from about 3 million bytes to 1.5 million. I am trying to get this to smallest as possible and also fastest as possible but this is where I am completely lost. I saw that Skype requires only 1.2Mbps connection for a 720p video calling at 22 fps. I have no idea how they can achieve such a small frame, but of course I don't need it to be that small. I need to be at least decent.
Please give me a lecture on how this can be done! And let me know if you need anything else from me.
I found a link that may be very useful to anyone working on something similar. https://www.cs.utexas.edu/~teammco/misc/udp_video/
https://github.com/chenxiaoqino/udp-image-streaming/
I'm using OpenCV to convert image data captured using an IDS uEye camera into a useful format, using the following code:
IplImage* tmpImg = cvCreateImage(cvSize(width,height),IPL_DEPTH_8U,3);
tmpImg->imageData = pFrameBuffer[k];
frame = cv::cvarrToMat(tmpImg);
This works perfectly - I can then use imwrite(filename,frame); further downstream to write the processed images out as a sensible format. I would ideally like to be able to save the RGB channels as separate 'grayscale' image files, but I don't understand the OpenCV documentation regarding single-channel operations. Can anyone suggest a means of accomplishing this? Preferably it's not overly computationally expensive (looping over an image pixel-by pixel isn't an option - I'm working with 60-230fps video at up to 1280x1064, and all the processing has to be done at the point of capture).
Running the latest Debian Testing if that makes any difference (I don't think it should).
Once you have a cv::Mat object it's pretty simple:
std::vector<cv::Mat> grayPlanes;
cv::split(frame, grayPlanes);
cv::imwrite("blue.png", grayPlanes[0]);
cv::imwrite("green.png", grayPlanes[1]);
cv::imwrite("red.png", grayPlanes[2]);
The split function can directly write to a standard vector and you don't really have to think about memory management and other stuff.
My aim is to capture all the frames (RGB) from Kinect at 30 fps and save them to my hard drive. For doing this I took the following approach.
Get the frames from Kinect and store them in an array buffer. Since writing to disk (using imwrite()) takes a bit of time and I may miss some frames while doing so, so instead of directly saving them to the disk, I store them in an array. Now, I have another parallel thread that accesses this array and writes the individual frames to the disk as images.
Now I have used a static array of size 3000 and type Mat. This will suffice since I need to store frames for 1.5 minute videos (1.5 minutes = 2700 frames). I have declared the array as follows :
#define NUM_FRAMES 3000
Mat rgb[NUM_FRAMES];
I have already tested this limit by reading images and saving them to the array using the following code :
for(int i=0; i<NUM_FRAMES; i++)
{
Mat img = imread("image.jpg", CV_LOAD_IMAGE_COLOR);
rgb[i] = img;
imshow("Image", img);
cvWaitKey(10);
}
The above code executed flawlessly.
But one problem is that the code I am using for capturing image using Kinect, captures the image in an IplImage. Thus I need to convert the image to cv::Mat format before using it. I convert it using the following command:
IplImage* color = cvCreateImageHeader(cvSize(COLOR_WIDTH, COLOR_HEIGHT), IPL_DEPTH_8U, 4);
cvSetData(color, colorBuffer, colorLockedRect.Pitch); // colorBuffer and colorLockedRect.Pitch is something that Kinect uses. Not related to OpenCv
rgb[rgb_read++] = Mat(color, FLAG);
Now here lies my problem. Whenever I am setting #define FLAG true, it causes memory leaks and gives me OpenCv Error: Insufficient memory (failed to allocate 1228804 bytes) error.
But if I use #define FLAG false it works correctly, but the frames that I am getting is erroneous as shown below. They are three consecutive frames.
I was moving around my arm and the image got cut in between as can be seen from above.
Can someone please point out the reason for this weird behavior or any other alternate way of obtaining the desired result. I have been struggling with this since a few days now. Please ask for if any further clarifications are required.
I am using OpenCV 2.4.8, Kinect SDK for Windows version-1.8.0 and Microsoft Visual Studio 2010.
Also can someone please explan to me the role of the CopyData parameter in Mat::Mat. I have already gone through this link, but still it's not completely clear. Maybe that's why I could not solve the above error in the first place since it's working is not very clear.
Thanks in advance.
first, do not use IplImages, stick with cv::Mat, please.
the equivalent code for that would be:
Mat img_borrowed = Mat( height, width, CV_8U4C, colorBuffer, colorLockedRect.Pitch );
note, that this does not do any allocation on its own, it's still the kinect's pixels, so you will have to clone() it:
rgb[rgb_read++] = img_borrowed.clone();
this is the same as setting the flag in your code above to 'true'. (deep-copy the data)
[edit] maybe it's a good idea to skip the useless 4th channel (also less mem required), so , instead of the above you could do:
cvtColor( img_borrowed, rgb[rgb_read++], CV_BGRA2BGR); // will make a 'deep copy', too.
so, - here's the bummer: if you don't save a deep-copy in your array, you'll end up with garbled (and all the same!) images, probably even with undefined behaviour due to the locking/unlocking of the kinect buffer, if you do copy it (and you must), you will need a lot of memory.
unlikely, that you can keep 3000 *1024*786*4 = 9658368000 bytes in memory, you'll have to cut it down one way or another.
I have an bumblebee2 and I'm using the flycapture SDK to capture the incoming images. I then convert the left flycapture2 image to an openCV format so I can do some basic manipulations to it. Then I'd like to feed it back into the Flycapture SDK but I cant seem to figure out how. To convert from Flycapture to OpenCV I do the following:
FlyCapture2::Image cf2Img;
grabbedImage.Convert(FlyCapture2::PIXEL_FORMAT_BGR, &cf2Img );
unsigned int rowBytes = (double)cf2Img.GetReceivedDataSize()/(double)cf2Img.GetRows();
cv::Mat cvImage = cv::Mat( cf2Img.GetRows(), cf2Img.GetCols(), CV_8UC3, cf2Img.GetData(), rowBytes );
I then do my manipulations (thresholding/contour detections/background removal etc), I'd then like to feed this image back into Flycapture. My attempts at converting it back haven't worked.
Does anyone have any code they have used before to take an OpenCV format back to Flycapture?
I work for Point Grey and I'll try to help here. Please note though you can contact us directly via our support site at ptgrey.com/support/ and we can help you out as well.
Looking at the code you attached and looking at the openCV source, when you make the cvImage, you are just reassigning the pointer to the data, you are not making an actual copy of the data.
So as long as the size of the data stays the same (ie. you keep it at 24 bits per pixel), any changes you make to the openCV image should be reflected in the flycapture (cf2Img) data and be able to save properly.
If you can explain the problems you are having trying to move back to a flycapture image, or send us the source code of how you are doing that, we can help you further.
To summarize, I expect any manipulations you do to the cvImage after the code you have provided should just be reflected in cf2Img without the need to convert back, assuming you are not changing the bit depth of the image.
I hope that helps, but please let me know if I can help clarify anything or if you can provide an example of the failure to convert back to fc2.
Thank you,
Point Grey Support
I am starting to learn CUDA GPU programming from Udacity video course (course is 2 yrs old). I am using CUDA 5.5 with Visual Studio Express 2012 (students edition, so not all features of CUDA debugging is not available) on Nvidia GeForce GT 630M GPU.
Just implemented some vector addition and other simple operations.
Now I am trying to convert a RGB image to Grayscale. I am reading image with help of OpenCV. (Anyway I failed whatever methods I tried. That is why I am here)
Below is my .cpp file : https://gist.github.com/abidrahmank/7020863
Below is my .cu file : https://gist.github.com/abidrahmank/7020910
My input image is a simple 64x64 color image (Actually I used 512x512 image first, didn't work, so brought down to 64x64 to check if that is the problem. It doesn't seem so)
Problem
My output image of CUDA implementation is a white image. All value 255. Somewhere here and there, there are some gray pixels, may be less than 1%. Remaining everything is white.
What I tried:
For three days, I tried following things:
I thought problem may be due image size, so that number of threads may not be optimal or something like that, So reduced image size. Still same result.
I tried a similar example, created a 64x64 array. Take its two pixels at a time, and find the square of their sums, and it worked fine. Here is the code : https://gist.github.com/abidrahmank/7021023
Started checking data one-by-one at each stage. Input image just before loading to GPU is fine. But input data, when I checked inside kernel, is always 255. (Check line 14 here)
Finally I set all GPU data to zero using CudaMemset and checked input data inside kernel, it is still 255.
So I don't have any other option to do other asking at StackOverflow.
Can anyone tell me what is the mistake I am making?
Your kernel signature says:
__global__ void kernel(unsigned char* d_in, unsigned char* d_out)
But you call it like:
kernel<<<rows,cols>>>(d_out, d_in);
Which one is in and which one is out?
Having done quite a bit of CUDA programming in the past, I would strongly recommend that you use Thrust instead of hand-crafting kernels. Even thrust::for_each is hard to beat with raw kernels.
Besides the parameter issue indicated by DanielKO, you also have problems on thread/block settings.
Since you've already treat your 2-D image as a 1-D array, here's a good example showing how to set thread/block for data with arbitrary size.
https://developer.nvidia.com/content/easy-introduction-cuda-c-and-c