I've just completed the Udacity Parallel programming stage 2 course, and I'm now implementing what I've learnt into a basic app with OpenCV which applies a gaussian blur to a constant stream of images coming through a webcam.
I'm loading frames into a Mat object, and whilst in my loop I want to call a method gaussian_cpu, the only problem is it requires a uchar4 to be passed to both the input and output parameters. How would I convert a Mat object to uchar4?
// Keep processing frames - Do CPU First
while(cpu_frames > 0)
{
cout << cpu_frames << "\n";
camera >> frameIn;
gaussian_cpu(frameIn, frameOut, numRows(), numCols(), h_filter__, 9);
imshow("Source", frameIn);
imshow("Dest", frameOut);
// 2ms delay to prevent system from being interrupted whilst drawing the new frame
waitKey(2);
cpu_frames--;
}
My method signature then looks like this:
void gaussian_cpu(
const uchar4* const rgbaImage, // input image from the camera
uchar4* const outputImage, // The image we are writing back for display
size_t numRows, size_t numCols, // Width and Height of the input image (rows/cols)
const float* const filter, // The value of sigma
const int filterWidth // The size of the stencil (3x3) 9
)
I need to use uchar4 so I can split the channels, do my convolution and then recombine the channels to return the output image. Is there any way to do this?
opencv generally uses bgr, 3 channel Mats, but a basic:
Mat bgra;
cvtColor( frameIn, bgra, CV_BGR2BGRA );
will generate an (unused) 4th channel. now you probably have to allocate mem for you outputImage:
Mat frameOut( bgra.size(), bgra.type() );
then you can feed those into your gaussian_cpu():
int filterWidth=5;
float *filter = ... // your job, not mine ;)
gaussian_cpu( (uchar4*)(bgra.data), (uchar4*)(frameOut.data), bgra.rows, bgra.cols, filter, filterWidth );
Related
I have a Mipi camera that captures frames and stores them into the struct buffer that you can see below. Once the frame is stored I want to convert it into a cv::Mat, the thing is that the Mat ends up looking like the first pic.
The var buf.index is just part of the V4L2 API, useful to understand which buffer I'm using.
//The structure where the data is stored
struct buffer{
void *start;
size_t length;
};
struct buffer *buffers;
//buffer->mat
cv::Mat im = cv::Mat(cv::Size(width, height), CV_8UC3, ((uint8_t*)buffers[buf.index].start));
At first I thought that the data might be corrupted but storing the image with lodepng results in a nice image without any distortion.
unsigned char* out_buf = (unsigned char*)malloc( width * height * 3);
for(int pix = 0; pix < width*height; ++pix) {
memcpy(out_buf + pix*3, ((uint8_t*)buffers[buf.index].start)+4*pix+1, 3);
}
lodepng_encode24_file(filename, out_buf, width, height);
I bet it's something really silly.
the picture you post has oddly colored pixels and the patterns look like there's more information than simply 24 bits per pixel.
after inspecting the data, it appears that V4L gives you four bytes per pixel, and the first byte is always 0xFF (let's call that X). further, the channel order seems to be XRGB.
create a cv::Mat using 8UC4 to contain the data.
to use the picture in OpenCV, you need BGR order. cv::split the received data into its four color planes which are X,R,G,B. use cv::merge to reassemble the B,G,R planes into a picture that OpenCV can handle, or reassemble into R,G,B to create a Mat for other purposes (that other library you seem to use).
Is there a way to convert from RGB to YUYV (YUY 4:2:2) format? I noted that OpenCV has reverse operation, but not RGB to YUYV for some reason. Maybe someone can point to code which does that (even outside of OpenCV library)?
UPDATE
I found libyuv library which may work for this purpose by doing BGR to ARGB conversion and then ARGB to YUY2 format (hopefully this is the same as YUYV 4:2:2). But it doesn't seem to work. Do you happen to know what yuyv buffer dimensions/type should look like? What its stride?
To clarify YUYV and YUY2 are the same formats if it helps.
UPDATE 2
Here is my code of using libyuv library:
Mat frame;
// Convert original image im from BGR to BGRA for further use in libyuv
cvtColor(im, frame, CVX_BGR2BGRA);
// Actually libyuv requires ARGB (i.e. reverse of BGRA), so I swap channels here
int from_to[] = { 0,3, 1,2, 2,1, 3,0 };
mixChannels(&frame, 1, &frame, 1, from_to, 4);
// This is the most confusing part. Not sure what argb_stride suppose to be - length of a row in bytes or size of single value in the array?
const uint8_t* argb_data = frame.data;
int argb_stride = 8;
// Also it is not clear what size of yuyv frame should be since we duplicate one Y
Mat yuyv(frame.rows, frame.cols, CVX_8UC2);
uint8_t* yuyv_data = yuyv.data;
int yuyv_stride = 16;
// Do actual conversion
libyuv::ARGBToYUY2(argb_data, argb_stride, yuyv_data, yuyv_stride,
frame.cols, frame.rows);
// Then I feed yuyv_data to video stream buffer and see green or purple image instead of video stream.
UPDATE 3
Mat frame;
cvtColor(im, frame, CVX_BGR2BGRA);
// ARGB
int from_to[] = { 0,3, 1,2, 2,1, 3,0 };
Mat rgba(frame.size(), frame.type());
mixChannels(&frame, 1, &rgba, 1, from_to, 4);
const uint8_t* argb_data = rgba.data;
int argb_stride = rgba.cols*4;
Mat yuyv(rgba.rows, rgba.cols, CVX_8UC2);
uint8_t* yuyv_data = yuyv.data;
int yuyv_stride = width * 2;
int res = libyuv::ARGBToYUY2(argb_data, argb_stride, yuyv_data, yuyv_stride, rgba.cols, rgba.rows);
It appears that although method is called ARGBToYUY2 it requires BGRA order of channels (not reverse).
I'm new with OpenCV library, and I would like to use it to detect circles in a video stream captured from an iPad's back camera. I figured out how to do it and with OpenCV 2.4.2, it can be done in less than 10 lines of code. But it doesn't work for me, and I think I missed something because of some weird behaviours I obtain.
The code is very simple and begins in the Objective-C callback triggers each time a new frame is captured by the camera. Here is what I do in this callback:
- (void)captureOutput:(AVCaptureOutput *)captureOutput
didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer
fromConnection:(AVCaptureConnection *)connection
{
// Convert CMSampleBufferRef to CVImageBufferRef
CVImageBufferRef imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer);
// Lock pixel buffer
CVPixelBufferLockBaseAddress(imageBuffer, kCVPixelBufferLock_ReadOnly);
// Construct VideoFrame struct
uint8_t *baseAddress = (uint8_t*)CVPixelBufferGetBaseAddress(imageBuffer);
size_t width = CVPixelBufferGetWidth(imageBuffer);
size_t height = CVPixelBufferGetHeight(imageBuffer);
size_t stride = CVPixelBufferGetBytesPerRow(imageBuffer);
// Unlock pixel buffer
CVPixelBufferUnlockBaseAddress(imageBuffer, 0);
std::vector<unsigned char> data(baseAddress, baseAddress + (stride * height));
// Call C++ function with these arguments => (data, (int)width, (int)height)
}
And here is the C++ function that process the image with OpenCV:
void proccessImage(std::vector<unsigned char>& imageData, int width, int height)
{
// Create cv::Mat from std::vector<unsigned char>
Mat src(width, height, CV_8UC4, const_cast<unsigned char*>(imageData.data()));
Mat final;
// Draw a circle at position (300, 200) with a radius of 30
cv::Point center(300, 200);
circle(src, center, 30.f, CV_RGB(0, 0, 255), 3, 8, 0);
// Convert the gray image to RGBA
cvtColor(src, final, CV_BGRA2RGBA);
// Reform the std::vector from cv::Mat data
std::vector<unsigned char> array;
array.assign((unsigned char*)final.datastart, (unsigned char*)final.dataend);
// Send final image data to GPU and draw it
}
The image retrieve from iPad's back camera is in BGRA (32 bits) format.
What I expected was an image from the iPad's back camera with a simple circle drawn at the position x = 300px, y = 200px and with a radius of 30px.
And this is what I got: http://i.stack.imgur.com/bWfwa.jpg
Do you know what is wrong with my code?
Thanks in advance.
Thanks for your help, I finally figured out what happen, and it's my entire fault...
When you create a new Mat you need to pass it the image's height as first argument, and not width. The circle is drawn properly if I switch the arguments.
I'm currently using the following function to apply my alpha channels (stored as separate GRAY cv::Mats), to images:
void percepUnit::applyAlpha() {
int x,y,w,h;
/*vector<cv::Mat> channels;
if (image.rows == mask.rows and image.cols == mask.cols) {
cv::split(image,channels); // break image into channels
channels.push_back(mask); // append alpha channel
cv::merge(channels,alphaImage); // combine channels
}*/
// Avoid merge
cv::Mat src[] = {this->image, this->mask};
int from_to[] = {0,0, 1,1, 2,2, 3,3};
this->alphaImage = Mat(image.rows, image.cols, CV_8UC4);
cv::mixChannels(src, 2, &(this->alphaImage), 1, from_to, 4); // &(*alphaImage)?
}
I've had to increase the resolution of the cv::Mats to 1280x720 (due to: How to replace an instance with another instance via pointer?) and now this function is running quite slowly, using up almost 50% of what is already a heavy meanshift segmentation application.
Any suggestions on how to apply these alpha channels faster? I'm running OpenCV with GPU, if you have any GPU based solutions.)
I ended up doing split / merge on the GPU:
void percepUnit::applyAlpha() {
cv::gpu::GpuMat tmpImage, tmpMask, tmpAlphaImage;
std::vector<cv::gpu::GpuMat> channels;
tmpImage.upload(this->image);
tmpMask.upload(this->mask);
cv::gpu::split(tmpImage,channels); // break image into channels
channels.push_back(tmpMask); // append alpha channel
cv::gpu::merge(channels,tmpAlphaImage); // combine channels
tmpAlphaImage.download(this->alphaImage);
tmpAlphaImage.release();
tmpImage.release();
tmpMask.release();
channels[0].release();
channels[1].release();
channels[2].release();
}
I get this error while trying to run OpenCV C++ code using qtcreator and the code used to
add some features to video file
opencv error :assertion failed <fixed type<> ::<<Mat*>obj>->type<>==m type> in create file c:\opencv\sources\mogules\core\src\matrix.cpp
I tried to add file dWidth and dHeight but it did not work.
int main(){
Mat frame;
Mat image;
VideoCapture cap("file source");
namedWindow("window",1);
while(1){
cap>>frame;
GpyrTempIIR g;
g.processOnFrame(frame,image); //this is the process i do on frame
imshow("window",image);
waitKey(33);
}
}
void GpyrTempIIR::processOnFrame(const Mat& src, Mat& out) {
src.convertTo(src,CV_32F); //Convert to Float
resize(srcFloat,blurred,blurredsize,0,0,CV_INTER_AREA);
/* Method in openCv to resize a video
* INTER_AREA is a fast method that gets the average of several pixels, which
is good for shrinking an image but not so good for enlarging an image. */
if(first){
first=false;
blurred.copyTo(LowPassHigh); // Using Storing Method
blurred.copyTo(LowPassLow);
src.copyTo(out);
} else {
//apply Temporal filter substraction of two IIR LowPass Filters
LowPassHigh = LowPassHigh * (1-fHigh) + fHigh * blurred;
LowPassLow = LowPassLow * (1-fLow) + fLow * blurred;
blurred = LowPassHigh - LowPassLow;
blurred*=alpha; // amplify , multuplying by alpha value
resize(blurred, outFloat, src.size(), 0, 0, CV_INTER_LINEAR); // resize back
outFloat += srcFloat; // add back to original frame
outFloat.convertTo(out, CV_8U); // convert to 8 bit
}
}