I'm new with OpenCV library, and I would like to use it to detect circles in a video stream captured from an iPad's back camera. I figured out how to do it and with OpenCV 2.4.2, it can be done in less than 10 lines of code. But it doesn't work for me, and I think I missed something because of some weird behaviours I obtain.
The code is very simple and begins in the Objective-C callback triggers each time a new frame is captured by the camera. Here is what I do in this callback:
- (void)captureOutput:(AVCaptureOutput *)captureOutput
didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer
fromConnection:(AVCaptureConnection *)connection
{
// Convert CMSampleBufferRef to CVImageBufferRef
CVImageBufferRef imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer);
// Lock pixel buffer
CVPixelBufferLockBaseAddress(imageBuffer, kCVPixelBufferLock_ReadOnly);
// Construct VideoFrame struct
uint8_t *baseAddress = (uint8_t*)CVPixelBufferGetBaseAddress(imageBuffer);
size_t width = CVPixelBufferGetWidth(imageBuffer);
size_t height = CVPixelBufferGetHeight(imageBuffer);
size_t stride = CVPixelBufferGetBytesPerRow(imageBuffer);
// Unlock pixel buffer
CVPixelBufferUnlockBaseAddress(imageBuffer, 0);
std::vector<unsigned char> data(baseAddress, baseAddress + (stride * height));
// Call C++ function with these arguments => (data, (int)width, (int)height)
}
And here is the C++ function that process the image with OpenCV:
void proccessImage(std::vector<unsigned char>& imageData, int width, int height)
{
// Create cv::Mat from std::vector<unsigned char>
Mat src(width, height, CV_8UC4, const_cast<unsigned char*>(imageData.data()));
Mat final;
// Draw a circle at position (300, 200) with a radius of 30
cv::Point center(300, 200);
circle(src, center, 30.f, CV_RGB(0, 0, 255), 3, 8, 0);
// Convert the gray image to RGBA
cvtColor(src, final, CV_BGRA2RGBA);
// Reform the std::vector from cv::Mat data
std::vector<unsigned char> array;
array.assign((unsigned char*)final.datastart, (unsigned char*)final.dataend);
// Send final image data to GPU and draw it
}
The image retrieve from iPad's back camera is in BGRA (32 bits) format.
What I expected was an image from the iPad's back camera with a simple circle drawn at the position x = 300px, y = 200px and with a radius of 30px.
And this is what I got: http://i.stack.imgur.com/bWfwa.jpg
Do you know what is wrong with my code?
Thanks in advance.
Thanks for your help, I finally figured out what happen, and it's my entire fault...
When you create a new Mat you need to pass it the image's height as first argument, and not width. The circle is drawn properly if I switch the arguments.
Related
I have a Mipi camera that captures frames and stores them into the struct buffer that you can see below. Once the frame is stored I want to convert it into a cv::Mat, the thing is that the Mat ends up looking like the first pic.
The var buf.index is just part of the V4L2 API, useful to understand which buffer I'm using.
//The structure where the data is stored
struct buffer{
void *start;
size_t length;
};
struct buffer *buffers;
//buffer->mat
cv::Mat im = cv::Mat(cv::Size(width, height), CV_8UC3, ((uint8_t*)buffers[buf.index].start));
At first I thought that the data might be corrupted but storing the image with lodepng results in a nice image without any distortion.
unsigned char* out_buf = (unsigned char*)malloc( width * height * 3);
for(int pix = 0; pix < width*height; ++pix) {
memcpy(out_buf + pix*3, ((uint8_t*)buffers[buf.index].start)+4*pix+1, 3);
}
lodepng_encode24_file(filename, out_buf, width, height);
I bet it's something really silly.
the picture you post has oddly colored pixels and the patterns look like there's more information than simply 24 bits per pixel.
after inspecting the data, it appears that V4L gives you four bytes per pixel, and the first byte is always 0xFF (let's call that X). further, the channel order seems to be XRGB.
create a cv::Mat using 8UC4 to contain the data.
to use the picture in OpenCV, you need BGR order. cv::split the received data into its four color planes which are X,R,G,B. use cv::merge to reassemble the B,G,R planes into a picture that OpenCV can handle, or reassemble into R,G,B to create a Mat for other purposes (that other library you seem to use).
I've just completed the Udacity Parallel programming stage 2 course, and I'm now implementing what I've learnt into a basic app with OpenCV which applies a gaussian blur to a constant stream of images coming through a webcam.
I'm loading frames into a Mat object, and whilst in my loop I want to call a method gaussian_cpu, the only problem is it requires a uchar4 to be passed to both the input and output parameters. How would I convert a Mat object to uchar4?
// Keep processing frames - Do CPU First
while(cpu_frames > 0)
{
cout << cpu_frames << "\n";
camera >> frameIn;
gaussian_cpu(frameIn, frameOut, numRows(), numCols(), h_filter__, 9);
imshow("Source", frameIn);
imshow("Dest", frameOut);
// 2ms delay to prevent system from being interrupted whilst drawing the new frame
waitKey(2);
cpu_frames--;
}
My method signature then looks like this:
void gaussian_cpu(
const uchar4* const rgbaImage, // input image from the camera
uchar4* const outputImage, // The image we are writing back for display
size_t numRows, size_t numCols, // Width and Height of the input image (rows/cols)
const float* const filter, // The value of sigma
const int filterWidth // The size of the stencil (3x3) 9
)
I need to use uchar4 so I can split the channels, do my convolution and then recombine the channels to return the output image. Is there any way to do this?
opencv generally uses bgr, 3 channel Mats, but a basic:
Mat bgra;
cvtColor( frameIn, bgra, CV_BGR2BGRA );
will generate an (unused) 4th channel. now you probably have to allocate mem for you outputImage:
Mat frameOut( bgra.size(), bgra.type() );
then you can feed those into your gaussian_cpu():
int filterWidth=5;
float *filter = ... // your job, not mine ;)
gaussian_cpu( (uchar4*)(bgra.data), (uchar4*)(frameOut.data), bgra.rows, bgra.cols, filter, filterWidth );
I am working on a project with OpenFrameworks using ofxCV, ofxOpencv and ofxColorQuantizer. Technically, the project is analyzing live video captured via webcam and analysis's the image in real time to gather and output the most prominent color in the current frame. When generating the most prominent color I am using the pixel difference between the current frame and the previous frame to generate the what colors have updated and use the updated or moving areas of the video frame to figure out the most prominent colors.
The reason for using the pixel difference's to generate the color pallet is because I want to solve for the case of a user walks into the video frame, I want try and gather the color pallet of the person, for instance what they are wearing. For example red shirt, blue pants will be in the pallet and the white background will be excluded.
I have a strong background in Javascript and canvas but am fairly new to OpenFrameworks and C++ which is why I think I am running into a roadblock with this problem I described above.
Along with OpenFrameworks I am using ofxCV, ofxOpencv and ofxColorQuantizer as tools for this installation. I am taking a webcam image than making it a cv:Mat than using pyrdown on the webcam image twice followed by a absdiff of the mat which I am than trying to pass the mat into the ofxColorQuantizer. This is where I think I am running into problems — I don't think the ofxColorQuantizer likes the mat format of the image I am trying to use. I've tried looking for the different image format to try and convert the image to to solve this issue but I haven't been able to come to solution.
For efficiencies I am hoping to to the color difference and color prominence calculations on the smaller image (after I pyrdown' the image) and display the full image on the screen and the generated color palette is displayed at the bottom left like in the ofxColorQuantizer example.
I think there may be other ways to speed up the code but at the moment I am trying to get this portion of the app working first.
I have my main.cpp set up as follows:
#include "ofMain.h"
#include "ofApp.h"
#include "ofAppGlutWindow.h"
//========================================================================
int main( ){
ofAppGlutWindow window;
ofSetupOpenGL(&window, 1024,768, OF_WINDOW); // <-------- setup the GL context
// ofSetupOpenGL(1024,768,OF_WINDOW); // <-------- setup the GL context
// this kicks off the running of my app
// can be OF_WINDOW or OF_FULLSCREEN
// pass in width and height too:
ofRunApp(new ofApp());
}
My ofApp.h file is as follows:
#pragma once
#include "ofMain.h"
#include "ofxOpenCv.h"
#include "ofxCv.h"
#include "ofxColorQuantizer.h"
class ofApp : public ofBaseApp{
public:
void setup();
void update();
void draw();
ofVideoGrabber cam;
ofPixels previous;
ofImage diff;
void kMeansTest();
ofImage image;
ofImage img;
cv::Mat matA, matB;
ofImage diffCopy;
ofImage outputImage;
ofxCv::RunningBackground background;
ofxColorQuantizer colorQuantizer;
// a scalar is like an ofVec4f but normally used for storing color information
cv::Scalar diffMean;
};
And finally my ofApp.cpp is below:
#include "ofApp.h"
using namespace ofxCv;
using namespace cv;
//--------------------------------------------------------------
void ofApp::setup(){
ofSetVerticalSync(true);
cam.initGrabber(320, 240);
// get our colors
colorQuantizer.setNumColors(3);
// resize the window to match the image
// ofSetWindowShape(image.getWidth(), image.getHeight());
ofSetWindowShape(800, 600);
// imitate() will set up previous and diff
// so they have the same size and type as cam
imitate(previous, cam);
imitate(diff, cam);
imitate(previous, outputImage);
imitate(diff, outputImage);
}
//--------------------------------------------------------------
void ofApp::update(){
cam.update();
if(cam.isFrameNew()) {
matA = ofxCv::toCv(cam.getPixelsRef());
ofxCv::pyrDown(matA, matB);
ofxCv::pyrDown(matB, matA);
ofxCv::medianBlur(matA, 3);
ofxCv::toOf(matA, outputImage);
// take the absolute difference of prev and cam and save it inside diff
absdiff(previous, outputImage, diff);
}
}
//--------------------------------------------------------------
void ofApp::draw(){
// If the image is ready to draw, then draw it
if(outputImage.isAllocated()) {
outputImage.update();
outputImage.draw(0, 0, ofGetWidth(), ofGetHeight());
}
ofBackground(100,100,100);
ofSetColor(255);
ofImage diffCopy;
diffCopy = diff;
diffCopy.resize(diffCopy.getWidth()/2, diffCopy.getHeight()/2);
// there is some sort of bug / issue going on here...
// prevent the app from compiling
// comment out to run and see blank page
colorQuantizer.quantize(diffCopy.getPixelsRef());
ofLog() << "the number is " << outputImage.getHeight();
ofLog() << "the number is " << diffCopy.getHeight();
ofSetColor(255);
img.update();
// cam.draw(0, 0, 800, 600);
outputImage.draw(0, 0, 800, 600);
// colorQuantizer.draw(ofPoint(0, cam.getHeight()-20));
colorQuantizer.draw(ofPoint(0, 600-20));
// use the [] operator to get elements from a Scalar
float diffRed = diffMean[0];
float diffGreen = diffMean[1];
float diffBlue = diffMean[2];
ofSetColor(255, 0, 0);
ofRect(0, 0, diffRed, 10);
ofSetColor(0, 255, 0);
ofRect(0, 15, diffGreen, 10);
ofSetColor(0, 0, 255);
ofRect(0, 30, diffBlue, 10);
}
//--------------------------------------------------------------
void ofApp::kMeansTest(){
cv::Mat samples = (cv::Mat_<float>(8, 1) << 31 , 2 , 10 , 11 , 25 , 27, 2, 1);
cv::Mat labels;
// double kmeans(const Mat& samples, int clusterCount, Mat& labels,
cv::TermCriteria termcrit;
int attempts, flags;
cv::Mat centers;
double compactness = cv::kmeans(samples, 3, labels, cv::TermCriteria(), 2, cv::KMEANS_PP_CENTERS, centers);
cout<<"labels:"<<endl;
for(int i = 0; i < labels.rows; ++i)
{
cout<<labels.at<int>(0, i)<<endl;
}
cout<<"\ncenters:"<<endl;
for(int i = 0; i < centers.rows; ++i)
{
cout<<centers.at<float>(0, i)<<endl;
}
cout<<"\ncompactness: "<<compactness<<endl;
}
Apologies in advance for the state of my code — it's getting late and I'm trying to get this done.
My question is what is the image format openFrameworks is using for grabbing the webcam image, what is the image format that openCV expects and what should I use to switch back from a mat image to an ofImage and is there a way to getPixelsRef from a mat image?
The area of code that I think I have something wrong is the following logic.
I have this line of code which gets the video frame from the webcam matA = ofxCv::toCv(cam.getPixelsRef());
Than do a couple ofxCv procedures on the frame such as ofxCv::pyrDown(matA, matB); which I think changes the image format or pixel format of the frame
Than I convert the frame back to OF with ofxCv::toOf(matA, outputImage);,
Next I get the difference in the pixels between the current frame and the last frame, create a copy of the difference between the two frames. Potentially the issue lies here with the diff output image format
Which I pass the diff copy to colorQuantizer.quantize(diffCopy.getPixelsRef()); to try and generate the color palette in for the change in pixels.
It is the colorQuantizer class and function call that is giving me an error which reads thread error [ error ] ofTexture: allocate(): ofTextureData has 0 width and/or height: 0x0
with an EXC_BAD_ACCESS
And lastly, could there be an alternative cause for the exc_bad_access thread error rather than image formatting? Being new to c++ I'm just guessing and going off instinct of what I think the rood cause of my problem is.
Many thanks.
I'm currently using the following function to apply my alpha channels (stored as separate GRAY cv::Mats), to images:
void percepUnit::applyAlpha() {
int x,y,w,h;
/*vector<cv::Mat> channels;
if (image.rows == mask.rows and image.cols == mask.cols) {
cv::split(image,channels); // break image into channels
channels.push_back(mask); // append alpha channel
cv::merge(channels,alphaImage); // combine channels
}*/
// Avoid merge
cv::Mat src[] = {this->image, this->mask};
int from_to[] = {0,0, 1,1, 2,2, 3,3};
this->alphaImage = Mat(image.rows, image.cols, CV_8UC4);
cv::mixChannels(src, 2, &(this->alphaImage), 1, from_to, 4); // &(*alphaImage)?
}
I've had to increase the resolution of the cv::Mats to 1280x720 (due to: How to replace an instance with another instance via pointer?) and now this function is running quite slowly, using up almost 50% of what is already a heavy meanshift segmentation application.
Any suggestions on how to apply these alpha channels faster? I'm running OpenCV with GPU, if you have any GPU based solutions.)
I ended up doing split / merge on the GPU:
void percepUnit::applyAlpha() {
cv::gpu::GpuMat tmpImage, tmpMask, tmpAlphaImage;
std::vector<cv::gpu::GpuMat> channels;
tmpImage.upload(this->image);
tmpMask.upload(this->mask);
cv::gpu::split(tmpImage,channels); // break image into channels
channels.push_back(tmpMask); // append alpha channel
cv::gpu::merge(channels,tmpAlphaImage); // combine channels
tmpAlphaImage.download(this->alphaImage);
tmpAlphaImage.release();
tmpImage.release();
tmpMask.release();
channels[0].release();
channels[1].release();
channels[2].release();
}
I got this image and I'd like to fill the upper left black area with white, but all I get is a completely white image … Any ideas what's wrong with my code?
Code:
...
cv::Rect rect;
roi = cv::floodFill(roi, cv::Point(1,1), cv::Scalar(0), &rect, cv::Scalar(0), cv::Scalar(0), 4);
...
Input image:
This is the image I get with the following code:
int main()
{
cv::Mat image = cv::imread("TF2XE.jpg", -1);
cv::imshow("image before filling", image);
int filling = cv::floodFill(image, cv::Point(0,0), 255, (cv::Rect*)0, cv::Scalar(), 200);
cv::imshow("image after filling", image);
cv::waitKey();
return 0;
}
Notice that I used 200 as upDiff parameter, since if you set it to 0 there will be some gray pixels that will not be considered inside the connected component, change that if that is indeed what you want.
Not sure, but according to the documentation I was able to dig up, it says that cv::floodFill() returns an int. So assuming that roi is a matrix and the openCV matrix class defines operator= for int parameters, you could be assigning some int to each element of the matrix.