OpenCV OpenGLDrawCallback doesn't get called - c++

I initialize OpenCV window with following code:
cv::VideoCapture * stream = new cv::VideoCapture("stream_ip");
if (!stream->isOpened()){
printf("Couldn't open stream! %s\n", strerror(errno));
}
//We create window with OpenGL enabled.
cv::namedWindow("rtsp_stream", cv::WINDOW_OPENGL);
//Make it fullscreen (I also tried with fixed screen size without luck.)
cv::setWindowProperty("rtsp_stream", cv::WND_PROP_FULLSCREEN, cv::WINDOW_FULLSCREEN);
//Set OpenGL context to use this window.
cv::setOpenGlContext("rtsp_stream");
//Set openGlDrawCallback.
cv::setOpenGlDrawCallback("rtsp_stream", on_opengl, NULL);
//This is the material that the image will be rendered on.
cv::Mat frame;
char k;
bool continueStream = true;
while (continueStream) {
//We read data from the stream and write it on the frame.
if((stream->read(frame)) != 0){
//Then we display/render the image using imshow.
cv::imshow("rtsp_stream", frame);
k = cv::waitKey(1);
//I'm not sure if updateWindow needs to be manually called to make openGLDrawCallback or if imshow calls it automatically after done rendering. So I have tried with and without it.
//cv::updateWindow("rtsp_stream");
switch(k){
case 0x1b: //ESC key
printf("Closing stream.\n");
continueStream = false;
break;
}
}
}
open_gl function just simply fprints some text to see if opengldrawcallback gets called. I have also made sure I have OpenGL and QT enabled on my OpenCV with std::cout << cv::getBuildInformation() << std::endl;. I have tried to find solutions from many different websites and sources, including this book. Program works as expected in every manner, except that openglcallback never gets called. All the help is much appreciated.

Okay. so as I yesterday found out, cv::imshow prevented me from using OpenGL commands on that window.
So what I had to do to overcome this, was to read the data on a cv::Mat, just like before. But instead of rendering that cv::Mat straight on the window using cv::imshow, I had to first store the data on a texture and then render that texture on screen. To store the data on a texture I used the following method:
/**
*texture: Pointer to OpenGL texture that we want to render our stream on.
*data: cv::Mat that contains the data we want to render.
**/
void storeStreamToTexture(GLuint texture, cv::Mat* data){
//Bind the texture we want to render to.
glBindTexture(GL_TEXTURE_2D, texture);
//we flip the Mat to start reading from the beginning.
cv::flip(*data, *data, 0);
//Store mat data to our texture.
glTexImage2D(GL_TEXTURE_2D, 0, GL_RGB, data->cols, data->rows, 0, GL_BGR, GL_UNSIGNED_BYTE, data->data);
}
And after that we can use our texture like we would use any other texture on OpenGL. Hope someone finds this useful in the future! :)

Related

Wrong colors when converting an AVFrame to QVideoFrame

I read videos with libav and display them in a QAbstractVideoSurface in QML. It works so far, however, I did not manage to get the colors right.
My av_read_frame Loop looks like this:
if (frameFinished)
{
SwsContext* context = nullptr;
context = sws_getContext(_frame->width, _frame->height, (AVPixelFormat)_frame->format, _frame->width, _frame->height, AVPixelFormat::AV_PIX_FMT_RGBA, SWS_BICUBIC, nullptr, nullptr, nullptr);
QImage img(_frame->width, _frame->height, QImage::Format_RGBA8888);
uint8_t* dstSlice[] = { img.bits() };
int dstStride = img.width() * 4;
sws_scale(context, _frame->data, _frame->linesize,
0, _frame->height, dstSlice, &dstStride);
av_packet_unref(&packet);
sws_freeContext(context);
}
If I save the image to disk at this point, the colors are already wrong (everything looks red).
Later, I display the images in a video surface with the format QVideoFrame::Format_ARGB32, and the colors are wrong again, but look different than the saved image (everything looks blue).
I started to experiment with libav/ffmpeg recently, so maybe the problem is something else and I just have no clue. Let me know, if you need more information :)

draw part of image with openGL glDrawPixels

I have a function to draw an image in an openGL context. (used in that case to render to a texture) That works for the whole image, but should also be able to render only a rectangular part. Rendering parts works if the part has the same width as the image. For parts that are less wide than the image-data it fails.
Here is the function (reduced to only the part for small width, no cleanup,etc)
void drawImage(uint32 imageWidth, uint32 imageHeight, uint8* pData,
uint32 offX, uint32 partWidth) // (offX+partWidth<=imageWidth)
{
uint8* p(pData);
if (partWidth != imageWidth)
{
glPixelStorei(GL_PACK_ROW_LENGTH, imageWidth);
p = calcFrom(offX, pData); // point at pixel in row
}
glDrawPixels(partWidth, ImageHeight, GL_BGRA, GL_UNSIGNED_BYTE, p);
}
As said: if (widthPart==imageWidth) the rendering works fine. For some combinations of partWidth and imageWidth it works also but that seems to be a very special case, mainly width very small images and a some special partWidths.
I found no examples for this, but from the docs I think this shold be possible to do somehow like that. Did I missunderstand the whole thing, or have I just overseen a small pit-fall??
Thanks,
Moritz
P.S: it's running on windows
[Edited:] P.P.S: by now I have tried to do that as texture. If I replace glDrawPixels with glTexImage2D I have the same problem...(could upload the whole image and render only part, but for small small parts of big pictures that might not e the best way...)
AAArrrghh!!
GL_UNPACK_ROW_LENGTH not GL_PACK_ROW_LENGTH!!!!

openFrameworks and openCv image processing issue with doing analysing video and rendering manipulated images back to the user with color palette

I am working on a project with OpenFrameworks using ofxCV, ofxOpencv and ofxColorQuantizer. Technically, the project is analyzing live video captured via webcam and analysis's the image in real time to gather and output the most prominent color in the current frame. When generating the most prominent color I am using the pixel difference between the current frame and the previous frame to generate the what colors have updated and use the updated or moving areas of the video frame to figure out the most prominent colors.
The reason for using the pixel difference's to generate the color pallet is because I want to solve for the case of a user walks into the video frame, I want try and gather the color pallet of the person, for instance what they are wearing. For example red shirt, blue pants will be in the pallet and the white background will be excluded.
I have a strong background in Javascript and canvas but am fairly new to OpenFrameworks and C++ which is why I think I am running into a roadblock with this problem I described above.
Along with OpenFrameworks I am using ofxCV, ofxOpencv and ofxColorQuantizer as tools for this installation. I am taking a webcam image than making it a cv:Mat than using pyrdown on the webcam image twice followed by a absdiff of the mat which I am than trying to pass the mat into the ofxColorQuantizer. This is where I think I am running into problems — I don't think the ofxColorQuantizer likes the mat format of the image I am trying to use. I've tried looking for the different image format to try and convert the image to to solve this issue but I haven't been able to come to solution.
For efficiencies I am hoping to to the color difference and color prominence calculations on the smaller image (after I pyrdown' the image) and display the full image on the screen and the generated color palette is displayed at the bottom left like in the ofxColorQuantizer example.
I think there may be other ways to speed up the code but at the moment I am trying to get this portion of the app working first.
I have my main.cpp set up as follows:
#include "ofMain.h"
#include "ofApp.h"
#include "ofAppGlutWindow.h"
//========================================================================
int main( ){
ofAppGlutWindow window;
ofSetupOpenGL(&window, 1024,768, OF_WINDOW); // <-------- setup the GL context
// ofSetupOpenGL(1024,768,OF_WINDOW); // <-------- setup the GL context
// this kicks off the running of my app
// can be OF_WINDOW or OF_FULLSCREEN
// pass in width and height too:
ofRunApp(new ofApp());
}
My ofApp.h file is as follows:
#pragma once
#include "ofMain.h"
#include "ofxOpenCv.h"
#include "ofxCv.h"
#include "ofxColorQuantizer.h"
class ofApp : public ofBaseApp{
public:
void setup();
void update();
void draw();
ofVideoGrabber cam;
ofPixels previous;
ofImage diff;
void kMeansTest();
ofImage image;
ofImage img;
cv::Mat matA, matB;
ofImage diffCopy;
ofImage outputImage;
ofxCv::RunningBackground background;
ofxColorQuantizer colorQuantizer;
// a scalar is like an ofVec4f but normally used for storing color information
cv::Scalar diffMean;
};
And finally my ofApp.cpp is below:
#include "ofApp.h"
using namespace ofxCv;
using namespace cv;
//--------------------------------------------------------------
void ofApp::setup(){
ofSetVerticalSync(true);
cam.initGrabber(320, 240);
// get our colors
colorQuantizer.setNumColors(3);
// resize the window to match the image
// ofSetWindowShape(image.getWidth(), image.getHeight());
ofSetWindowShape(800, 600);
// imitate() will set up previous and diff
// so they have the same size and type as cam
imitate(previous, cam);
imitate(diff, cam);
imitate(previous, outputImage);
imitate(diff, outputImage);
}
//--------------------------------------------------------------
void ofApp::update(){
cam.update();
if(cam.isFrameNew()) {
matA = ofxCv::toCv(cam.getPixelsRef());
ofxCv::pyrDown(matA, matB);
ofxCv::pyrDown(matB, matA);
ofxCv::medianBlur(matA, 3);
ofxCv::toOf(matA, outputImage);
// take the absolute difference of prev and cam and save it inside diff
absdiff(previous, outputImage, diff);
}
}
//--------------------------------------------------------------
void ofApp::draw(){
// If the image is ready to draw, then draw it
if(outputImage.isAllocated()) {
outputImage.update();
outputImage.draw(0, 0, ofGetWidth(), ofGetHeight());
}
ofBackground(100,100,100);
ofSetColor(255);
ofImage diffCopy;
diffCopy = diff;
diffCopy.resize(diffCopy.getWidth()/2, diffCopy.getHeight()/2);
// there is some sort of bug / issue going on here...
// prevent the app from compiling
// comment out to run and see blank page
colorQuantizer.quantize(diffCopy.getPixelsRef());
ofLog() << "the number is " << outputImage.getHeight();
ofLog() << "the number is " << diffCopy.getHeight();
ofSetColor(255);
img.update();
// cam.draw(0, 0, 800, 600);
outputImage.draw(0, 0, 800, 600);
// colorQuantizer.draw(ofPoint(0, cam.getHeight()-20));
colorQuantizer.draw(ofPoint(0, 600-20));
// use the [] operator to get elements from a Scalar
float diffRed = diffMean[0];
float diffGreen = diffMean[1];
float diffBlue = diffMean[2];
ofSetColor(255, 0, 0);
ofRect(0, 0, diffRed, 10);
ofSetColor(0, 255, 0);
ofRect(0, 15, diffGreen, 10);
ofSetColor(0, 0, 255);
ofRect(0, 30, diffBlue, 10);
}
//--------------------------------------------------------------
void ofApp::kMeansTest(){
cv::Mat samples = (cv::Mat_<float>(8, 1) << 31 , 2 , 10 , 11 , 25 , 27, 2, 1);
cv::Mat labels;
// double kmeans(const Mat& samples, int clusterCount, Mat& labels,
cv::TermCriteria termcrit;
int attempts, flags;
cv::Mat centers;
double compactness = cv::kmeans(samples, 3, labels, cv::TermCriteria(), 2, cv::KMEANS_PP_CENTERS, centers);
cout<<"labels:"<<endl;
for(int i = 0; i < labels.rows; ++i)
{
cout<<labels.at<int>(0, i)<<endl;
}
cout<<"\ncenters:"<<endl;
for(int i = 0; i < centers.rows; ++i)
{
cout<<centers.at<float>(0, i)<<endl;
}
cout<<"\ncompactness: "<<compactness<<endl;
}
Apologies in advance for the state of my code — it's getting late and I'm trying to get this done.
My question is what is the image format openFrameworks is using for grabbing the webcam image, what is the image format that openCV expects and what should I use to switch back from a mat image to an ofImage and is there a way to getPixelsRef from a mat image?
The area of code that I think I have something wrong is the following logic.
I have this line of code which gets the video frame from the webcam matA = ofxCv::toCv(cam.getPixelsRef());
Than do a couple ofxCv procedures on the frame such as ofxCv::pyrDown(matA, matB); which I think changes the image format or pixel format of the frame
Than I convert the frame back to OF with ofxCv::toOf(matA, outputImage);,
Next I get the difference in the pixels between the current frame and the last frame, create a copy of the difference between the two frames. Potentially the issue lies here with the diff output image format
Which I pass the diff copy to colorQuantizer.quantize(diffCopy.getPixelsRef()); to try and generate the color palette in for the change in pixels.
It is the colorQuantizer class and function call that is giving me an error which reads thread error [ error ] ofTexture: allocate(): ofTextureData has 0 width and/or height: 0x0
with an EXC_BAD_ACCESS
And lastly, could there be an alternative cause for the exc_bad_access thread error rather than image formatting? Being new to c++ I'm just guessing and going off instinct of what I think the rood cause of my problem is.
Many thanks.

Display contents of OpenglES buffer

I want to display a yuv to rgb converted frame to the default display. Currently i am doing it with the following code where the yuv to rgb conversion is done by an assembly code which loads CPU. I have found some code to do the same with opengles.
Yuv420_to_RGB(ui8buf, buffer1, h1, w1); /* RGB data will be resulted in buffer1 */
window = ANativeWindow_fromSurface(env, surface);
ANativeWindow_acquire(window);
wid = ANativeWindow_getWidth(window);
hei = ANativeWindow_getHeight(window);
ANativeWindow_setBuffersGeometry(window,w1,h1,1)
if (ANativeWindow_lock(window, &buffer, NULL) == 0)
{
memcpy(buffer.bits, buffer1, (4* w1*h1));
ANativeWindow_unlockAndPost(window);
}
ANativeWindow_release(window);
I have the opengles routine ending with glDrawArrays. How can i display the result of opengles conversion?
Nothing of the code you posted does anything with OpenGL-ES. The typical method to implement color space conversion with OpenGL(-ES) is to load the image into a texture, load a fragment shader performing the color conversion and draw a (full viewport) textured quad (that's what glDrawArrays will do, if a quad's geometry has been loaded into the vertex arrays before).

opengl video freeze

I have an IDS ueye cam and proceed the capture via PBO to OpenGL (OpenTK). On my developer-pc it works great, but on slower machines the video freezes after some time.
Code for allocating memory via opengl and map to ueye, so camera saves processed images in here:
// Generate PBO and save id
GL.GenBuffers(1, out this.frameBuffer[i].BufferID);
// Define the type of the buffer.
GL.BindBuffer(BufferTarget.PixelUnpackBuffer, this.frameBuffer[i].BufferID);
// Define buffer size.
GL.BufferData(BufferTarget.PixelUnpackBuffer, new IntPtr(width * height * depth), IntPtr.Zero, BufferUsageHint.StreamDraw);
// Get pointer to by openGL allocated buffer and
// lock global with uEye.
this.frameBuffer[i].PointerToNormalMemory = GL.MapBuffer(BufferTarget.PixelUnpackBuffer, BufferAccess.WriteOnly);
this.frameBuffer[i].PointerToLockedMemory = uEye.GlobalLock(this.frameBuffer[i].PointerToNormalMemory);
// Unmap PBO after use.
GL.UnmapBuffer(BufferTarget.PixelUnpackBuffer);
// Set selected PBO to none.
GL.BindBuffer(BufferTarget.PixelUnpackBuffer, 0);
// Register buffer to uEye
this.Succeeded("SetAllocatedImageMem", this.cam.SetAllocatedImageMem(width, height, depth, this.frameBuffer[i].PointerToLockedMemory, ref this.frameBuffer[i].MemId));
// Add buffer to uEye-Ringbuffer
this.Succeeded("AddToSequence", this.cam.AddToSequence(this.frameBuffer[i].PointerToLockedMemory, this.frameBuffer[i].MemId));
To copy the image from pbo to an texture (Texture is created and ok):
// Select PBO with new video image
GL.BindBuffer(BufferTarget.PixelUnpackBuffer, nextBufferId);
// Select videotexture as current
GL.BindTexture(TextureTarget.Texture2D, this.videoTextureId);
// Copy PBO to texture
GL.TexSubImage2D(
TextureTarget.Texture2D,
0,
0,
0,
nextBufferSize.Width,
nextBufferSize.Height,
OpenTK.Graphics.OpenGL.PixelFormat.Bgr,
PixelType.UnsignedByte,
IntPtr.Zero);
// Release Texture
GL.BindTexture(TextureTarget.Texture2D, 0);
// Release PBO
GL.BindBuffer(BufferTarget.PixelUnpackBuffer, 0);
Maybe someone can see the mistake... After about 6 seconds the ueye events don't deliver any images any more. When I remove TexSubImage2D it works well, but of course no image appears.
Is there maybe a lock or something from opengl?
Thanks in advance - Thomas
it seems like a shared buffer problem. you may try to implement a simple queue mechanism to get rid of that problem.
sample code (not meant to be working):
queue< vector<BYTE> > frames;
...
frames.push(vector<BYTE>(frameBuffer, frameBuffer + frameSize));
...
// use frame here at GL.TexSubImage2D using frames.front()
frames.pop();
Found the failure by myself. Just replace in the code above StreamDraw with StreamRead.
GL.BufferData(BufferTarget.PixelUnpackBuffer, new IntPtr(width * height * depth), IntPtr.Zero, BufferUsageHint.StreamRead);