I have an IDS ueye cam and proceed the capture via PBO to OpenGL (OpenTK). On my developer-pc it works great, but on slower machines the video freezes after some time.
Code for allocating memory via opengl and map to ueye, so camera saves processed images in here:
// Generate PBO and save id
GL.GenBuffers(1, out this.frameBuffer[i].BufferID);
// Define the type of the buffer.
GL.BindBuffer(BufferTarget.PixelUnpackBuffer, this.frameBuffer[i].BufferID);
// Define buffer size.
GL.BufferData(BufferTarget.PixelUnpackBuffer, new IntPtr(width * height * depth), IntPtr.Zero, BufferUsageHint.StreamDraw);
// Get pointer to by openGL allocated buffer and
// lock global with uEye.
this.frameBuffer[i].PointerToNormalMemory = GL.MapBuffer(BufferTarget.PixelUnpackBuffer, BufferAccess.WriteOnly);
this.frameBuffer[i].PointerToLockedMemory = uEye.GlobalLock(this.frameBuffer[i].PointerToNormalMemory);
// Unmap PBO after use.
GL.UnmapBuffer(BufferTarget.PixelUnpackBuffer);
// Set selected PBO to none.
GL.BindBuffer(BufferTarget.PixelUnpackBuffer, 0);
// Register buffer to uEye
this.Succeeded("SetAllocatedImageMem", this.cam.SetAllocatedImageMem(width, height, depth, this.frameBuffer[i].PointerToLockedMemory, ref this.frameBuffer[i].MemId));
// Add buffer to uEye-Ringbuffer
this.Succeeded("AddToSequence", this.cam.AddToSequence(this.frameBuffer[i].PointerToLockedMemory, this.frameBuffer[i].MemId));
To copy the image from pbo to an texture (Texture is created and ok):
// Select PBO with new video image
GL.BindBuffer(BufferTarget.PixelUnpackBuffer, nextBufferId);
// Select videotexture as current
GL.BindTexture(TextureTarget.Texture2D, this.videoTextureId);
// Copy PBO to texture
GL.TexSubImage2D(
TextureTarget.Texture2D,
0,
0,
0,
nextBufferSize.Width,
nextBufferSize.Height,
OpenTK.Graphics.OpenGL.PixelFormat.Bgr,
PixelType.UnsignedByte,
IntPtr.Zero);
// Release Texture
GL.BindTexture(TextureTarget.Texture2D, 0);
// Release PBO
GL.BindBuffer(BufferTarget.PixelUnpackBuffer, 0);
Maybe someone can see the mistake... After about 6 seconds the ueye events don't deliver any images any more. When I remove TexSubImage2D it works well, but of course no image appears.
Is there maybe a lock or something from opengl?
Thanks in advance - Thomas
it seems like a shared buffer problem. you may try to implement a simple queue mechanism to get rid of that problem.
sample code (not meant to be working):
queue< vector<BYTE> > frames;
...
frames.push(vector<BYTE>(frameBuffer, frameBuffer + frameSize));
...
// use frame here at GL.TexSubImage2D using frames.front()
frames.pop();
Found the failure by myself. Just replace in the code above StreamDraw with StreamRead.
GL.BufferData(BufferTarget.PixelUnpackBuffer, new IntPtr(width * height * depth), IntPtr.Zero, BufferUsageHint.StreamRead);
Related
I am playing around with NVDEC H.264 decoder from NVIDIA CUDA samples, one thing I've found out is once frame is decoded, it's converted from NV12 to BGRA buffer which is allocated on CUDA's side, then this buffer is copied to D3D BGRA texture.
I find this not very efficient in terms of memory usage, and want to convert NV12 frame directly to D3D texture with this kernel:
void Nv12ToBgra32(uint8_t *dpNv12, int nNv12Pitch, uint8_t *dpBgra, int nBgraPitch, int nWidth, int nHeight, int iMatrix)
So, create D3D texture (BGRA, D3D11_USAGE_DEFAULT, D3D11_BIND_SHADER_RESOURCE | D3D11_BIND_UNORDERED_ACCESS, D3D11_CPU_ACCESS_WRITE, 1 mipmap),
then register and write it on CUDA side:
//Register
ck(cuGraphicsD3D11RegisterResource(&cuTexResource, textureResource, CU_GRAPHICS_REGISTER_FLAGS_NONE));
...
//Write output:
CUarray retArray;
ck(cuGraphicsMapResources(1, &cuTexResource, 0));
ck(cuGraphicsSubResourceGetMappedArray(&retArray, cuTexResource, 0, 0));
/*
yuvFramePtr (NV12) is uint8_t* from decoded frame,
it's stored within CUDA memory I believe
*/
Nv12ToBgra32(yuvFramePtr, w, (uint8_t*)retArray, 4 * w, w, h);
ck(cuGraphicsUnmapResources(1, &cuTexResource, 0));
Once kernel is called, I get crash. May be because of misusing CUarray, can anybody please clarify how to use output of cuGraphicsSubResourceGetMappedArray to write texture memory from CUDA kernel? (since writing raw memory is only needed, there is no need to handle correct clamp, filtering and value scaling)
Ok, for anyone who struggling on question "How to write D3D11 texture from CUDA kernel", here is how:
Create D3D texture with D3D11_BIND_UNORDERED_ACCESS.
Then, register resource:
//ID3D11Texture2D *textureResource from D3D texture
CUgraphicsResource cuTexResource;
ck(cuGraphicsD3D11RegisterResource(&cuTexResource, textureResource, CU_GRAPHICS_REGISTER_FLAGS_NONE));
//You can also add write-discard if texture will be fully written by kernel
ck(cuGraphicsResourceSetMapFlags(cuTexResource, CU_GRAPHICS_MAP_RESOURCE_FLAGS_WRITE_DISCARD));
Once texture is created and registered we can use it as write surface.
ck(cuGraphicsMapResources(1, &cuTexResource, 0));
//Get array for first mip-map
CUArray retArray;
ck(cuGraphicsSubResourceGetMappedArray(&retArray, cuTexResource, 0, 0));
//Create surface from texture
CUsurfObject surf;
CUDA_RESOURCE_DESC surfDesc{};
surfDesc.res.array.hArray = retArray;
surfDesc.resType = CU_RESOURCE_TYPE_ARRAY;
ck(cuSurfObjectCreate(&surf, &surfDesc));
/*
Kernel declaration is:
void Nv12ToBgra32Surf(uint8_t* dpNv12, int nNv12Pitch, cudaSurfaceObject_t surf, int nBgraPitch, int nWidth, int nHeight, int iMatrix)
Surface write:
surf2Dwrite<uint>(VALUE, surf, x * sizeof(uint), y);
For BGRA surface we are writing uint, X offset is in bytes,
so multiply it with byte-size of type.
Run kernel:
*/
Nv12ToBgra32Surf(yuvFramePtr, w, /*out*/surf, 4 * w, w, h);
ck(cuGraphicsUnmapResources(1, &cuTexResource, 0));
ck(cuSurfObjectDestroy(surf));
I want to do texture atlas with Xlib in X11. I Created a pixmap by loading pixel data from an image file which contains all sprites that will be used as texture. I can copy part of texture atlas pixmap (single spirit) to another pixmap created as off-screen drawable successfully.
Here comes the problem. I want the texture copied to the destination pixmap with partial transparent so that there will not be a background rectangle appears behind each spirit. To do that I created a pixmap with depth equals 1 for the whole texture atlas image(500 * 500).
The pMaskData is the pixel data with depth 1.
Pixmap texAtlasMask = XCreatePixmapFromBitmapData(kTheDisplay, kRootWindow,
(char*)pMaskData, 500, 500, 1, 0, 1);
Then I create a clip_mask pixmap for a single sprite, the size of the sprite is 16*16, by first creating a 1 depth pixmap:
Pixmap clipMask = XCreatePixmap(kTheDisplay, kRootWindow, 16, 16, 1);
then use the following call to fill the content of clipMask:
// Error occurs here
// reqest code: 62:X_CopyArea
// Error code: 8:BadMatch (invalid parameter attributes)
XCopyArea(kTheDisplay, texAtlasMask, clipMask, m_gc, 0, 0,16, 16, 0, 0);
After that:
XSetClipMask(kTheDisplay, m_gc, clipMask);
// Copy source spirit to backing store pixmap
XSetClipOrigin(kTheDisplay, m_gc, destX, destY);
XCopyArea(kTheDisplay, m_symAtlas, m_backStore, m_gc, srcLeft, srcTop,
width, height, destX, destY);
The m_symAtlas is the texture atlas pixmap, m_backStore is the destination pixmap we are drawing to.
As listed above error happens in the first call of XCopyArea. I tried XCopyPlane, but nothing changed.
And I play around with XCopyArea and found that as long as the pixmap 's depth is 32 the XCopyArea works fine, it fails when the depth is not 32. Any idea what is wrong?
I initialize OpenCV window with following code:
cv::VideoCapture * stream = new cv::VideoCapture("stream_ip");
if (!stream->isOpened()){
printf("Couldn't open stream! %s\n", strerror(errno));
}
//We create window with OpenGL enabled.
cv::namedWindow("rtsp_stream", cv::WINDOW_OPENGL);
//Make it fullscreen (I also tried with fixed screen size without luck.)
cv::setWindowProperty("rtsp_stream", cv::WND_PROP_FULLSCREEN, cv::WINDOW_FULLSCREEN);
//Set OpenGL context to use this window.
cv::setOpenGlContext("rtsp_stream");
//Set openGlDrawCallback.
cv::setOpenGlDrawCallback("rtsp_stream", on_opengl, NULL);
//This is the material that the image will be rendered on.
cv::Mat frame;
char k;
bool continueStream = true;
while (continueStream) {
//We read data from the stream and write it on the frame.
if((stream->read(frame)) != 0){
//Then we display/render the image using imshow.
cv::imshow("rtsp_stream", frame);
k = cv::waitKey(1);
//I'm not sure if updateWindow needs to be manually called to make openGLDrawCallback or if imshow calls it automatically after done rendering. So I have tried with and without it.
//cv::updateWindow("rtsp_stream");
switch(k){
case 0x1b: //ESC key
printf("Closing stream.\n");
continueStream = false;
break;
}
}
}
open_gl function just simply fprints some text to see if opengldrawcallback gets called. I have also made sure I have OpenGL and QT enabled on my OpenCV with std::cout << cv::getBuildInformation() << std::endl;. I have tried to find solutions from many different websites and sources, including this book. Program works as expected in every manner, except that openglcallback never gets called. All the help is much appreciated.
Okay. so as I yesterday found out, cv::imshow prevented me from using OpenGL commands on that window.
So what I had to do to overcome this, was to read the data on a cv::Mat, just like before. But instead of rendering that cv::Mat straight on the window using cv::imshow, I had to first store the data on a texture and then render that texture on screen. To store the data on a texture I used the following method:
/**
*texture: Pointer to OpenGL texture that we want to render our stream on.
*data: cv::Mat that contains the data we want to render.
**/
void storeStreamToTexture(GLuint texture, cv::Mat* data){
//Bind the texture we want to render to.
glBindTexture(GL_TEXTURE_2D, texture);
//we flip the Mat to start reading from the beginning.
cv::flip(*data, *data, 0);
//Store mat data to our texture.
glTexImage2D(GL_TEXTURE_2D, 0, GL_RGB, data->cols, data->rows, 0, GL_BGR, GL_UNSIGNED_BYTE, data->data);
}
And after that we can use our texture like we would use any other texture on OpenGL. Hope someone finds this useful in the future! :)
I want to use the CImg library (http://cimg.sourceforge.net/) to rotate an image with an arbitrary angle (the image is read by Qt which should not perform the rotation):
QImage img("sample_with_alpha.png");
img = img.convertToFormat(QImage::Format_ARGB32);
float angle = 45;
cimg_library::CImg<uint8_t> src(img.bits(), img.width(), img.height(), 1, 4);
cimg_library::CImg<uint8_t> out = src.get_rotate(angle);
// Further processing:
// Data: out.data(), out.width(), out.height(), Stride: out.width() * 4
The final data in "out.data()" is ok when the the angle is set to 0. But for other angles the output data is distorted. I assume that the CImg library changes the output format and/or stride during rotation?
Regards,
CImg does not store the pixel buffer of an image in interleaved mode, as RGBARGBARGBA... but uses a channel by channel structure RRRRRRRR.....GGGGGGGGG.......BBBBBBBBB.....AAAAAAAAA.
I assume your img.bits() pointer points to pixels with interleaved channels, so if you want to pass this to CImg, you'll need to permute the buffer structure before you can apply any of the CImg method.
Try this :
cimg_library::CImg<uint8_t> src(img.bits(), 4,img.width(), img.height(), 1);
src.permute_axes("yzcx");
cimg_library::CImg<uint8_t> out = src.get_rotate(angle);
// Here, the out image should be OK, try displaying it with out.display();
// But you still need to go back to an interleaved image pointer if you want to
// get it back in Qt.
out.permute_axes("cxyz"); // Do the inverse permutation.
const uint8_t *p_out = out.data(); // Interleaved result.
I guess this should work as expected.
I am working on a system that sends a compressed video to a client from 3d graphics that are done in the server as soon as they are rendered.
I already have the code working, but I feel it could be much faster (and it is already a bottleneck in the system)
Here is what I am doing:
First I grab the framebuffer
glReadBuffer( GL_FRONT );
glReadPixels( 0, 0, width, height, GL_RGB, GL_UNSIGNED_BYTE, buffer );
Then I flip the framebuffer, because there is a weird bug with swsScale (which I am using for colorspace conversion) that flips the image vertically when I convert. I am flipping in advance, nothing fancy.
void VerticalFlip(int width, int height, byte* pixelData, int bitsPerPixel)
{
byte* temp = new byte[width*bitsPerPixel];
height--; //remember height array ends at height-1
for (int y = 0; y < (height+1)/2; y++)
{
memcpy(temp,&pixelData[y*width*bitsPerPixel],width*bitsPerPixel);
memcpy(&pixelData[y*width*bitsPerPixel],&pixelData[(height-y)*width*bitsPerPixel],width*bitsPerPixel);
memcpy(&pixelData[(height-y)*width*bitsPerPixel],temp,width*bitsPerPixel);
}
delete[] temp;
}
Then I convert it to YUV420p
convertCtx = sws_getContext(width, height, PIX_FMT_RGB24, width, height, PIX_FMT_YUV420P, SWS_FAST_BILINEAR, NULL, NULL, NULL);
uint8_t *src[3]= {buffer, NULL, NULL};
sws_scale(convertCtx, src, &srcstride, 0, height, pic_in.img.plane, pic_in.img.i_stride);
Then I pretty much just call the x264 encoder. I am already using the zerolatency preset.
int frame_size = x264_encoder_encode(_encoder, &nals, &i_nals, _inputPicture, &pic_out);
My guess is that there should be a faster way to do this. Capturing the frame and converting it to YUV420p. It would be nice to convert it to YUV420p in the GPU and only after that copying it to system memory, and hopefully there is a way to do color conversion without the need to flip.
If there is no better way, at least this question may help someone trying to do this, to do it the same way I did.
First , use async texture read using PBOs.Here is example It speeds ups the read by using 2 PBOs which work asynchronously without stalling the pipeline like readPixels does when used directly.In my app I got 80% performance boost when switched to PBOs.
Additionally , on some GPUs glGetTexImage() works faster than glReadPixels() so try it out.
But if you really want to take the video encoding to the next level you can do it via CUDA using Nvidia Codec Library.I recently asked the same question so this can be helpful.