Here is basically what I'm doing :
I have a video stream (YUV format). Each frame is extracted into a buffer (frameBytes). Later, this buffer is used to do the YUV->RGB conversion, then transfered in an IplImage. The IplImage is then transfered into a cv::Mat and displayed in an OpenGL context. Everything works fine.
What I would like to do is bypass the IplImage and cv::Mat section to directly work with the frameBytes buffer in OpenGL and do the conversion in the shaders.
This explaination is just for the context, since the problem I'm having is simpler.
To see if I can work with the buffer earlier, I try to copy it with memcpy and then save it in a QImage then in a file.
Here is my code for this part :
unsigned char *mycopy = new unsigned char[1920*1080*3];
memcpy(mycopy, frameBytes, sizeof(1920*1080*3));
QImage *img = new QImage(mycopy, 1920, 1080, QImage::Format_RGB888);
img->save("image.jpg",0,-1);
frameBytes contains the YUV data from the video stream. I know it's YUV and I'm trying to create a QImage with RGB888 format but since QImage doesn't support the format, I didn't make the conversion there, I thought it would still save an image but with the wrong colors so I don't care for the moment (Maybe this assumption is wrong ?).
Problem is, the image saved is black.
Just for more information, here is an example where I use frameBytes for the YUV->RGB conversion.
void DeckLinkCaptureDelegate::convertFrameToOpenCV(void* frameBytes, IplImage * m_RGB){
if(!m_RGB) m_RGB = cvCreateImage(cvSize(1920, 1080), IPL_DEPTH_8U, 3);
unsigned char* pData = (unsigned char *) frameBytes;
for(int i = 0, j=0; i < 1920 * 1080 * 3; i+=6, j+=4)
{
unsigned char u = pData[j];
unsigned char y = pData[j+1];
unsigned char v = pData[j+2];
//fprintf(stderr, "%d\n", v);
m_RGB->imageData[i+2] = 1.0*y + 8 + 1.402*(v-128); // r
m_RGB->imageData[i+1] = 1.0*y - 0.34413*(u-128) - 0.71414*(v-128); // g
m_RGB->imageData[i] = 1.0*y + 1.772*(u-128) + 0; // b
y = pData[j+3];
m_RGB->imageData[i+5] = 1.0*y + 8 + 1.402*(v-128); // r
m_RGB->imageData[i+4] = 1.0*y - 0.34413*(u-128) - 0.71414*(v-128); // g
m_RGB->imageData[i+3] = 1.0*y + 1.772*(u-128) + 0;
}
}
Fixing the bug
You didn't copy all the data to your new buffer:
unsigned char *mycopy = new unsigned char[1920*1080*3];
memcpy(mycopy, frameBytes, sizeof(1920*1080*3));
That sizeof in there means that you're only copying an int-sized block, rather than 6MB. It looks like an accidental holdover from using a static array? Replace it with
const size_t bufsize = 1920*1080*3;
auto *mycopy = new unsigned char[bufsize];
memcpy(mycopy, frameBytes, bufsize);
A simpler approach
Alternatively, instead of doing the memory allocation yourself (and being responsible for delete[]ing it after the QImage is destructed), you could copy the image instead:
const unsigned char *bytes = frameBytes;
QImage img = QImage(bytes, 1920, 1080, QImage::Format_RGB888).copy();
The way this works is that we create a temporary QImage using frameBytes as its source (we pass it as pointer to const to insist it's read-only). We then copy() the whole of it to a new QImage, throwing away the temporary. The copy() does something similar to your code above, but we're now saved from having to do the calculations, eliminating some of the consequent potential for error.
Note also that I prefer to pass QImage by value. Although this might seem inefficient, most (copyable) Qt types are designed as reference-counted copy-on-write structures, and can be safely used this way, eliminating another class of errors (memory management). This is true of Qt's collection types, too, and very useful.
Related
I'm looking for the most efficient way to flatten an array of structs in C++ for passing the flattend 1D array data as input to a cv::Mat. The struct looks as follows:
struct Color3
{
uint8_t red, green, blue;
}
My code then looks like this:
// Update color frame
cv::Mat colorMat = cv::Mat::zeros(cv::Size(1920, 1080), CV_8UC3)
const Color3* colorPtr = colorFrame->getData(); // Get Frame from Library
std::vector<uchar> vecColorData;
data.reserve(1920 * 1080 * 3);
for (int i = 0; i < 1920 * 1080; ++i)
{
auto color = *colorPtr;
vecColorData.push_back(color.red);
vecColorData.push_back(color.green);
vecColorData.push_back(color.blue);
vecColorData++;
}
colorMat.data = vecColorData.data();
Is there a more efficient way than creating an intermediate std::vector and looping over the entire array? I guess I'm looking for something like:
colorMat.data = colorFrame->getData()
However, I'm getting the following error: a value of type Color3* cannot be assigned to an entity of type uchar*.
you don't need an intermediate vector.
If I understood, you want to assign the same RGB triple to all data.
It is also unclear to me if you have to allocate colorMat.data on your own or not.
If this is the case, once colorMat.data is allocated and sized 1920 * 1080 * 3, you can do something like the following:
uchar * data = colorMat.data;
for (int i = 0; i < 1920 * 1080; ++i)
{
*data++ = (uchar)colorPtr->red;
*data++ = (uchar)colorPtr->green;
*data++ = (uchar)colorPtr->.blue;
}
The following answer is not technically portable but will work on the vast majority of platforms you will encounter in real life.
It is extremely likely that your Color3 struct has no padding. You can veryify this by using a static_assert:
static_assert(sizeof(Color3) == sizeof(uint8_t) * 3);
With this confirmed you can cast an array of Color3 to an array of uint8_t and pass it directly to colorMat.data (assuming that member actually accepts uint8_t).
Your code therefore becomes:
cv::Mat colorMat = cv::Mat::zeros(cv::Size(1920, 1080), CV_8UC3)
const Color3* colorPtr = colorFrame->getData(); // Get Frame from Library
colorMat.data = reinterpret_cast<const uint8_t*>(colorPtr);
Bear in mind I have never used the cv library and know nothing about the ownership requirements of the data pointer. The above just replicates what you're doing without the unnecessary std::vector.
I am trying to encode an MP4 video using raw YUV frames data, but I am not sure how can I fill the plane data (preferably without using other libraries like ffmpeg)
The frame data is already encoded in I420, and does not need conversion.
Here is what I am trying to do:
const char *frameData = /* Raw frame data */;
x264_t *encoder = x264_encoder_open(¶m);
x264_picture_t imgInput, imgOutput;
x264_picture_alloc(&imgInput, X264_CSP_I420, width, height);
// how can I fill the struct data of imgInput
x264_nal_t *nals;
int i_nals;
int frameSize = x264_encoder_encode(encoder, &nals, &i_nals, &imgInput, &imgOutput);
The equivalent command line that I have found is :
x264 --output video.mp4 --fps 15 --input-res 1280x800 imgdata_01.raw
But I could not figure out how the app does it.
Thanks.
Look at libx264 API usage example. This example use fread() to fill frame allocated by x264_picture_alloc() with actual i420 data from stdin. If you already have i420 data in memory and want to skip memcpy step than instead of it you can:
Use x264_picture_init() instead of x264_picture_alloc() and x264_picture_clean(). Because you don't need allocate memory on heap for frame data.
Fill x264_picture_t.img struct fields:
i_csp = X264_CSP_I420;
i_plane = 3;
plane[0] = pointer to Y-plane;
i_stride[0] = stride in bytes for Y-plane;
plane[1] = pointer to U-plane;
i_stride[1] = stride in bytes for U-plane;
plane[2] = pointer to V-plane;
i_stride[2] = stride in bytes for V-plane;
To complete the answer above, this is an example to fill an x264_picture_t image.
int fillImage(uint8_t* buffer, int width, int height, x264_picture_t*pic){
int ret = x264_picture_alloc(pic, X264_CSP_I420, width, height);
if (ret < 0) return ret;
pic->img.i_plane = 3; // Y, U and V
pic->img.i_stride[0] = width;
// U and V planes are half the size of Y plane
pic->img.i_stride[1] = width / 2;
pic->img.i_stride[2] = width / 2;
int uvsize = ((width + 1) >> 1) * ((height + 1) >> 1);
pic->img.plane[0] = buffer; // Y Plane pointer
pic->img.plane[1] = buffer + (width * height); // U Plane pointer
pic->img.plane[2] = pic->img.plane[1] + uvsize; // V Plane pointer
return ret;
}
to circumvent some (a lot) of problems with the Actionscript Camera API on Windows 8 Systems,
I decided to create a native extension to deal with the camera.
Right now, the camera part and all the glue to communicate with the AIR Runtime is actually working, so clicking on a button in AIR will open a new Windows window that will return a System::Drawing::Bitmap.
My task would be now to
a) Create a FREBitmapData object and
b) Fill in the BitmapData from the Windows Bitmap.
Should be easy, I thought, many days ago... As I'm not really familiar with C++ I didn't get this to work at all.
Here's what I tried so far:
bmp = form1->bitmap; // bmp is a handle to the System::Drawing::Bitmap returned from the external window
// Lock the bitmap's bits.
Rectangle rect = Rectangle(0, 0, bmp->Width, bmp->Height);
System::Drawing::Imaging::BitmapData^ bmpData = bmp->LockBits(rect, System::Drawing::Imaging::ImageLockMode::ReadWrite, bmp->PixelFormat);
// Get the address of the first line.
IntPtr ptr = bmpData->Scan0;
// Declare an array to hold the bytes of the bitmap.
// This code is specific to a bitmap with 24 bits per pixels.
int inputLength = Math::Abs(bmpData->Stride) * bmp->Height;
array<Byte>^ input = gcnew array<Byte>(inputLength);
// Copy the RGB values into the array.
System::Runtime::InteropServices::Marshal::Copy(ptr, input, 0, inputLength);
// Unlock the bits.
bmp->UnlockBits(bmpData);
// Create a FREByteArray to hold the data.
// Don't know, if this is necessary
FREObject* outputObject;
FREByteArray* outputBytes = new FREByteArray;
outputBytes->length = inputLength;
outputBytes->bytes = (uint8_t *) &input;
FREAcquireByteArray(outputObject, outputBytes);
// now copy it over
memcpy(outputBytes->bytes, &input, inputLength);
FREReleaseByteArray(outputObject);
// we create a new instance of BitmapData here,
// as we cannot simply pass it over in the args,
// because we don't know it's correct size at extension creation
FREObject* width;
FRENewObjectFromUint32(bmp->Width, width);
FREObject* height;
FRENewObjectFromUint32(bmp->Height, height);
FREObject* transparent;
FRENewObjectFromBool(uint32_t(0), transparent);
FREObject* fillColor;
FRENewObjectFromUint32(uint32_t(0xFFFFFF), fillColor);
FREObject obs[4] = { width, height, transparent, fillColor };
// we create some Actionscript Intsances here, we want to send back
FREObject* asBmpObj;
FRENewObject("BitmapData", 4, obs, asBmpObj, NULL);
// Now we have our AS bitmap data, copy bytes over
FREBitmapData* asData;
FREAcquireBitmapData(asBmpObj, asData);
// Now what? asData->bits32 won't accept array<Bytes> or any other value I've tried.
return asBmpObj;
The basic idea was:
a) find out the size and bit-depth of the original Win Bitmap (size is determinded by cam resolution picked in the Camera window)
b) write it's bytes to an array. Convert to 32 bits as necessary. (Still missing any idea.)
c) create AS Bitmap of the same size. Bit-depth must always be 32.
d) copy over array to AS Bitmap.
But I just can't achieve this.
Any advice? Thank you!
I don't think the following straight copy will work.
// Copy the RGB values into the array.
System::Runtime::InteropServices::Marshal::Copy(ptr, input, 0, inputLength);
You have to convert pixel by pixel. I don't know how to convert it to FREBitmapData. Here are the examples you can following on msdn
I finally figured it out:
the code below doesn't deal with the 24to32 bit conversion though, but it actually works in my application quite well, so I thought, i might share it:
FREObject launch(FREContext ctx, void* funcData, uint32_t argc, FREObject argv[])
{
System::Drawing::Bitmap^ windowsBitmap;
SKILLCamControl::CamControlForm^ form1;
form1 = gcnew SKILLCamControl::CamControlForm();
DialogResult dr;
// Show testDialog as a modal dialog and determine if DialogResult = OK.
dr = form1->ShowDialog();
if (dr == DialogResult::OK) {
windowsBitmap = form1->bitmap;
int bmpW = windowsBitmap->Width;
int bmpH = windowsBitmap->Height;
// we create a new instance of BitmapData here,
// as we cannot simply pass it over in the args,
// because we don't know it's correct size at extension creation
FREObject width;
FRENewObjectFromUint32( uint32_t(bmpW), &width);
FREObject height;
FRENewObjectFromUint32( uint32_t(bmpH), &height);
FREObject transparent;
FRENewObjectFromBool( uint32_t(0), &transparent);
FREObject fillColor;
FRENewObjectFromUint32( uint32_t(0xFF0000), &fillColor);
FREObject obs[4] = { width, height, transparent, fillColor };
FREObject freBitmap;
FRENewObject((uint8_t *)"flash.display.BitmapData", 4, obs, &freBitmap , NULL);
FREBitmapData2 freBitmapData;
FREAcquireBitmapData2(freBitmap, &freBitmapData);
// is inverted?
if (&freBitmapData.isInvertedY != (uint32_t*)(0) ) windowsBitmap->RotateFlip(RotateFlipType::RotateNoneFlipY);
int pixelSize = 4;
//Rect rect( 0, 0, freBitmap.width, freBitmap.height );
System::Drawing::Rectangle rect(0, 0, bmpW, bmpH);
BitmapData^ windowsBitmapData = windowsBitmap->LockBits(rect, ImageLockMode::ReadOnly, PixelFormat::Format32bppArgb);
for (int y = 0; y < bmpH ; y++)
{
//get pixels from each bitmap
byte* oRow = (byte*)windowsBitmapData->Scan0.ToInt32() + (y * windowsBitmapData->Stride);
byte* nRow = (byte*)freBitmapData.bits32 + (y * freBitmapData.lineStride32 * 4);
for (int x = 0; x < bmpW ; x++)
{
// set pixels
nRow[x * pixelSize] = oRow[x * pixelSize]; //B
nRow[x * pixelSize + 1] = oRow[x * pixelSize + 1]; //G
nRow[x * pixelSize + 2] = oRow[x * pixelSize + 2]; //R
}
}
// Free resources
FREReleaseBitmapData(freBitmap);
FREInvalidateBitmapDataRect(freBitmap, 0, 0, bmpW, bmpH);
windowsBitmap->UnlockBits(windowsBitmapData);
delete windowsBitmapData;
delete windowsBitmap;
return freBitmap;
}
else if (dr == DialogResult::Cancel)
{
return NULL;
}
return NULL;
}
I dont use C++ myself so this is not a full answer but just something to consider...
Bitmap data is universal raw pixel data. It should be passable within different software. Unless you are actually creating .BMP files with header etc??
...that will return a System::Drawing::Bitmap does this mean you have a bitmap's data held by C++ (as raw uncompressed RGBA pixels)? If so then just either put that inside a byteArray and send to AS3 or a if you can get that bitmap copied to the Windows clipboard then use AS3 to read from clipboard into a new AS3 Bitmap.
these might help you:
AS3: Copy image from clipboard
AS3: Serialize Bitmaps : Scroll down to the section ByteArray to BitmapData (for this to work you must first store the C++ bitmap bytes as a file call it what you want, example tempIMG.dat or myPIc.bin or whatever since file extension does not really matter just that you need a loadable URL).
I am struggling with release version of my opencv wrapper function.
The function code runs fine, but upon function block completition, memory access violation happens.
This problem does not appear in debug mode. The segfault happens upon freeing the heap.
int Myfunc(Arr1D_floatHdl FeatArrHdl, IMAQ_Image *img, someparams
*Params)
{
ImageInfo *Info = NULL;
//IplImage *CVImage = NULL;
Info = (ImageInfo*)img->address;
CheckImage(Info, Info);
//CVImage = cvCreateImageHeader( cvSize(Info->xRes, Info->yRes), IPL_DEPTH_8U, 4);
//CVImage->imageData = (char*)Info->imageStart;
//CVImage->widthStep = Info->xRes*sizeof(IPL_DEPTH_8U);
cv::Mat BGRAimg = cv::Mat(Info->yRes, Info->xRes, CV_8UC4, (char*)Info->imageStart, sizeof(CV_8UC4)*Info->xRes);
//cv::Mat BGRAimg(CVImage);
//cv::Mat BGRAimg = imread( "MyImg.png", cv::IMREAD_COLOR );
cv::Mat GREYimg;
cv::cvtColor(BGRAimg, GREYimg, CV_BGR2GRAY);
Here is the code where I create Mat object from user supplied data.
I tried to create IplImage first (commented version in code) and use Mat constructor with IplImage argument, but eneded up with the same problem.
I know I am doing something very wrong during the Mat construction, since manualy loading the image from disk does not cause the issue.
After creating the Mat object, all its parameters are correct and the image is fine. When comparing with the grey matrix created of it, it has refcount NULL, which I have read is perfectly fine since it is supposed to keep user data intact.
Please help.
UPDATE to give more information
Thank you for suggestions. I am obviously prone to create such errors, I am new to C/C++.
Unfortunately, the access violation still persists.
Here is the complete wrapper function as it is. I tried to narrow down the problem, and skipping the HOG.compute function I do no longer get memory corruption. Skipping the memcpy acrobatics in the end, I still get the memory corrupted.
int GetHOGFeatures(Arr1D_floatHdl FeatArrHdl, IMAQ_Image *img, HogParams *Params) //returns -1 on HOG window parameters missmatch
{
ImageInfo *Info = NULL;
Info = (ImageInfo*)img->address;
CheckImage(Info, Info);
cv::Mat BGRAimg = cv::Mat(Info->yRes, Info->xRes, CV_8UC4, (char*)Info->imageStart, sizeof(cv::Vec4b)*Info->xRes);
cv::Mat GREYimg;
cv::cvtColor(BGRAimg, GREYimg, CV_BGRA2GRAY);
//set params into hog object
cv::HOGDescriptor hog;
hog.winSize = cv::Size(Params->winsize_width, Params->winsize_height);
hog.blockSize = cv::Size(Params->blocksize_width, Params->blocksize_height);
hog.blockStride = cv::Size(Params->blockstride_x, Params->blockstride_y);
hog.cellSize = cv::Size(Params->cellsize_width, Params->cellsize_height);
hog.nbins = Params->nBins;
hog.derivAperture = Params->derivAperture;
hog.winSigma = Params->win_sigma;
hog.L2HysThreshold = Params->threshold_L2hys;
hog.gammaCorrection = (Params->gammaCorrection != 0);
MgErr error = mgNoErr;
cv::vector<float> ders;
cv::vector<cv::Point> locations;
try
{
//winstride - step of window
//padding - borderpadding
//raises exception with incorrect params ... todo replace trycatch with paramchecking
hog.compute(GREYimg, ders, cv::Size(Params->winstride_x, Params->winstride_y), cv::Size(0,0), locations);
}
catch(...)
{
return -1;
}
//copy out the data into LabView
error = DSSetHandleSize(FeatArrHdl, sizeof(int32_t) + ders.size()*sizeof(float));
memcpy((*FeatArrHdl)->Arr, ders.data(), sizeof(float)*ders.size());
(*FeatArrHdl)->dimSize = ders.size();
return error;
}
I am running this function with following parameters:
Window size 32
Block size 16
Cell size 8
Block stride 8
Window stride 32
the rest of parameters is default.
I decided to include the look of the Mat object once constructed, I hope it can help.
This is the BGRA constructed from user data. It is supposed to be 640*640 BGRA
BGRAimg {flags=1124024344 dims=2 rows=640 ...} cv::Mat
flags 1124024344 int
dims 2 int
rows 640 int
cols 640 int
data 0x12250040 "e9%" unsigned char *
101 'e' unsigned char
refcount 0x00000000 int *
CXX0030: Error: expression cannot be evaluated
datastart 0x12250040 "e9%" unsigned char *
101 'e' unsigned char
dataend 0x123e0040 "" unsigned char *
0 unsigned char
datalimit 0x123e0040 "" unsigned char *
0 unsigned char
allocator 0x00000000 cv::MatAllocator *
__vfptr CXX0030: Error: expression cannot be evaluated
size {p=0x0012f44c } cv::Mat::MSize
p 0x0012f44c int *
640 int
step {p=0x0012f474 buf=0x0012f474 } cv::Mat::MStep
p 0x0012f474 unsigned int *
2560 unsigned int
buf 0x0012f474 unsigned int [2]
[0] 2560 unsigned int
[1] 4 unsigned int
And the Grey image that enters the HOG descriptors calculator
GREYimg {flags=1124024320 dims=2 rows=640 ...} cv::Mat
flags 1124024320 int
dims 2 int
rows 640 int
cols 640 int
refcount 0x0c867ff0 int *
1 int
dataend 0x0c867ff0 "" unsigned char *
1 '' unsigned char
datalimit 0x0c867ff0 "" unsigned char *
1 '' unsigned char
allocator 0x00000000 cv::MatAllocator *
__vfptr CXX0030: Error: expression cannot be evaluated
size {p=0x0012f40c } cv::Mat::MSize
p 0x0012f40c int *
640 int
step {p=0x0012f434 buf=0x0012f434 } cv::Mat::MStep
p 0x0012f434 unsigned int *
640 unsigned int
buf 0x0012f434 unsigned int [2]
[0] 640 unsigned int
[1] 1 unsigned int
I had to ommit the data and datastart fields, because unlike for the BGRA image MSVS actually shows some data in it.
UPDATE2
changed multi-threaded for multi-threaded DLL in project properities, and the issue is gone.
The problem persisted even if I was using code like this :
int dim = 32;
BYTE *mydata = NULL;
mydata = (BYTE*)malloc(sizeof(BYTE)*dim*dim);
Mat img;
img = Mat(Size(dim,dim), CV_8U, mydata, dim*sizeof(BYTE));
Might this indicate my code was not the cause, and this is somewhat opencv x windows runtime issue, or did I just hide the problem ?
UPDATE3
After reading something about microsoft runtime, I decided to check how was my opencv built, and it is using /MD, and I was building with /MT. I hope this was the cause.
this might not work like you expect:
sizeof(CV_8UC4)*Info->xRes
CV_8UC4 is an enum, not a type, you can't use sizeof() here.
if your data is continuous, you probably might just skip the stride param completely, or:
sizeof(Vec4b)*Info->xRes
another thing:
your BGRAimg has 4 channels, right ? so, use
cv::cvtColor(BGRAimg, GREYimg, CV_BGRA2GRAY);
instead
Let me start with a code clip:
QByteArray ba;
ba.resize(500000);
int encsize = avcodec_encode_video(context, (uint8_t*)ba.data(), 500000, frame.unownedPointer());
What I'm doing is encoding the data from frame and putting the data into the buffer pointed at QByteArray. If I comment out the avcodec_encode_video line my memory leak goes away. unownedPointer() looks like this:
if (this->frame != NULL) return this->frame;
this->frame = avcodec_alloc_frame();
uchar *data = this->img.bits();
frame->data[0] = (uint8_t *)data;
frame->data[1] = (uint8_t *)data + 1;
frame->data[2] = (uint8_t *)data + 2;
frame->linesize[0] = width * lineSize(this->fmt);
frame->linesize[1] = width * lineSize(this->fmt);
frame->linesize[2] = width * lineSize(this->fmt);
return this->frame;
Where this->frame is a AVFrame *, and this->img is a QImage.
At a encoding rate of about 30fps, I'm getting a memory leak of about 50MB/sec. So I'm not sure what the issue could be. It seems as if avcodec_encode_video() is copying memory and never freeing it or something. Any ideas?
If avcodec_encode_video is converting my RGB24 data to YUV420P would it be modifying the data pointed to by frame.unownedPointer()?
Take a look at the code for QtFFmpegwrapper it uses a saved context to do this efficently, or you can just use the QtFFMpegwrapper directly