Let me start with a code clip:
QByteArray ba;
ba.resize(500000);
int encsize = avcodec_encode_video(context, (uint8_t*)ba.data(), 500000, frame.unownedPointer());
What I'm doing is encoding the data from frame and putting the data into the buffer pointed at QByteArray. If I comment out the avcodec_encode_video line my memory leak goes away. unownedPointer() looks like this:
if (this->frame != NULL) return this->frame;
this->frame = avcodec_alloc_frame();
uchar *data = this->img.bits();
frame->data[0] = (uint8_t *)data;
frame->data[1] = (uint8_t *)data + 1;
frame->data[2] = (uint8_t *)data + 2;
frame->linesize[0] = width * lineSize(this->fmt);
frame->linesize[1] = width * lineSize(this->fmt);
frame->linesize[2] = width * lineSize(this->fmt);
return this->frame;
Where this->frame is a AVFrame *, and this->img is a QImage.
At a encoding rate of about 30fps, I'm getting a memory leak of about 50MB/sec. So I'm not sure what the issue could be. It seems as if avcodec_encode_video() is copying memory and never freeing it or something. Any ideas?
If avcodec_encode_video is converting my RGB24 data to YUV420P would it be modifying the data pointed to by frame.unownedPointer()?
Take a look at the code for QtFFmpegwrapper it uses a saved context to do this efficently, or you can just use the QtFFMpegwrapper directly
Related
I've been trying to copy a AVFrame just like what was answered in ffmpeg: make a copy from a decoded frame (AVFrame). However but I can't seem to get it to get a positive return code from av_frame_copy().
Here is basically what I'm doing:
AVFrame *copyFrame = NULL;
copyFrame = av_frame_alloc();
int return_code = av_frame_copy(copyFrame, originalFrame);
if(return_code < 0){
fprintf(stderr, "av_frame_copy failed with return code %d\n", return_code);
return(1);
}
If it helps, the return code I get from av_frame_copy is -22.
If you read the documentation for av_frame_copy, it says "This function does not allocate anything, dst must be already initialized and allocated with the same parameters as src."
av_frame_alloc doesn't do anything other than allocate the AVFrame struct and initialize it to some default values. Most importantly, it doesn't allocate buffers for the frame data or prepare the frame to be used. av_frame_copy is failing because the destination frame doesn't have the correct pixel format set or buffers allocated.
If you want to clone a frame (by incrementing its reference counter, not creating a deep copy) you can use av_frame_clone or av_frame_ref.
If you want to move the frame you can use av_frame_move_ref.
But you probably want to do a proper deep copy. In that case, you can look at the source code of the av_frame_make_writable. This function makes a deep copy of the frame if it isn't writeable, so we can use the same logic to make a deep copy of the frame here:
AVFrame *copyFrame = av_frame_alloc();
copyFrame->format = frame->format;
copyFrame->width = frame->width;
copyFrame->height = frame->height;
copyFrame->channels = frame->channels;
copyFrame->channel_layout = frame->channel_layout;
copyFrame->nb_samples = frame->nb_samples;
av_frame_get_buffer(copyFrame, 32);
av_frame_copy(copyFrame, frame);
av_frame_copy_props(copyFrame, frame);
Note that I haven't checked for errors in the functions I've called. You should do that in your real code. I omitted it here for brevity.
I had AVFrame * on GPU. This worked for me:
int ret;
AVFrame *dst;
dst = av_frame_alloc();
memcpy(dst,src,sizeof(AVFrame));
dst->format = src->format;
dst->width = src->width;
dst->height = src->height;
dst->channels = src->channels;
dst->channel_layout = src->channel_layout;
dst->nb_samples = src->nb_samples;
dst->extended_data = src->extended_data;
memcpy(dst->data, src->data, sizeof(src->data));
ret = av_frame_copy_props(dst, src);
if (ret < 0) { av_frame_unref(dst);}
AVFrame *copyFrame = new AVFrame;
copyFrame = av_frame_alloc();
*copyFrame = *inAVFrame;
if (int iRet = av_frame_copy(copyFrame, inAVFrame) == 0) {
//av_log(NULL, AV_LOG_INFO, "Ok");
} else {
//av_log(NULL, AV_LOG_INFO, "Error: %s\n", AV_err2str(iRet));
}
Here is basically what I'm doing :
I have a video stream (YUV format). Each frame is extracted into a buffer (frameBytes). Later, this buffer is used to do the YUV->RGB conversion, then transfered in an IplImage. The IplImage is then transfered into a cv::Mat and displayed in an OpenGL context. Everything works fine.
What I would like to do is bypass the IplImage and cv::Mat section to directly work with the frameBytes buffer in OpenGL and do the conversion in the shaders.
This explaination is just for the context, since the problem I'm having is simpler.
To see if I can work with the buffer earlier, I try to copy it with memcpy and then save it in a QImage then in a file.
Here is my code for this part :
unsigned char *mycopy = new unsigned char[1920*1080*3];
memcpy(mycopy, frameBytes, sizeof(1920*1080*3));
QImage *img = new QImage(mycopy, 1920, 1080, QImage::Format_RGB888);
img->save("image.jpg",0,-1);
frameBytes contains the YUV data from the video stream. I know it's YUV and I'm trying to create a QImage with RGB888 format but since QImage doesn't support the format, I didn't make the conversion there, I thought it would still save an image but with the wrong colors so I don't care for the moment (Maybe this assumption is wrong ?).
Problem is, the image saved is black.
Just for more information, here is an example where I use frameBytes for the YUV->RGB conversion.
void DeckLinkCaptureDelegate::convertFrameToOpenCV(void* frameBytes, IplImage * m_RGB){
if(!m_RGB) m_RGB = cvCreateImage(cvSize(1920, 1080), IPL_DEPTH_8U, 3);
unsigned char* pData = (unsigned char *) frameBytes;
for(int i = 0, j=0; i < 1920 * 1080 * 3; i+=6, j+=4)
{
unsigned char u = pData[j];
unsigned char y = pData[j+1];
unsigned char v = pData[j+2];
//fprintf(stderr, "%d\n", v);
m_RGB->imageData[i+2] = 1.0*y + 8 + 1.402*(v-128); // r
m_RGB->imageData[i+1] = 1.0*y - 0.34413*(u-128) - 0.71414*(v-128); // g
m_RGB->imageData[i] = 1.0*y + 1.772*(u-128) + 0; // b
y = pData[j+3];
m_RGB->imageData[i+5] = 1.0*y + 8 + 1.402*(v-128); // r
m_RGB->imageData[i+4] = 1.0*y - 0.34413*(u-128) - 0.71414*(v-128); // g
m_RGB->imageData[i+3] = 1.0*y + 1.772*(u-128) + 0;
}
}
Fixing the bug
You didn't copy all the data to your new buffer:
unsigned char *mycopy = new unsigned char[1920*1080*3];
memcpy(mycopy, frameBytes, sizeof(1920*1080*3));
That sizeof in there means that you're only copying an int-sized block, rather than 6MB. It looks like an accidental holdover from using a static array? Replace it with
const size_t bufsize = 1920*1080*3;
auto *mycopy = new unsigned char[bufsize];
memcpy(mycopy, frameBytes, bufsize);
A simpler approach
Alternatively, instead of doing the memory allocation yourself (and being responsible for delete[]ing it after the QImage is destructed), you could copy the image instead:
const unsigned char *bytes = frameBytes;
QImage img = QImage(bytes, 1920, 1080, QImage::Format_RGB888).copy();
The way this works is that we create a temporary QImage using frameBytes as its source (we pass it as pointer to const to insist it's read-only). We then copy() the whole of it to a new QImage, throwing away the temporary. The copy() does something similar to your code above, but we're now saved from having to do the calculations, eliminating some of the consequent potential for error.
Note also that I prefer to pass QImage by value. Although this might seem inefficient, most (copyable) Qt types are designed as reference-counted copy-on-write structures, and can be safely used this way, eliminating another class of errors (memory management). This is true of Qt's collection types, too, and very useful.
I wrote an npm module for capturing webcam input on linux. The captured frame in yuyv format is converted to rgb24 and after compressed to a jpeg image. In the jpeg compression there appears to be a memory leak. So the usage of memory increases continuously.
Image* rgb24_to_jpeg(Image *img, Image *jpeg) { // img = RGB24
jpeg_compress_struct cinfo;
jpeg_error_mgr jerr;
cinfo.err = jpeg_std_error(&jerr);
jerr.trace_level = 10;
jpeg_create_compress(&cinfo);
unsigned char *imgd = new unsigned char[img->size];
long unsigned int size = 0;
jpeg_mem_dest(&cinfo, &imgd, &size);
cinfo.image_width = img->width;
cinfo.image_height = img->height;
cinfo.input_components = 3;
cinfo.in_color_space = JCS_RGB;
jpeg_set_defaults(&cinfo);
jpeg_set_quality(&cinfo, 100, true);
jpeg_start_compress(&cinfo, true);
int row_stride = cinfo.image_width * 3;
JSAMPROW row_pointer[1];
while (cinfo.next_scanline < cinfo.image_height) {
row_pointer[0] = &img->data[cinfo.next_scanline * row_stride];
jpeg_write_scanlines(&cinfo, row_pointer, 1);
}
jpeg_finish_compress(&cinfo);
jpeg_destroy_compress(&cinfo);
// size += 512; // TODO: actual value to expand jpeg buffer... JPEG header?
if (jpeg->data == NULL) {
jpeg->data = (unsigned char *) malloc(size);
} else {
jpeg->data = (unsigned char *) realloc(jpeg->data, size);
}
memcpy(jpeg->data, imgd, size);
delete[] imgd;
jpeg->size = size;
return jpeg;
}
The rgb24 and jpeg buffers are reallocated on every cycle. So it looks like the leak is inside libjpeg layer. Is this true or I simply made a mistake somewhere in the code?
Note: the compressed image shall not be saved as a file, since the data might be used for live streaming.
You are using the jpeg_mem_dest in a wrong way - the second parameter is pointer to pointer to char because it is actually set by the library and then you must free it after you are done. Now you are initializing it with a pointer, it gets overwritten and you free the memory region allocated by the library but the original memory region is leaked.
This is how you should change your function:
Image* rgb24_to_jpeg(Image *img, Image *jpeg) { // img = RGB24
jpeg_compress_struct cinfo;
jpeg_error_mgr jerr;
cinfo.err = jpeg_std_error(&jerr);
jerr.trace_level = 10;
jpeg_create_compress(&cinfo);
unsigned char *imgd = 0;
long unsigned int size = 0;
cinfo.image_width = img->width;
cinfo.image_height = img->height;
cinfo.input_components = 3;
cinfo.in_color_space = JCS_RGB;
jpeg_set_defaults(&cinfo);
jpeg_set_quality(&cinfo, 100, true);
jpeg_mem_dest(&cinfo, &imgd, &size); // imgd will be set by the library
jpeg_start_compress(&cinfo, true);
int row_stride = cinfo.image_width * 3;
JSAMPROW row_pointer[1];
while (cinfo.next_scanline < cinfo.image_height) {
row_pointer[0] = &img->data[cinfo.next_scanline * row_stride];
jpeg_write_scanlines(&cinfo, row_pointer, 1);
}
jpeg_finish_compress(&cinfo);
jpeg_destroy_compress(&cinfo);
// size += 512; // TODO: actual value to expand jpeg buffer... JPEG header?
if (jpeg->data == NULL) {
jpeg->data = (unsigned char *) malloc(size);
} else if (jpeg->size != size) {
jpeg->data = (unsigned char *) realloc(jpeg->data, size);
}
memcpy(jpeg->data, imgd, size);
free(imgd); // dispose of imgd when you are done
jpeg->size = size;
return jpeg;
}
This snippet form jpeg_mem_dest explains the memory management:
if (*outbuffer == NULL || *outsize == 0) {
/* Allocate initial buffer */
dest->newbuffer = *outbuffer = (unsigned char *) malloc(OUTPUT_BUF_SIZE);
if (dest->newbuffer == NULL)
ERREXIT1(cinfo, JERR_OUT_OF_MEMORY, 10);
*outsize = OUTPUT_BUF_SIZE;
}
So, if you pass a an empty pointer or a zero sized buffer the library will perform an allocation for you. Thus - another approach is also to set the size correctly and then you can use the originally supplied pointer
In my case I did not solve the issue with previous answer, there was no way to free the memory image pointer, the only way to do that was reserving enough memory to the image and that way the library will not reserve memory and I have the control over the memory and is on the same heap of my application and not on the library's heap, here is my example:
//previous code...
struct jpeg_compress_struct cinfo;
//reserving the enough memory for my image (width * height)
unsigned char* _image = (unsigned char*)malloc(Width * Height);
//putting the reserved size into _imageSize
_imageSize = Width * Height;
//call the function like this:
jpeg_mem_dest(&cinfo, &_image, &_imageSize);
................
//releasing the reserved memory
free(_image);
NOTE: if you put _imageSize = 0, the library will assume that you have not reserve memory and the own library will do it.. so you need to put in _imageSize the amount of bytes reserved in _image
That way you have total control over the reserved memory and you can release it whenever you want in your software..
I am trying to encode an MP4 video using raw YUV frames data, but I am not sure how can I fill the plane data (preferably without using other libraries like ffmpeg)
The frame data is already encoded in I420, and does not need conversion.
Here is what I am trying to do:
const char *frameData = /* Raw frame data */;
x264_t *encoder = x264_encoder_open(¶m);
x264_picture_t imgInput, imgOutput;
x264_picture_alloc(&imgInput, X264_CSP_I420, width, height);
// how can I fill the struct data of imgInput
x264_nal_t *nals;
int i_nals;
int frameSize = x264_encoder_encode(encoder, &nals, &i_nals, &imgInput, &imgOutput);
The equivalent command line that I have found is :
x264 --output video.mp4 --fps 15 --input-res 1280x800 imgdata_01.raw
But I could not figure out how the app does it.
Thanks.
Look at libx264 API usage example. This example use fread() to fill frame allocated by x264_picture_alloc() with actual i420 data from stdin. If you already have i420 data in memory and want to skip memcpy step than instead of it you can:
Use x264_picture_init() instead of x264_picture_alloc() and x264_picture_clean(). Because you don't need allocate memory on heap for frame data.
Fill x264_picture_t.img struct fields:
i_csp = X264_CSP_I420;
i_plane = 3;
plane[0] = pointer to Y-plane;
i_stride[0] = stride in bytes for Y-plane;
plane[1] = pointer to U-plane;
i_stride[1] = stride in bytes for U-plane;
plane[2] = pointer to V-plane;
i_stride[2] = stride in bytes for V-plane;
To complete the answer above, this is an example to fill an x264_picture_t image.
int fillImage(uint8_t* buffer, int width, int height, x264_picture_t*pic){
int ret = x264_picture_alloc(pic, X264_CSP_I420, width, height);
if (ret < 0) return ret;
pic->img.i_plane = 3; // Y, U and V
pic->img.i_stride[0] = width;
// U and V planes are half the size of Y plane
pic->img.i_stride[1] = width / 2;
pic->img.i_stride[2] = width / 2;
int uvsize = ((width + 1) >> 1) * ((height + 1) >> 1);
pic->img.plane[0] = buffer; // Y Plane pointer
pic->img.plane[1] = buffer + (width * height); // U Plane pointer
pic->img.plane[2] = pic->img.plane[1] + uvsize; // V Plane pointer
return ret;
}
I'm using dcmtk library to modify the pixel data of a multi frame compressed dicom image. So, to do that, at one stage in an for loop I take the pixel data of each decompressed frame and modify them according my wish and try to concatenate each modify pixel data in a big memory buffer frame by frame. This core process of for loop is as below.
The problem is after the first iteration it gives memory at the line of the code where I call the function getUncompressedFrame. I think it's happening because of the line memcpy(fullBuffer+(i*sizeF),newBuffer,sizeF);, as when I remove that line there's no error at that time and the whole for loop works absolutely fine.
Could you please say me if I'm making a mistake in working with memcpy? Thanks.
Uint32 sizeF=828072;// I just wrote it to show what is the data type.
Uint8 * fullBuffer = new Uint8(int(sizeF*numOfFrames));//The big memory buffer
for(int i=0;i<numOfFrames;i++)
{
Uint8 * buffer = new Uint8[int(sizeF)];//Buffer for each frame
Uint8 * newBuffer = new Uint8[int(sizeF)];//Buffer in which the modified frame data is stored
DcmFileCache * cache=NULL;
OFCondition cond=element->getUncompressedFrame(dataset,i,startFragment,buffer,sizeF,decompressedColorModel,cache);
//I get the uncompressed individual frame pixel data
if(buffer != NULL)
{
for(unsigned long y = 0; y < rows; y++)
{
for(unsigned long x = 0; x < cols; x++)
{
if(planarConfiguration==0)
{
if(x>xmin && x<xmax && y>ymin && y<ymax)
{
index=(x + y + y*(cols-1))*samplePerPixel;
if(index<sizeF-2)
{
newBuffer[index] = 0;
newBuffer[index + 1] = 0;
newBuffer[index +2] = 0;
}
}
else
{
index=(x + y + y*(cols-1))*samplePerPixel;
if(index<sizeF-2)
{
newBuffer[index] = buffer[index];
newBuffer[index + 1] = buffer[index + 1];
newBuffer[index + 2] = buffer[index + 2];
}
}
}
}
}
memcpy(fullBuffer+(i*sizeF),newBuffer,sizeF);
//concatenate the modified frame by frame pixel data
}
Change the declaration of fullBuffer to this:
Uint8 * fullBuffer = new Uint8[int(sizeF*numOfFrames)];
Your code didn't allocate an array, it allocated a single Uint8 with the value int(sizeF*numOfFrames).
Uint8 * fullBuffer = new Uint8(int(sizeF*numOfFrames));
This allocates a single byte, giving it an initial value of sizeF*numOfFrames (after truncating it first to int and then to Uint8). You want an array, and you don't want to truncate the size to int:
Uint8 * fullBuffer = new Uint8[sizeF*numOfFrames];
^ ^
or, to fix the likely memory leaks in your code:
std::vector<Uint8> fullBuffer(sizeF*numOfFrames);
If the method getUncompressedFrame is doing an inner memcpy to cache, then it makes sense why, as you are passing a null pointer as argument for the cache, with no memory allocated.