I am writing a gstreamer plugin to render the camera video(height=480, width=640 , format UYVY) on top of a HD video frame. In my chain function (gst_plugin_template_chain (GstPad * pad, GstObject * parent, GstBuffer * buf)) I can see the buf size as 480x640x2. I allocated a new buffer equivalent to Full HD and copied this UYVY data on top of the new buffer and replaced the existing buffer. But output video is still height=480, width=640.
Here are the code snippet.
gst_plugin_template_chain (GstPad * pad, GstObject * parent, GstBuffer * buf)
{
GstPluginTemplate *filter;
GstMapInfo map;
guchar *data;
gint width = 640, height = 480;
GstMemory *mem;
int row = 0;
filter = GST_PLUGIN_TEMPLATE (parent);
gst_buffer_map (buf, &map, GST_MAP_READWRITE);
mem = gst_allocator_alloc(NULL, 4147200, NULL); //1080*1920*2 (Full HD UYVY) = 4147200
GstMapInfo info_out;
gst_memory_map(mem, &info_out, GST_MAP_WRITE);
for(int i = 0; i < 480 ; i++) {
memcpy(info_out.data + row, map.data + row, 1280);
row += 1280; //640*2
}
gst_buffer_replace_all_memory(buf, mem);
gst_buffer_unmap (buf, &map);
gst_memory_unmap(mem, &info_out);
return gst_pad_push (filter->srcpad, buf);
}
You need to send a new CAPS event downstream. Probably you want gst_event_new_caps() and push it through the source pad. But double check the documentation about what you want exactly.
Thank you Florian Zwoch for the support . It is working now after adding below changes.
Invoked custom gst_my_filter_setcaps (filter, caps) from gst_my_filter_sink_event.
Related
I am trying to extract frames from a stream which I create with Gstreamer and trying to save them with FreeImage or QImage ( this one is for testing ).
GstMapInfo bufferInfo;
GstBuffer *sampleBuffer;
GstStructure *capsStruct;
GstSample *sample;
GstCaps *caps;
int width, height;
const int BitsPP = 32;
/* Retrieve the buffer */
g_signal_emit_by_name (sink, "pull-sample", &sample);
if (sample) {
sampleBuffer = gst_sample_get_buffer(sample);
gst_buffer_map(sampleBuffer,&bufferInfo,GST_MAP_READ);
if (!bufferInfo.data) {
g_printerr("Warning: could not map GStreamer buffer!\n");
throw;
}
caps = gst_sample_get_caps(sample);
capsStruct= gst_caps_get_structure(caps,0);
gst_structure_get_int(capsStruct,"width",&width);
gst_structure_get_int(capsStruct,"height",&height);
auto bitmap = FreeImage_Allocate(width, height, BitsPP,0,0,0);
memcpy( FreeImage_GetBits( bitmap ), bufferInfo.data, width * height * (BitsPP/8));
// int pitch = ((((BitsPP * width) + 31) / 32) * 4);
// auto bitmap = FreeImage_ConvertFromRawBits(bufferInfo.data,width,height,pitch,BitsPP,0, 0, 0);
FreeImage_FlipHorizontal(bitmap);
bitmap = FreeImage_RotateClassic(bitmap,180);
static int id = 0;
std::string name = "/home/stadmin/pic/sample" + std::to_string(id++) + ".png";
#ifdef FREE_SAVE
FreeImage_Save(FIF_PNG,bitmap,name.c_str());
#endif
#ifdef QT_SAVE
//Format_ARGB32
QImage image(bufferInfo.data,width,height,QImage::Format_ARGB32);
image.save(QString::fromStdString(name));
#endif
fibPipeline.push(bitmap);
gst_sample_unref(sample);
gst_buffer_unmap(sampleBuffer, &bufferInfo);
return GST_FLOW_OK;
The color output in FreeImage are totally wrong like when Qt - Format_ARGB32 [ greens like blue or blues like oranges etc.. ] but when I test with Qt - Format_RGBA8888 I can get correct output. I need to use FreeImage and I wish to learn how to correct this.
Since you say Qt succeeds using Format_RGBA8888, I can only guess: the gstreamer frame has bytes in RGBA order while FreeImage expects ARGB.
Quick fix:
//have a buffer the same length of the incoming bytes
size_t length = width * height * (BitsPP/8);
BYTE * bytes = (BYTE *) malloc(length);
//copy the incoming bytes to it, in the right order:
int index = 0;
while(index < length)
{
bytes[index] = bufferInfo.data[index + 2]; //B
bytes[index + 1] = bufferInfo.data[index + 1]; //G
bytes[index + 2] = bufferInfo.data[index]; //R
bytes[index + 3] = bufferInfo.data[index + 3]; //A
index += 4;
}
//fill the bitmap using the buffer
auto bitmap = FreeImage_Allocate(width, height, BitsPP,0,0,0);
memcpy( FreeImage_GetBits( bitmap ), bytes, length);
//don't forget to
free(bytes);
I would like to update an AV Audio encoder using function avcodec_encode_audio (deprecated) to avcodec_encode_audio2, without modifying the structure of existing encoder:
outBytes = avcodec_encode_audio(m_handle, dst, sizeBytes, (const short int*)m_samBuf);
where:
1) m_handle AVCodecContext
2) dst, uint8_t * destination buffer
3) sizeBytes, uint32_t size of the destination buffer
4) m_samBuf void * to the input chunk of data to encode (this is casted to: const short int*)
is there a simply way to do it?
Im tryng with:
int gotPack = 1;
memset (&m_Packet, 0, sizeof (m_Packet));
m_Frame = av_frame_alloc();
av_init_packet(&m_Packet);
m_Packet.data = dst;
m_Packet.size = sizeBytes;
uint8_t* buffer = (uint8_t*)m_samBuf;
m_Frame->nb_samples = m_handle->frame_size;
avcodec_fill_audio_frame(m_Frame,m_handle->channels,m_handle->sample_fmt,buffer,m_FrameSize,1);
outBytes = avcodec_encode_audio2(m_handle, &m_Packet, m_Frame, &gotPack);
char error[256];
av_strerror(outBytes,error,256);
if (outBytes<0){
m_server->log(1,1,"Input data: %d, encode function call error: %s \n",gotPack, error);
return AUDIOWRAPPER_ERROR;
}
av_frame_free(&m_Frame);
it compiles but it does not encode anything, i dont here audio at the output if I pipe the output stream on mplayer, wich was warking prior to the upgrade.
What am I doing wrong?
The encoder accept only two sample formats:
AV_SAMPLE_FMT_S16, ///< signed 16 bits
AV_SAMPLE_FMT_FLT, ///< float
here is how the buffer is allocated:
free(m_samBuf);
int bps = 2;
if(m_handle->codec->sample_fmts[0] == AV_SAMPLE_FMT_FLT) {
bps = 4;
}
m_FrameSize = bps * m_handle->frame_size * m_handle->channels;
m_samBuf = malloc(m_FrameSize);
m_numSam = 0;
avcodec_fill_audio_frame should get you there
memset (&m_Packet, 0, sizeof (m_Packet));
memset (&m_Frame, 0, sizeof (m_Frame));
av_init_packet(&m_Packet);
m_Packet.data = dst;
m_Packet.size = sizeBytes;
m_Frame->nb_samples = //you need to get this value from somewhere, it is the number of samples (per channel) this frame represents
avcodec_fill_audio_frame(m_Frame, m_handle->channels, m_handle->sample_fmt,
buffer,
sizeBytes, 1);
int gotPack = 1;
avcodec_encode_audio2(m_handle, &m_Packet, &m_Frame, &gotPack);
I wrote an npm module for capturing webcam input on linux. The captured frame in yuyv format is converted to rgb24 and after compressed to a jpeg image. In the jpeg compression there appears to be a memory leak. So the usage of memory increases continuously.
Image* rgb24_to_jpeg(Image *img, Image *jpeg) { // img = RGB24
jpeg_compress_struct cinfo;
jpeg_error_mgr jerr;
cinfo.err = jpeg_std_error(&jerr);
jerr.trace_level = 10;
jpeg_create_compress(&cinfo);
unsigned char *imgd = new unsigned char[img->size];
long unsigned int size = 0;
jpeg_mem_dest(&cinfo, &imgd, &size);
cinfo.image_width = img->width;
cinfo.image_height = img->height;
cinfo.input_components = 3;
cinfo.in_color_space = JCS_RGB;
jpeg_set_defaults(&cinfo);
jpeg_set_quality(&cinfo, 100, true);
jpeg_start_compress(&cinfo, true);
int row_stride = cinfo.image_width * 3;
JSAMPROW row_pointer[1];
while (cinfo.next_scanline < cinfo.image_height) {
row_pointer[0] = &img->data[cinfo.next_scanline * row_stride];
jpeg_write_scanlines(&cinfo, row_pointer, 1);
}
jpeg_finish_compress(&cinfo);
jpeg_destroy_compress(&cinfo);
// size += 512; // TODO: actual value to expand jpeg buffer... JPEG header?
if (jpeg->data == NULL) {
jpeg->data = (unsigned char *) malloc(size);
} else {
jpeg->data = (unsigned char *) realloc(jpeg->data, size);
}
memcpy(jpeg->data, imgd, size);
delete[] imgd;
jpeg->size = size;
return jpeg;
}
The rgb24 and jpeg buffers are reallocated on every cycle. So it looks like the leak is inside libjpeg layer. Is this true or I simply made a mistake somewhere in the code?
Note: the compressed image shall not be saved as a file, since the data might be used for live streaming.
You are using the jpeg_mem_dest in a wrong way - the second parameter is pointer to pointer to char because it is actually set by the library and then you must free it after you are done. Now you are initializing it with a pointer, it gets overwritten and you free the memory region allocated by the library but the original memory region is leaked.
This is how you should change your function:
Image* rgb24_to_jpeg(Image *img, Image *jpeg) { // img = RGB24
jpeg_compress_struct cinfo;
jpeg_error_mgr jerr;
cinfo.err = jpeg_std_error(&jerr);
jerr.trace_level = 10;
jpeg_create_compress(&cinfo);
unsigned char *imgd = 0;
long unsigned int size = 0;
cinfo.image_width = img->width;
cinfo.image_height = img->height;
cinfo.input_components = 3;
cinfo.in_color_space = JCS_RGB;
jpeg_set_defaults(&cinfo);
jpeg_set_quality(&cinfo, 100, true);
jpeg_mem_dest(&cinfo, &imgd, &size); // imgd will be set by the library
jpeg_start_compress(&cinfo, true);
int row_stride = cinfo.image_width * 3;
JSAMPROW row_pointer[1];
while (cinfo.next_scanline < cinfo.image_height) {
row_pointer[0] = &img->data[cinfo.next_scanline * row_stride];
jpeg_write_scanlines(&cinfo, row_pointer, 1);
}
jpeg_finish_compress(&cinfo);
jpeg_destroy_compress(&cinfo);
// size += 512; // TODO: actual value to expand jpeg buffer... JPEG header?
if (jpeg->data == NULL) {
jpeg->data = (unsigned char *) malloc(size);
} else if (jpeg->size != size) {
jpeg->data = (unsigned char *) realloc(jpeg->data, size);
}
memcpy(jpeg->data, imgd, size);
free(imgd); // dispose of imgd when you are done
jpeg->size = size;
return jpeg;
}
This snippet form jpeg_mem_dest explains the memory management:
if (*outbuffer == NULL || *outsize == 0) {
/* Allocate initial buffer */
dest->newbuffer = *outbuffer = (unsigned char *) malloc(OUTPUT_BUF_SIZE);
if (dest->newbuffer == NULL)
ERREXIT1(cinfo, JERR_OUT_OF_MEMORY, 10);
*outsize = OUTPUT_BUF_SIZE;
}
So, if you pass a an empty pointer or a zero sized buffer the library will perform an allocation for you. Thus - another approach is also to set the size correctly and then you can use the originally supplied pointer
In my case I did not solve the issue with previous answer, there was no way to free the memory image pointer, the only way to do that was reserving enough memory to the image and that way the library will not reserve memory and I have the control over the memory and is on the same heap of my application and not on the library's heap, here is my example:
//previous code...
struct jpeg_compress_struct cinfo;
//reserving the enough memory for my image (width * height)
unsigned char* _image = (unsigned char*)malloc(Width * Height);
//putting the reserved size into _imageSize
_imageSize = Width * Height;
//call the function like this:
jpeg_mem_dest(&cinfo, &_image, &_imageSize);
................
//releasing the reserved memory
free(_image);
NOTE: if you put _imageSize = 0, the library will assume that you have not reserve memory and the own library will do it.. so you need to put in _imageSize the amount of bytes reserved in _image
That way you have total control over the reserved memory and you can release it whenever you want in your software..
I am trying to encode an MP4 video using raw YUV frames data, but I am not sure how can I fill the plane data (preferably without using other libraries like ffmpeg)
The frame data is already encoded in I420, and does not need conversion.
Here is what I am trying to do:
const char *frameData = /* Raw frame data */;
x264_t *encoder = x264_encoder_open(¶m);
x264_picture_t imgInput, imgOutput;
x264_picture_alloc(&imgInput, X264_CSP_I420, width, height);
// how can I fill the struct data of imgInput
x264_nal_t *nals;
int i_nals;
int frameSize = x264_encoder_encode(encoder, &nals, &i_nals, &imgInput, &imgOutput);
The equivalent command line that I have found is :
x264 --output video.mp4 --fps 15 --input-res 1280x800 imgdata_01.raw
But I could not figure out how the app does it.
Thanks.
Look at libx264 API usage example. This example use fread() to fill frame allocated by x264_picture_alloc() with actual i420 data from stdin. If you already have i420 data in memory and want to skip memcpy step than instead of it you can:
Use x264_picture_init() instead of x264_picture_alloc() and x264_picture_clean(). Because you don't need allocate memory on heap for frame data.
Fill x264_picture_t.img struct fields:
i_csp = X264_CSP_I420;
i_plane = 3;
plane[0] = pointer to Y-plane;
i_stride[0] = stride in bytes for Y-plane;
plane[1] = pointer to U-plane;
i_stride[1] = stride in bytes for U-plane;
plane[2] = pointer to V-plane;
i_stride[2] = stride in bytes for V-plane;
To complete the answer above, this is an example to fill an x264_picture_t image.
int fillImage(uint8_t* buffer, int width, int height, x264_picture_t*pic){
int ret = x264_picture_alloc(pic, X264_CSP_I420, width, height);
if (ret < 0) return ret;
pic->img.i_plane = 3; // Y, U and V
pic->img.i_stride[0] = width;
// U and V planes are half the size of Y plane
pic->img.i_stride[1] = width / 2;
pic->img.i_stride[2] = width / 2;
int uvsize = ((width + 1) >> 1) * ((height + 1) >> 1);
pic->img.plane[0] = buffer; // Y Plane pointer
pic->img.plane[1] = buffer + (width * height); // U Plane pointer
pic->img.plane[2] = pic->img.plane[1] + uvsize; // V Plane pointer
return ret;
}
I'm using the Leptonica Library to process some pictures. After that I want to show them in my QT GUI. Leptonica is using their own format Pix for the images, while QT is using their own format QPixmap. At the moment the only way for me is to save the pictures after processing as a file ( like bmp ) and then load them again with a QT function call. Now I want to convert them in my code, so I dont need to take the detour with saving them on the filesystem. Any ideas how to do this?
Best Regards
// edit:
Okay as already suggested I tried to convert the PIX* to a QImage.
The PIX* is defined like this:
http://tpgit.github.com/Leptonica/pix_8h_source.html
struct Pix
{
l_uint32 w; /* width in pixels */
l_uint32 h; /* height in pixels */
l_uint32 d; /* depth in bits */
l_uint32 wpl; /* 32-bit words/line */
l_uint32 refcount; /* reference count (1 if no clones) */
l_int32 xres; /* image res (ppi) in x direction */
/* (use 0 if unknown) */
l_int32 yres; /* image res (ppi) in y direction */
/* (use 0 if unknown) */
l_int32 informat; /* input file format, IFF_* */
char *text; /* text string associated with pix */
struct PixColormap *colormap; /* colormap (may be null) */
l_uint32 *data; /* the image data */
};
while QImage offers me a method like this:
http://developer.qt.nokia.com/doc/qt-4.8/qimage.html#QImage-7
QImage ( const uchar * data,
int width,
int height,
int bytesPerLine,
Format format )
I assume I cant just copy the data from the PIX to the QImage when calling the constructor. I guess I need to fill the QImage Pixel by Pixel, but actually I dont know how? Do I need to loop through all the coordinates? How do I regard the bit depth? Any ideas here?
I use this for conversion QImage to PIX:
PIX* TessTools::qImage2PIX(QImage& qImage) {
PIX * pixs;
l_uint32 *lines;
qImage = qImage.rgbSwapped();
int width = qImage.width();
int height = qImage.height();
int depth = qImage.depth();
int wpl = qImage.bytesPerLine() / 4;
pixs = pixCreate(width, height, depth);
pixSetWpl(pixs, wpl);
pixSetColormap(pixs, NULL);
l_uint32 *datas = pixs->data;
for (int y = 0; y < height; y++) {
lines = datas + y * wpl;
QByteArray a((const char*)qImage.scanLine(y), qImage.bytesPerLine());
for (int j = 0; j < a.size(); j++) {
*((l_uint8 *)lines + j) = a[j];
}
}
return pixEndianByteSwapNew(pixs);
}
And this for conversion PIX to QImage:
QImage TessTools::PIX2QImage(PIX *pixImage) {
int width = pixGetWidth(pixImage);
int height = pixGetHeight(pixImage);
int depth = pixGetDepth(pixImage);
int bytesPerLine = pixGetWpl(pixImage) * 4;
l_uint32 * s_data = pixGetData(pixEndianByteSwapNew(pixImage));
QImage::Format format;
if (depth == 1)
format = QImage::Format_Mono;
else if (depth == 8)
format = QImage::Format_Indexed8;
else
format = QImage::Format_RGB32;
QImage result((uchar*)s_data, width, height, bytesPerLine, format);
// Handle pallete
QVector<QRgb> _bwCT;
_bwCT.append(qRgb(255,255,255));
_bwCT.append(qRgb(0,0,0));
QVector<QRgb> _grayscaleCT(256);
for (int i = 0; i < 256; i++) {
_grayscaleCT.append(qRgb(i, i, i));
}
if (depth == 1) {
result.setColorTable(_bwCT);
} else if (depth == 8) {
result.setColorTable(_grayscaleCT);
} else {
result.setColorTable(_grayscaleCT);
}
if (result.isNull()) {
static QImage none(0,0,QImage::Format_Invalid);
qDebug() << "***Invalid format!!!";
return none;
}
return result.rgbSwapped();
}
This code accepts a const QImage& parameter.
static PIX* makePIXFromQImage(const QImage &image)
{
QByteArray ba;
QBuffer buf(&ba);
buf.open(QIODevice::WriteOnly);
image.save(&buf, "BMP");
return pixReadMemBmp(ba.constData(), ba.size());
}
I do not know the Leptonica Library, but I had a short look at the documentation and found the documentation about the PIX structure. You can create a QImage from the raw data and convert this to a QPixmap with convertFromImage.
Well I could solve the problem this way:
Leptonica offers a function
l_int32 pixWriteMemBmp (l_uint8 **pdata, size_t *psize, PIX *pix)
With this function you can write into the memory instead of a filestream. Still ( in this example ) the Bmp Header and format persists ( there are the same functions for other image formats too ).
The corresponding function from QT is this one:
bool QImage::loadFromData ( const uchar * data, int len, const char * format = 0 )
Since the the Header persits I just need to pass the data ptr and the size to the loadFromData function and QT does the rest.
So all together it would be like this:
PIX *m_pix;
FILE * pFile;
pFile = fopen( "PathToFile", "r" );
m_pix = pixReadStreamBmp(pFile); // If other file format use the according function
fclose(pFile);
// Now we have a Pix object from leptonica
l_uint8* ptr_memory;
size_t len;
pixWriteMemBmp(&ptr_memory, &size, m_pix);
// Now we have the picture somewhere in the memory
QImage testimage;
QPixmap pixmap;
testimage.loadFromData((uchar *)ptr_memory,len);
pixmap.convertFromImage(testimage);
// Now we have the image as a pixmap in Qt
This actually works for me, tho I don't know if there is a way to do this backwards so easy. ( If there is, please let me know )
Best Regards
You can save your pixmap to RAM instead of file (use QByteArray to store the data, and QBuffer as your I/O device).