I'm trying to convert RGB32 frames to NV12 Frames to feed into an encoder.
m_iWidthIn = 1920;
m_iHeightIn = 1080;
m_iWidthOut = (((iWidthIn + 31) >> 5) << 5) //32bit align
m_heightOut = (((iHeightIn + 31) >> 5) << 5) //32bit align
m_outputPixelFormat = AV_PIX_FMT_NV12;
// allocate and fill buffers
m_sws = ::sws_getContext(m_iWidthIn, m_iHeightIn, AV_PIX_FMT_RGB32, m_iWidthOut, m_iHeightOut, m_outputPixelFormat, SWS_FAST_BILINEAR, nullptr, nullptr, nullptr);
AVFrame* frameOut = av_frame_alloc();
frameOut->height = m_iHeightOut;
frameOut->width = m_iWidthOut;
frameOut->format = m_outputPixelFormat;
av_frame_get_buffer(frameOut, 32);
int linesize[1] = { m_iWidthIn * 4 };
uint8_t * data[1] = { m_inputBuffer };
if (m_bFlip)
{
data[0] += linesize[0] * (m_iHeightIn - 1);
linesize[0] = -linesize[0];
}
::sws_scale(m_sws, data, linesize, 0, m_iHeightIn, frameOut->data, frameOut->linesize);
::av_image_copy_to_buffer(pOutputBuffer, lDataLen, frameOut->data, frameOut->linesize, m_outputPixelFormat, m_iWidthOut, m_iHeightOut, 32);
If I make m_outputPixelFormat AV_PIX_FMT_RGB32 and use a DMO colorspace converter, the video comes out correctly. However if I change it to NV12, I end up with a slanted video with missing data at the bottom.
I know this is caused by me copying the data incorrectly out of the buffer, but I'm unsure what I'm doing incorrectly.
Your problem is here:
m_heightOut = (((iHeightIn + 31) >> 5) << 5) //32bit align
You don't need to align height. So frameOut->data has m_iHeightIn height.
The correct line is:
m_heightOut = iHeightIn;
Related
I am trying to extract frames from a stream which I create with Gstreamer and trying to save them with FreeImage or QImage ( this one is for testing ).
GstMapInfo bufferInfo;
GstBuffer *sampleBuffer;
GstStructure *capsStruct;
GstSample *sample;
GstCaps *caps;
int width, height;
const int BitsPP = 32;
/* Retrieve the buffer */
g_signal_emit_by_name (sink, "pull-sample", &sample);
if (sample) {
sampleBuffer = gst_sample_get_buffer(sample);
gst_buffer_map(sampleBuffer,&bufferInfo,GST_MAP_READ);
if (!bufferInfo.data) {
g_printerr("Warning: could not map GStreamer buffer!\n");
throw;
}
caps = gst_sample_get_caps(sample);
capsStruct= gst_caps_get_structure(caps,0);
gst_structure_get_int(capsStruct,"width",&width);
gst_structure_get_int(capsStruct,"height",&height);
auto bitmap = FreeImage_Allocate(width, height, BitsPP,0,0,0);
memcpy( FreeImage_GetBits( bitmap ), bufferInfo.data, width * height * (BitsPP/8));
// int pitch = ((((BitsPP * width) + 31) / 32) * 4);
// auto bitmap = FreeImage_ConvertFromRawBits(bufferInfo.data,width,height,pitch,BitsPP,0, 0, 0);
FreeImage_FlipHorizontal(bitmap);
bitmap = FreeImage_RotateClassic(bitmap,180);
static int id = 0;
std::string name = "/home/stadmin/pic/sample" + std::to_string(id++) + ".png";
#ifdef FREE_SAVE
FreeImage_Save(FIF_PNG,bitmap,name.c_str());
#endif
#ifdef QT_SAVE
//Format_ARGB32
QImage image(bufferInfo.data,width,height,QImage::Format_ARGB32);
image.save(QString::fromStdString(name));
#endif
fibPipeline.push(bitmap);
gst_sample_unref(sample);
gst_buffer_unmap(sampleBuffer, &bufferInfo);
return GST_FLOW_OK;
The color output in FreeImage are totally wrong like when Qt - Format_ARGB32 [ greens like blue or blues like oranges etc.. ] but when I test with Qt - Format_RGBA8888 I can get correct output. I need to use FreeImage and I wish to learn how to correct this.
Since you say Qt succeeds using Format_RGBA8888, I can only guess: the gstreamer frame has bytes in RGBA order while FreeImage expects ARGB.
Quick fix:
//have a buffer the same length of the incoming bytes
size_t length = width * height * (BitsPP/8);
BYTE * bytes = (BYTE *) malloc(length);
//copy the incoming bytes to it, in the right order:
int index = 0;
while(index < length)
{
bytes[index] = bufferInfo.data[index + 2]; //B
bytes[index + 1] = bufferInfo.data[index + 1]; //G
bytes[index + 2] = bufferInfo.data[index]; //R
bytes[index + 3] = bufferInfo.data[index + 3]; //A
index += 4;
}
//fill the bitmap using the buffer
auto bitmap = FreeImage_Allocate(width, height, BitsPP,0,0,0);
memcpy( FreeImage_GetBits( bitmap ), bytes, length);
//don't forget to
free(bytes);
I was trying to pack multiple images in a single image, using Bin Packing algorithm. In the part of adding images in a single image I was trying with collecting all the image pixel values and put them in the empty frame, but this is not working. Is there any suggestions?
Hi Edited the question,
` FIBITMAP *out_bmp = FreeImage_Allocate(4096, 4096, 32, 0, 0, 0);
BYTE *out_bits = FreeImage_GetBits(out_bmp);
int out_pitch = FreeImage_GetPitch(out_bmp);
// copy all the images to the final one
for (int i = 0; i < files.size(); i++) {
string s = "PathToFile" + files[i];
FIBITMAP* img0 = FreeImage_Load(FreeImage_GetFileType(s.c_str(), 0), s.c_str());
// make sure the input picture is 32-bits
if (FreeImage_GetBPP(img0) != 32) {
FIBITMAP *new_bmp = FreeImage_ConvertTo32Bits(img0);
FreeImage_Unload(img0);
img0 = new_bmp;
}
int img_pitch = FreeImage_GetPitch(img0);
BYTE *img_bits = FreeImage_GetBits(img0);
BYTE *out_bits_ptr = out_bits + out_pitch *
FreeImage_GetHeight(img0) + 4 * FreeImage_GetWidth(img0);
for (int y = 0; y < FreeImage_GetHeight(img0); y += 1) {
memcpy(out_bits_ptr, img_bits, FreeImage_GetWidth(img0) * 4);
out_bits_ptr += out_pitch;
img_bits += img_pitch;
}
}`
i'm using the java method below to write an android.graphics.Bitmap to tga, i've opened the photo in photoshop and it's allright. in native i have to load and display this image with opengl, but the loading of image is incorrect and i see incorrect colors on the screen, the c++ tga loader is below. anybody has any ideea what's the problem?
java write tga method:
public static void writeTGA(Bitmap src, String path) throws IOException {
ByteBuffer buffer = ByteBuffer.allocate(src.getRowBytes() * src.getHeight());
src.copyPixelsToBuffer(buffer);
boolean alpha = src.hasAlpha();
byte[] data;
byte[] pixels = buffer.array();
if (pixels.length != src.getWidth() * src.getHeight() * (alpha ? 4 : 3))
throw new IllegalStateException();
data = new byte[pixels.length];
for(int i=0;i < pixels.length; i += 4){// rgba -> bgra
data[i] = pixels[i+2];
data[i+1] = pixels[i+1];
data[i+2] = pixels[i];
data[i+3] = pixels[i+3];
}
byte[] header = new byte[18];
header[2] = 2; // uncompressed, true-color image
header[12] = (byte) ((src.getWidth() >> 0) & 0xFF);
header[13] = (byte) ((src.getWidth() >> 8) & 0xFF);
header[14] = (byte) ((src.getHeight() >> 0) & 0xFF);
header[15] = (byte) ((src.getHeight() >> 8) & 0xFF);
header[16] = (byte) (alpha ? 32 : 24); // bits per pixel
header[17] = (byte) ((alpha ? 8 : 0) | (1 << 4));
File file = new File(path);
RandomAccessFile raf = new RandomAccessFile(file, "rw");
raf.write(header);
raf.write(data);
raf.setLength(raf.getFilePointer()); // trim
raf.close();
}
tga 18 bit header c++ :
typedef struct _tgaheader {
BYTE IDLength; /* 00h Size of Image ID field */
BYTE ColorMapType; /* 01h Color map type */
BYTE ImageType; /* 02h Image type code */
BYTE CMapStart[2]; /* 03h Color map origin */
BYTE CMapLength[2]; /* 05h Color map length */
BYTE CMapDepth; /* 07h Depth of color map entries */
WORD XOffset; /* 08h X origin of image */
WORD YOffset; /* 0Ah Y origin of image */
WORD Width; /* 0Ch Width of image */
WORD Height; /* 0Eh Height of image */
BYTE PixelDepth; /* 10h Image pixel size */
BYTE ImageDescriptor; /* 11h Image descriptor byte */
} TGAHEADER;
tga loader method:
void TgaFormat:: LoadImage(const char *path) {
FILE* filePtr = fopen(path, "rb");
long imageSize;
short pixel_size;
unsigned char colorSwap;
// Open the TGA file.
if( filePtr == NULL){
LOGI("cannot find Tga File!");
return;
}
fread(&file_header, 1, sizeof(TGAHEADER), filePtr);
short sz = sizeof(TGAHEADER);
// 2 (uncompressed RGB image), 3 (uncompressed black-and-white images).
if (file_header.ImageType != 2 ){
fclose(filePtr);
LOGI("this file is not a TGA!");
return;
}
// Color mode -> 3 = BGR, 4 = BGRA.
pixel_size = file_header.PixelDepth / 8;
imageSize = file_header.Width * file_header.Height * pixel_size;
m_rgba_data = (BYTE* )malloc( sizeof(BYTE) * imageSize );
if( fread(m_rgba_data, 1, imageSize, filePtr) != imageSize ) {
fclose(filePtr);
return ;
}
fclose(filePtr);
// Change from BGRA to RGBA so OpenGL can read the image data.
for (int imageIdx = 0; imageIdx < imageSize; imageIdx += pixel_size) {
colorSwap = m_rgba_data[imageIdx];
m_rgba_data[imageIdx] = m_rgba_data[imageIdx + 2];
m_rgba_data[imageIdx + 2] = colorSwap;
}
}
after reading the tga file in android native and rendered with opengles
the generated qr code into sdcard the opened with photoshop
the second photo was writed in java, then open in photoshop. i found the mistake. as i was been thinking, i've got a wrong offset, but not in writing/reading process.
into the upload to gpu:
I had
glTexImage2D(GL_TEXTURE_2D, 0, GL_RGB.....);
instead of
glTexImage2D(GL_TEXTURE_2D, 0, GL_RGBA....);
because my pixel size is 4 (RGBA) not 3 (RGB).
I am trying to encode an MP4 video using raw YUV frames data, but I am not sure how can I fill the plane data (preferably without using other libraries like ffmpeg)
The frame data is already encoded in I420, and does not need conversion.
Here is what I am trying to do:
const char *frameData = /* Raw frame data */;
x264_t *encoder = x264_encoder_open(¶m);
x264_picture_t imgInput, imgOutput;
x264_picture_alloc(&imgInput, X264_CSP_I420, width, height);
// how can I fill the struct data of imgInput
x264_nal_t *nals;
int i_nals;
int frameSize = x264_encoder_encode(encoder, &nals, &i_nals, &imgInput, &imgOutput);
The equivalent command line that I have found is :
x264 --output video.mp4 --fps 15 --input-res 1280x800 imgdata_01.raw
But I could not figure out how the app does it.
Thanks.
Look at libx264 API usage example. This example use fread() to fill frame allocated by x264_picture_alloc() with actual i420 data from stdin. If you already have i420 data in memory and want to skip memcpy step than instead of it you can:
Use x264_picture_init() instead of x264_picture_alloc() and x264_picture_clean(). Because you don't need allocate memory on heap for frame data.
Fill x264_picture_t.img struct fields:
i_csp = X264_CSP_I420;
i_plane = 3;
plane[0] = pointer to Y-plane;
i_stride[0] = stride in bytes for Y-plane;
plane[1] = pointer to U-plane;
i_stride[1] = stride in bytes for U-plane;
plane[2] = pointer to V-plane;
i_stride[2] = stride in bytes for V-plane;
To complete the answer above, this is an example to fill an x264_picture_t image.
int fillImage(uint8_t* buffer, int width, int height, x264_picture_t*pic){
int ret = x264_picture_alloc(pic, X264_CSP_I420, width, height);
if (ret < 0) return ret;
pic->img.i_plane = 3; // Y, U and V
pic->img.i_stride[0] = width;
// U and V planes are half the size of Y plane
pic->img.i_stride[1] = width / 2;
pic->img.i_stride[2] = width / 2;
int uvsize = ((width + 1) >> 1) * ((height + 1) >> 1);
pic->img.plane[0] = buffer; // Y Plane pointer
pic->img.plane[1] = buffer + (width * height); // U Plane pointer
pic->img.plane[2] = pic->img.plane[1] + uvsize; // V Plane pointer
return ret;
}
i'm trying to make a video from an OpenGl context.
I'm Using glReadPixel, to be sure RGB buffer data is Ok i save it into a bmp file, wich i can read correctly.
My .h264 video is encoded but there are some artefact and i don't understand why.
I tried a lot of different parameters for the x264_param_t but anything better !
Bitmap saved (OpenGL real data) : Bitmap from OpenGl (1mo)
Raw h264 with error : Raw h264 video (1mo)
OpenGl ReadPixel :
int nSize = ClientHeight * ClientWidth * 3;
GLubyte *inBuff = new GLubyte[nSize];
glReadBuffer(GL_FRONT);
glReadPixels(0, 0, ldwidth, ldheight, GL_BGR, GL_UNSIGNED_BYTE, inBuff);
The params define :
x264_param_default(&mX264_param_t);
x264_param_default_preset(&mX264_param_t, "placebo", "film");
mX264_param_t.i_csp = X264_CSP_BGR;
mX264_param_t.i_threads = 6;
mX264_param_t.i_width = mWidth;
mX264_param_t.i_height = mHeight;
mX264_param_t.i_fps_num = mFps;
mX264_param_t.i_fps_den = 1;
// Intra refres:
mX264_param_t.i_keyint_max = mFps;
mX264_param_t.b_intra_refresh = 1;
//Rate control:
mX264_param_t.rc.i_rc_method = X264_RC_CRF;
mX264_param_t.rc.f_rf_constant = 25;
mX264_param_t.rc.f_rf_constant_max = 35;
int bps = 5000;
mX264_param_t.rc.i_bitrate = bps;
mX264_param_t.rc.i_vbv_max_bitrate = bps;
mX264_param_t.i_bframe = 2;
mX264_param_t.i_keyint_min = mFps / 4;
//For streaming:
mX264_param_t.b_repeat_headers = 1;
mX264_param_t.b_annexb = 1;
mX264_param_t.i_log_level = X264_LOG_DEBUG;
x264_param_apply_profile(&mX264_param_t, "baseline");
mpEncoder = x264_encoder_open(&mX264_param_t);
x264_encoder_parameters(mpEncoder, &mX264_param_t);
mpPictureOut = new x264_picture_t();
mpPictureIn = new x264_picture_t();
x264_picture_alloc(mpPictureIn, X264_CSP_BGR | X264_CSP_VFLIP, mWidth, mHeight);
Then the encoding loop :
mpPictureIn->img.i_csp = X264_CSP_BGR;
mpPictureIn->img.i_plane = 1;
mpPictureIn->img.i_stride[0] = 3 * mWidth;
mpPictureIn->img.plane[0] = rgbframe;
mpPictureIn->i_pts = mFrameCount;
mpPictureIn->i_type = X264_TYPE_AUTO;
mpPictureOut->i_pts = mFrameCount;
int i_nals;
x264_nal_t* nals;
int frame_size = x264_encoder_encode(mpEncoder, &nals, &i_nals, mpPictureIn, mpPictureOut);
if(frame_size > 0)
{
mpFileOut->write_frame(nals[0].p_payload, frame_size, mpPictureOut);
mFrameCount++;
}
The write frame :
int TVideoFileWriter::write_frame(uint8_t *p_nalu, int i_size, x264_picture_t *p_picture)
{
if(fwrite(p_nalu, i_size, 1, mFileHandle))
return i_size;
return -1;
}
You opened your output file in text mode (and not binary mode) and so all 0x0A bytes where replaced with 0x0D 0x0A bytes.
Here is your output with this replace reverted: out_fixed.h264
And it plays fine.