sws_scale YUV --> RGB distorted image - c++

I want to convert YUV420P image (received from H.264 stream) to RGB, while also resizing it, using sws_scale.
The size of the original image is 480 × 800. Just converting with same dimensions works fine.
But when I try to change the dimensions, I get a distorted image, with the following pattern:
changing to 481 × 800 will yield a distorted B&W image which looks like it's cut in the middle
482 × 800 will be even more distorted
483 × 800 is distorted but in color
484 × 800 is ok (scaled correctly).
Now this pattern follows - scaling will only work fine if the difference between divides by 4.
Here's a sample code of the way that I decode and convert the image. All methods show "success".
int srcX = 480;
int srcY = 800;
int dstX = 481; // or 482, 483 etc
int dstY = 800;
AVFrame* avFrameYUV = avcodec_alloc_frame();
avpicture_fill((AVPicture *)avFrameYUV, decoded_yuv_frame, PIX_FMT_YUV420P, srcX , srcY);
AVFrame *avFrameRGB = avcodec_alloc_frame();
AVPacket avPacket;
av_init_packet(&avPacket);
avPacket.size = read; // size of raw data
avPacket.data = raw_data; // raw data before decoding to YUV
int frame_decoded = 0;
int decoded_length = avcodec_decode_video2(g_avCodecContext, avFrameYUV, &frame_decoded, &avPacket);
int size = dstX * dstY * 3;
struct SwsContext *img_convert_ctx = sws_getContext(srcX, srcY, SOURCE_FORMAT, dstX, dstY, PIX_FMT_BGR24, SWS_BICUBIC, NULL, NULL, NULL);
avpicture_fill((AVPicture *)avFrameRGB, rgb_frame, PIX_FMT_RGB24, dstX, dstY);
sws_scale(img_convert_ctx, avFrameYUV->data, avFrameYUV->linesize, 0, srcY, avFrameRGB->data, avFrameRGB->linesize);
// draws the resulting frame with windows BitBlt
DrawBitmap(hdc, dstX, dstY, rgb_frame, size);
sws_freeContext(img_convert_ctx);

When you make a bitmap image, the width of image MUST be multiple of 4.
So you have to change width like 480, 484, 488, 492 ...
Here is method to change to multiple of 4
#define WIDTHBYTES(bits) (((bits) + 31) / 32 * 4)
void main()
{
BITMAPFILEHEADER bmFileHeader;
BITMAPINFOHEADER bmInfoHeader;
// load image
// ...
// when you use the method, put parameter like this.
int tempWidth = WIDTHBYTES(width * bmInfoHeader.biBitCount);
}
I hope you solve the problem.

Related

sws_scale, YUV to RGB conversion

I need convert YUV to RGB. I also need the RGB values to be in the limited range (16-235).
I try to use sws_scale function for this task.
My code you can see below. But after conversion I got the black pixel is (0, 0, 0) instead of (16, 16, 16).
Maybe there are some options to tell sws_scale function to calculate the limited range.
AVFrame* frameRGB = avFrameConvertPixelFormat(_decodedBuffer[i].pAVFrame, AV_PIX_FMT_RGB24);
AVFrame* Decoder::avFrameConvertPixelFormat(const AVFrame* src, AVPixelFormat dstFormat) {
int width = src->width;
int height = src->height;
AVFrame* dst = allocPicture(dstFormat, width, height);
SwsContext* conversion = sws_getContext(width,
height,
(AVPixelFormat)src->format,
width,
height,
dstFormat,
SWS_FAST_BILINEAR,
NULL,
NULL,
NULL);
sws_scale(conversion, src->data, src->linesize, 0, height, dst->data, dst->linesize);
sws_freeContext(conversion);
dst->format = dstFormat;
dst->width = src->width;
dst->height = src->height;
return dst;
}
Also I tried convert YUV pixel to RGB pixel manualy with formula and I got correct result. From YUV (16, 128, 128) I got (16, 16, 16) in RGB.
cmpR = y + 1.402 * (v - 128);
cmpG = y - 0.3441 * (u - 128) - 0.7141 * (v - 128);
cmpB = y + 1.772 * (u - 128);
You may the source format to "full scale" YUVJ.
As far as I know, sws_scale has no option for selecting Studio RGB as output format.
Changing the input format is the best solution I can think of.
The color conversion formula of "JPEG: YUV -> RGB" is the same as the formula in your post.
Examples for setting the source format:
If src->format is PIX_FMT_YUV420P, set the format to PIX_FMT_YUVJ420P.
If src->format is PIX_FMT_YUV422P, set the format to PIX_FMT_YUVJ422P.
If src->format is PIX_FMT_YUV444P, set the format to PIX_FMT_YUVJ444P.
If PIX_FMT_YUV440P, use PIX_FMT_YUVJ440P.
I know the solution is not covering all the possibilists, and there might be some output pixels exceeding the range of [16, 235], so it's not the most general solution...
yuv to rgb conversion using FFMPEG I see lot of information given already for this above. However for code completeness I am re-sharing the code with missing allocPicture() function, header & library to include, it works for me like a charm. Thanks to #Валентин Никин & #Rotem for most of the info & code.
Headers:
#include <libswscale/swscale.h>
Link FFMPEG Library:
libswscale
static AVFrame* allocPicture(enum AVPixelFormat pix_fmt, int width, int height)
{
// Allocate a frame
AVFrame* frame = av_frame_alloc();
if (frame == NULL)
{
fprintf(stderr, "avcodec_alloc_frame failed");
}
if (av_image_alloc(frame->data, frame->linesize, width, height, pix_fmt, 1) < 0)
{
fprintf(stderr, "av_image_alloc failed");
}
frame->width = width;
frame->height = height;
frame->format = pix_fmt;
return frame;
}
static AVFrame* avFrameConvertPixelFormat(const AVFrame* src, enum AVPixelFormat dstFormat)
{
int width = src->width;
int height = src->height;
AVFrame* dst = allocPicture(dstFormat, width, height);
struct SwsContext* conversion = sws_getContext(width,
height,
(enum AVPixelFormat)src->format,
width,
height,
dstFormat,
SWS_FAST_BILINEAR | SWS_FULL_CHR_H_INT | SWS_ACCURATE_RND,
NULL,
NULL,
NULL);
sws_scale(conversion, src->data, src->linesize, 0, height, dst->data, dst->linesize);
sws_freeContext(conversion);
dst->format = dstFormat;
dst->width = src->width;
dst->height = src->height;
return dst;
}
// convert yuv420p10le to rgb24 (or any other RGB formats)
AVFrame* frame = avFrameConvertPixelFormat(frame, AV_PIX_FMT_RGB24);

BGRA to YUV420 FFmpeg giving bad output

My purpose is a screen recorder. I am using the Windows DXGI API to receive screenshots and I'm encoding the screenshots into a video using libx264. Feeding BGRA images directly to libx264 is producing weird colors in the output video. So, to get correct colors, I am trying to convert the BGRA to YUV420p. To speed up encoding, I am also trying to downscale the BGRA image.
So I am getting an 1920x1080 BGRA image and I want to convert it to 1280x720 YUV420p. For this, I am using FFmpeg swscale library to do both the format conversion and downscaling.
The problem is that the output video is coming like 3 images in the same frame. Please see this video. https://imgur.com/a/EYimjrJ
I tried just BGRA to YUV conversion without any downscaling and it is working fine. But BGRA to YUV with downscaling is giving this problem.
What is the cause for this problem? How do I fix it?
Here is my code snippet:
uint8_t* Image;
x264_picture_t picIn, picOut;
x264_picture_alloc(&picIn, X264_CSP_I420, 1280, 720);
SwsContext* sws = sws_getContext(1920, 1080, AV_PIX_FMT_BGRA, 1280, 720, AV_PIX_FMT_YUV420P, SWS_BILINEAR, NULL, NULL, NULL);
while (true)
{
take_screenshot(&Image);
AVFrame BGRA;
BGRA.linesize[0] = 1280 * 4;
BGRA.data[0] = Image;
sws_scale(sws, BGRA.data, BGRA.linesize, 0, 1080, picIn.img.plane, picIn.img.i_stride);
nal_size = x264_encoder_encode(h, &nals, &nal_count, &picIn, &picOut);
save_to_flv(nals, nal_size, nal_count);
}
Here are my libx264 parameters:
x264_param_default_preset(&param, preset, 0);
param.i_csp = X264_CSP_I420;
param.i_width = 1280;
param.i_height = 720;
param.i_fps_num = 30;
param.i_fps_den = 1;
param.rc.i_bitrate = 2500;
param.i_bframe = 0;
param.b_repeat_headers = 0;
param.b_annexb = 1;
x264_param_apply_profile(&param, 0);
h = x264_encoder_open(&param);
Change BGRA.linesize[0] to be 1920*4. You see this 3 images pattern because
1240*3 == 1920*2

Create a divx-encoded avi from frames using opencv

This question is similar to this one and particularly this one but my desired output is different. I'm trying to capture the desktop to video using opencv. The preferred output is an avi file with divx encoding. I'm new to opencv and bitmap programming in general.
As a first step, to make sure the divx codec is present, I create a single frame (cv::Mat) of a solid color (yellow) and write that 100 times to the video file, as shown here:
int main(int argc, char* argv[])
{
cv::Mat frame(1200, 1920, CV_8UC3, cv::Scalar(0, 50000, 50000));
cv::VideoWriter* videoWriter = new cv::VideoWriter(
"C:/videos/desktop.avi",
CV_FOURCC('D','I','V','3'),
5, cv::Size(1920, 1200), true);
int frameCount = 0;
while (frameCount < 100)
{
videoWriter->write(frame);
::Sleep(100);
frameCount++;
}
delete videoWriter;
return 0;
}
This works perfectly - the video file is created and can be played on my Win 10 machine with VLC, Windows Media Player or the Films&TV app. It's 100 frames of solid yellow, but it shows the video is being created properly.
Next step: replace the dummy cv::Mat frame in the code above with a series of screenshots of the desktop. I get a handle to the desktop window using GetDesktopWindow(), and then use the function hwnd2mat (taken from this SO question - thanks!) to convert the bitmap obtained from the desktop handle to a cv::Mat that I can write to my video.
I copied the hwnd2mat function verbatim except I don't scale the image - the desktop bitmap is already 1920x1200, and also the cv::Mat I create is CV_8UC3 instead of CV_8UC4 (CV_8UC4 causes my app to crash).
Here's the code, including a reprint of hwnd2mat:
int main(int argc, char* argv[])
{
cv::VideoWriter* videoWriter = new cv::VideoWriter(
"C:/videos/desktop.avi",
CV_FOURCC('D','I','V','3'),
5, Size(1920, 1200), true);
int frameCount = 0;
while (frameCount < 100)
{
HWND hDsktopWindow = ::GetDesktopWindow();
cv::Mat frame = hwnd2mat(hDsktopWindow);
videoWriter->write(frame);
::Sleep(100);
frameCount++;
}
delete videoWriter;
return 0;
}
cv::Mat hwnd2mat(HWND hwnd)
{
HDC hwindowDC, hwindowCompatibleDC;
int height, width, srcheight, srcwidth;
HBITMAP hbwindow;
cv::Mat src;
BITMAPINFOHEADER bi;
hwindowDC = GetDC(hwnd);
hwindowCompatibleDC = CreateCompatibleDC(hwindowDC);
SetStretchBltMode(hwindowCompatibleDC, COLORONCOLOR);
RECT windowsize; // get the height and width of the screen
GetClientRect(hwnd, &windowsize);
srcheight = windowsize.bottom;
srcwidth = windowsize.right;
height = windowsize.bottom / 1; //change this to whatever size you want to resize to
width = windowsize.right / 1;
src.create(height, width, CV_8UC3);
// create a bitmap
hbwindow = CreateCompatibleBitmap(hwindowDC, width, height);
bi.biSize = sizeof(BITMAPINFOHEADER);
bi.biWidth = width;
bi.biHeight = -height; //this is the line that makes it draw upside down or not
bi.biPlanes = 1;
bi.biBitCount = 32;
bi.biCompression = BI_RGB;
bi.biSizeImage = 0;
bi.biXPelsPerMeter = 0;
bi.biYPelsPerMeter = 0;
bi.biClrUsed = 0;
bi.biClrImportant = 0;
// use the previously created device context with the bitmap
SelectObject(hwindowCompatibleDC, hbwindow);
// copy from the window device context to the bitmap device context
StretchBlt(hwindowCompatibleDC, 0, 0, width, height, hwindowDC, 0, 0,srcwidth, srcheight, SRCCOPY);
GetDIBits(hwindowCompatibleDC, hbwindow, 0, height, src.data, (BITMAPINFO*)&bi, DIB_RGB_COLORS);
// avoid memory leak
DeleteObject(hbwindow); DeleteDC(hwindowCompatibleDC); ReleaseDC(hwnd,hwindowDC);
return src;
}
The result of this is that the video file is created and can be played without errors, but it's just solid grey. It seems like the bitmap of the desktop is not getting copied correctly into the cv::Mat frame. I've tried a zillion combinations of the values in the BITMAPINFOHEADER, but nothing works and I don't know what I'm doing to be honest. I know opencv has conversion functions but again, I don't even really know what I'm trying to convert to/from.
Any help appreciated!
Figured out a way to make it work - I have no idea if this is the best way, so comments or alternative solutions are still welcome.
It seems like for GetDIBits to work, the cv::Mat has to be 4-channel, i.e. CV_8UC4, like the original code of hwnd2mat before I changed it. If it is not CV_8UC4, no data is copied (GetDIBits returns 0 scan lines copied) and that's why my avi was just gray. So the first change was to create the src cv::Mat as 4-channel:
src.create(height, width, CV_8UC4);
But for the divx-encoded avi file that I'm trying to create, the frames should be 3-channel (don't ask me why). I added a call to an opencv conversion function after calling GetDIBits(), as follows:
cv::Mat dst;
dst.create(height, width, CV_8UC3);
cv::cvtColor(src, dst, CV_RGBA2RGB);
And then I return dst from hwnd2mat instead of src. The call to cvtColor removes the alpha channel (the A in RGBA) and dst ends up with just the R,G,B channels.
You can get bitmap with no alpha channel from GetDIBits and write it straight to cv::VideoWriter. Just change biBitCount to 24. Leave Mat format to CV_8IC3. This worked for me.
src.create(height, width, CV_8UC3);
bi.biBitCount = 24; // this is where to change

Process AVFrame using opencv mat causing encoding error

I'm trying to decode a video file using ffmpeg, grab the AVFrame object, convert it to opencv mat object, do some processing then convert it back to AVFrame object and encode it back to a video file.
Well, the program can run, but it produces bad result.
I Keep getting errors like "top block unavailable for requested intra mode at 7 19", "error while decoding MB 7 19, bytestream 358", "concealing 294 DC, 294AC, 294 MV errors in P frame" etc.
And the result video got glithes all over it. like this,
I'm guessing it's because my AVFrame to Mat and Mat to AVFrame methods, and here they are
//unspecified function
temp_rgb_frame = avcodec_alloc_frame();
int numBytes = avpicture_get_size(PIX_FMT_RGB24, width, height);
uint8_t * frame2_buffer = (uint8_t *)av_malloc(numBytes * sizeof(uint8_t));
avpicture_fill((AVPicture*)temp_rgb_frame, frame2_buffer, PIX_FMT_RGB24, width, height);
void CoreProcessor::Mat2AVFrame(cv::Mat **input, AVFrame *output)
{
//create a AVPicture frame from the opencv Mat input image
avpicture_fill((AVPicture *)temp_rgb_frame,
(uint8_t *)(*input)->data,
AV_PIX_FMT_RGB24,
(*input)->cols,
(*input)->rows);
//convert the frame to the color space and pixel format specified in the sws context
sws_scale(
rgb_to_yuv_context,
temp_rgb_frame->data,
temp_rgb_frame->linesize,
0, height,
((AVPicture *)output)->data,
((AVPicture *)output)->linesize);
(*input)->release();
}
void CoreProcessor::AVFrame2Mat(AVFrame *pFrame, cv::Mat **mat)
{
sws_scale(
yuv_to_rgb_context,
((AVPicture*)pFrame)->data,
((AVPicture*)pFrame)->linesize,
0, height,
((AVPicture *)temp_rgb_frame)->data,
((AVPicture *)temp_rgb_frame)->linesize);
*mat = new cv::Mat(pFrame->height, pFrame->width, CV_8UC3, temp_rgb_frame->data[0]);
}
void CoreProcessor::process_frame(AVFrame *pFrame)
{
cv::Mat *mat = NULL;
AVFrame2Mat(pFrame, &mat);
Mat2AVFrame(&mat, pFrame);
}
Am I doing something wrong with the memory? Because if I remove the processing part, just decode and then encode the frame, the result is correct.
Well, it turns out I made a mistake at the initialization of temp_rgb_frame,if should be like this,
temp_rgb_frame = avcodec_alloc_frame();
int numBytes = avpicture_get_size(PIX_FMT_RGB24, width, height);
uint8_t * frame2_buffer = (uint8_t *)av_malloc(numBytes * sizeof(uint8_t));
avpicture_fill((AVPicture*)temp_rgb_frame, frame2_buffer, PIX_FMT_RGB24, width, height);

StretchDIBits failing only at specific image dimensions

I'm using StretchDIBits to print an image and it is failing when the image is at certain sizes for some unknown reason.
The image data is loaded into an unsigned int array from some some other image source in 24-bit BGR format. I've already verified that the image and buffer are perfectly fine since, like I said, it works at some sizes but not at all.
The current size I'm testing with is 638x1014. If I change the height to 1013 it works fine, but for some reason it just flat out fails if it's 1014.
Here's some code to show you how it's all being setup:
unsigned int * buffer = new unsigned int[width * height * 3];
// Fill buffer with image data...
BITMAPINFOHEADER bi = { 0 };
bi.biSize = sizeof(BITMAPINFOHEADER);
bi.biWidth = width;
bi.biHeight = height;
bi.biPlanes = 1;
bi.biBitCount = 24;
bi.biCompression = BI_RGB;
bi.biSizeImage = width * height * 3; // Specifying this value because if I don't it will crash trying to read outside of the buffer.
StartPage(hdcPrint);
SetMapMode(hdcPrint, MM_ISOTROPIC);
SetWindowExtEx(hdcPrint, width, height, NULL);
SetViewportExtEx(hdcPrint, width, height, NULL);
SetViewportOrgEx(hdcPrint, 0, 0, NULL);
StretchDIBits(hdcPrint, 0, 0, width, width, 0, 0, width, height, buffer, (BITMAPINFO *) &bi, DIB_RGB_COLORS, SRCCOPY);
StretchDIBits returns zero when it fails and the print result is a blank page.
I have a vague idea of what the problem is because, like it says in the comment, if I don't specify biSizeImage and leave it at zero then StretchDIBits will cause a crash because it tries to read past the end of the buffer. Even so, I have no idea how to diagnose exactly why it's doing that since it works at some sizes but not others.
Your width is the wrong number of bytes. Windows requires that each line be a multiple of 4 bytes; 638*3 is 1914, which is 2 bytes shy.