Converting a data buffer (YUV) to QImage (RGBA) - c++

I'm trying to convert an image buffer with YUV data to a QImage (RGBA8888 format).
Here is the conversion from the buffer to an IplImage :
void DeckLinkCaptureDelegate::convertFrameToOpenCV(void* frameBytes, IplImage * m_RGB){
if(!m_RGB) m_RGB = cvCreateImage(cvSize(width, height), IPL_DEPTH_8U, 3);
unsigned char* pData = (unsigned char *) frameBytes;
for(int i = 0, j=0; i < width * height * 3; i+=6, j+=4)
{
unsigned char u = pData[j];
unsigned char y = pData[j+1];
unsigned char v = pData[j+2];
//fprintf(stderr, "%d\n", v);
m_RGB->imageData[i+2] = 1.0*y + 8 + 1.402*(v-128); // r
m_RGB->imageData[i+1] = 1.0*y - 0.34413*(u-128) - 0.71414*(v-128); // g
m_RGB->imageData[i] = 1.0*y + 1.772*(u-128) + 0; // b
y = pData[j+3];
m_RGB->imageData[i+5] = 1.0*y + 8 + 1.402*(v-128); // r
m_RGB->imageData[i+4] = 1.0*y - 0.34413*(u-128) - 0.71414*(v-128); // g
m_RGB->imageData[i+3] = 1.0*y + 1.772*(u-128) + 0;
}
}
The goal would be to replace IplImage * m_RGB to a QImage object. The format of the QImage can be change later with a Qt function to convert QImage format. I've managed to get a QImage from the IplImage but I'm trying to bypass this step to completely remove the IplImage step.
The problem is, I'm having a hard time understanding the code above so I don't know how I could do it for a QImage.
Could you help me do it or at least understand the code above ?
Thanks.
EDIT : Here is what I have for the moment but it's not working.
void DeckLinkCaptureDelegate::convertFrameToOpenCV(void* frameBytes, QImage m_RGB){
//if(!m_RGB) m_RGB = QImage(width, height, QImage::Format_RGB888);
unsigned char* pData = (unsigned char *) frameBytes;
for(int i = 0, j=0; i < width * height * 3; i+=6, j+=4)
{
unsigned char u = pData[j];
unsigned char y = pData[j+1];
unsigned char v = pData[j+2];
//fprintf(stderr, "%d\n", v);
m_RGB.bits()[i+2] = 1.0*y + 8 + 1.402*(v-128); // r
m_RGB.bits()[i+1] = 1.0*y - 0.34413*(u-128) - 0.71414*(v-128); // g
m_RGB.bits()[i] = 1.0*y + 1.772*(u-128) + 0; // b
y = pData[j+3];
m_RGB.bits()[i+5] = 1.0*y + 8 + 1.402*(v-128); // r
m_RGB.bits()[i+4] = 1.0*y - 0.34413*(u-128) - 0.71414*(v-128); // g
m_RGB.bits()[i+3] = 1.0*y + 1.772*(u-128) + 0;
}
}

Create another unsigned char array(say pDataRGB), convert yuv(pData) into rgb(pDataRGB), and then create QImage like-
QImage *m_RGB = new QImage(pDataRGB, width, height, QImage::Format_RGB888);
If you want to use only Qimage then setPixel data as following:-
m_RGB->setPixel(i, j, qRgb(r, g, b));

Related

How to set a value at memory offset from a const void pointer?

I have a pointer returned from a function
rs2::video_frame frame = frames.get_color_frame();
const void* data = frame.get_data();
I know that this pointer is an array of RGB values (i.e. 3 chars) of size frame.get_data_size().
How can I modify certain pixel colors given that
int bpp = frame.get_bytes_per_pixel();
int width = frame.get_width();
int height = frame.get_height();
int offset = (y * width * bpp) + (x * bpp);
int r = offset;
int g = offset + 1;
int b = offset + 2;
// ?data[r] = newRed;
// ?data[g] = newGreen;
// ?data[b] = newBlue;
You would have it easier if you would have an object oriented approach:
struct Pixel {
short red;
short green;
short blue;
};
Let your frame work with an std::vector<Pixel> pixels; which is returned by reference. std::vector<Pixel>& get_data();
pixels[y * width + x].red = newRed;
pixels[y * width + x].green = newGreen;
pixels[y * width + x].blue = newBlue;
If you really have to work with void* then try this
char* data = static_cast<char*>(const_cast<void*>(dataframe.get_data()));
// Since you also const cast it becomes more and more dangerous. You really need know what you are doing.
size_t bpp = frame.get_bytes_per_pixel();
size_t width = frame.get_width();
size_t height = frame.get_height();
size_t offset = (y * width * bpp) + (x * bpp);
size_t r = offset;
size_t g = offset + 1;
size_t b = offset + 2;
*(data + r) = newRed;
*(data + g) = newGreen;
*(data + b) = newBlue;
For bulk updates you can use memset.
See https://godbolt.org/z/xvc1xs for details.

MediaFoundation + VP8 + Color Formats

My ultimate goal is to add video support to my Virtual Class application. My method is:
Capture frames using Media Foundation
Encode with VP8 using LibVPX
Transmit with UDP
Decode at the receipient site
Show frames in a Window
The first problem arises with my WebCam's supported color encodings. The webcam Media Types contain only MFVideoFormat_NV12. My first debugging attempt saves the receiving image to a bitmap so I can test it is correctly captured (error handling removed):
HRESULT CAP::StartRecord(HWND hh, CComPtr<IMFMediaSource> src)
{
MFCreateSourceReaderFromMediaSource(src, 0, &sr);
CComPtr<IMFMediaType> fmt;
sr->GetCurrentMediaType(MF_SOURCE_READER_FIRST_VIDEO_STREAM,&fmt);
LogMediaType(fmt); // Shows: MFVideoFormat_NV12
auto [wi, he] = WidthHeight(fmt);
for (;;)
{
DWORD streamIndex = 0, flags = 0;
LONGLONG llTimeStamp = 0;
CComPtr<IMFSample> pSample;
hr = sr->ReadSample(MF_SOURCE_READER_FIRST_VIDEO_STREAM,0,&streamIndex,&flags,&llTimeStamp,&pSample);
if (FAILED(hr))
break;
if (!pSample)
continue;
CComPtr<IMFMediaBuffer> bu;
pSample->ConvertToContiguousBuffer(&bu);
SaveSampleNV12(bu, wi, he);
}
...
}
SaveSampleNV12 uses code from here to convert NV12 to RGB, then:
void SaveSampleNV12(CComPtr<IMFMediaBuffer> mm, int width32, int height32)
{
DWORD le = 0;
mm->GetCurrentLength(&le);
BYTE *pDatad = NULL;
auto hr = mm->Lock(&pDatad, NULL, NULL);
vector<char> rgb(1000000);
NV12ToRGB((BYTE*)rgb.data(), pDatad, width32, height32);
mm->Unlock();
HANDLE file;
BITMAPFILEHEADER fileHeader;
BITMAPINFOHEADER fileInfo;
DWORD write = 0;
auto df = L"r:\\f.bmp";
file = CreateFile(df.c_str(), GENERIC_WRITE, 0, NULL, CREATE_ALWAYS, FILE_ATTRIBUTE_NORMAL, NULL); //Sets up the new bmp to be written to
int bits = 24;
fileHeader.bfType = 19778; //Sets our type to BM or bmp
fileHeader.bfSize = sizeof(fileHeader.bfOffBits) + sizeof(RGBTRIPLE); //Sets the size equal to the size of the header struct
fileHeader.bfReserved1 = 0; //sets the reserves to 0
fileHeader.bfReserved2 = 0;
fileHeader.bfOffBits = sizeof(BITMAPFILEHEADER) + sizeof(BITMAPINFOHEADER); //Sets offbits equal to the size of file and info header
fileInfo.biSize = sizeof(BITMAPINFOHEADER);
fileInfo.biWidth = width32;
fileInfo.biHeight = height32;
fileInfo.biPlanes = 1;
fileInfo.biBitCount = bits;
fileInfo.biCompression = BI_RGB;
fileInfo.biSizeImage = width32 * height32 * (bits / 8);
fileInfo.biXPelsPerMeter = 0;// 2400;
fileInfo.biYPelsPerMeter = 0;// 2400;
fileInfo.biClrImportant = 0;
fileInfo.biClrUsed = 0;
WriteFile(file, &fileHeader, sizeof(fileHeader), &write, NULL);
WriteFile(file, &fileInfo, sizeof(fileInfo), &write, NULL);
unsigned char* ptrIn = (unsigned char*)rgb.data();
int rgbs = width32 * height32 * (bits / 8);
vector<char> d2(rgbs);
unsigned char* ptrOut = (unsigned char*)d2.data();
for (int i = 0; i < (width32*height32) / 2; ++i)
{
int y0 = ptrIn[0];
int u0 = ptrIn[1];
int y1 = ptrIn[2];
int v0 = ptrIn[3];
ptrIn += 4;
int c = y0 - 16;
int d = u0 - 128;
int e = v0 - 128;
int bb = clip((298 * c + 516 * d + 128) >> 8); // blue
int gg = clip((298 * c - 100 * d - 208 * e + 128) >> 8); // green
int rr = clip((298 * c + 409 * e + 128) >> 8); // red
ptrOut[0] = bb;
ptrOut[1] = gg;
ptrOut[2] = rr;
c = y1 - 16;
ptrOut[3] = clip((298 * c + 516 * d + 128) >> 8); // blue
ptrOut[4] = clip((298 * c - 100 * d - 208 * e + 128) >> 8); // green
ptrOut[5] = clip((298 * c + 409 * e + 128) >> 8); // red
ptrOut += 6;
}
unsigned char* cc = (unsigned char*)d2.data();
WriteFile(file, cc, rgbs, &write, NULL);
CloseHandle(file);
}
This returns a weird full of Pink image. Something I'm doing wrong, but what?
Thanks a lot.
The solution is to use IMFTransform to tranform between various color spaces.

How can I flip a buffer vertically during an YUV to RGB conversion

EDIT : The question was not clear sorry, I updated and added details.
I have a buffer with image data (YUV format) that I convert to an RGB format. The problem is, I would like to flip the image vertically (Invert the Y-position).
What I'm able to do for the moment is convert my YUV data to RGB data in a buffer, then flip this buffer vertically.
Here is the working code for this :
unsigned char* DeckLinkCaptureDelegate::convertYUVtoRGB(void* frameBytes)
{
unsigned char *mycopy = new unsigned char[height*width*3];
unsigned char *flippedCopy = new unsigned char[height*width*3];
unsigned char* pData = (unsigned char *) frameBytes;
//Conversion from YUV to RGB
for(int i = 0, j=0; i < width * height * 3; i+=6, j+=4)
{
unsigned char v = pData[j];
unsigned char y = pData[j+1];
unsigned char u = pData[j+2];
mycopy[i+2] = 1.0*y + 8 + 1.402*(v-128); // r
mycopy[i+1] = 1.0*y - 0.34413*(u-128) - 0.71414*(v-128); // g
mycopy[i] = 1.0*y + 1.772*(u-128) + 0; // b
y = pData[j+3];
mycopy[i+5] = 1.0*y + 8 + 1.402*(v-128); // r
mycopy[i+4] = 1.0*y - 0.34413*(u-128) - 0.71414*(v-128); // g
mycopy[i+3] = 1.0*y + 1.772*(u-128) + 0;
}
//Vertical flip
for (int i = 0; i < width; ++i) {
for (int j = 0; j < height; ++j) {
for (int k = 0; k < 3; ++k) {
flippedCopy[(i + j * width) * 3 + k] = mycopy[(i + (height - 1 - j) * width) * 3 + k];
}
}
}
return flippedCopy;
}
What I would like to do to gain performance is flip the buffer DURING the conversion from YUV to RGB. I had no idea how to do it and Yusuf answer helped me, so here is what I have for the moment :
unsigned char* DeckLinkCaptureDelegate::convertYUVtoRGB(void* frameBytes)
{
unsigned char *mycopy = new unsigned char[height*width*3];
unsigned char* pData = (unsigned char *) frameBytes;
int k = height - 1;
for(int i = 0, j=0; i < width * height * 3; i+=6, j+=4)
{
unsigned char v = pData[j];
unsigned char y = pData[j+1];
unsigned char u = pData[j+2];
mycopy[(width*k*3) + i+2] = 1.0*y + 8 + 1.402*(v-128); // r
mycopy[(width*k*3) + i+1] = 1.0*y - 0.34413*(u-128) - 0.71414*(v-128); // g
mycopy[(width*k*3) + i] = 1.0*y + 1.772*(u-128) + 0; // b
y = pData[j+3];
mycopy[(width*k*3) + i+5] = 1.0*y + 8 + 1.402*(v-128); // r
mycopy[(width*k*3) + i+4] = 1.0*y - 0.34413*(u-128) - 0.71414*(v-128); // g
mycopy[(width*k*3) + i+3] = 1.0*y + 1.772*(u-128) + 0;
if (<i multiple of (width*3)-1>){
k = k - 2;
}
}
return mycopy;
}
If I'm correct, this should work, assuming the if condition is right. But I don't know how to express this if condition, since i is incremented by 6 each time, so I might "skip" the right moment to decrement k
I hope I'm clear enough. Thanks
I assume that width is even, otherwise "reduce in new line"-if will be complicated, then you must use 2 loops. I didn't tested, but should look like this;
unsigned char* DeckLinkCaptureDelegate::convertYUVtoRGB(void* frameBytes)
{
unsigned char *mycopy = new unsigned char[height*width*3];
unsigned char* pData = (unsigned char *) frameBytes;
unsigned int k = height - 1;
for(int i = 0, j=0; i < width * height * 3; i+=6, j+=4)
{
unsigned char v = pData[j];
unsigned char y = pData[j+1];
unsigned char u = pData[j+2];
mycopy[(width*k*3) + i+2] = 1.0*y + 8 + 1.402*(v-128); // r
mycopy[(width*k*3) + i+1] = 1.0*y - 0.34413*(u-128) - 0.71414*(v-128); // g
mycopy[(width*k*3) + i] = 1.0*y + 1.772*(u-128) + 0; // b
y = pData[j+3];
mycopy[(width*k*3) + i+5] = 1.0*y + 8 + 1.402*(v-128); // r
mycopy[(width*k*3) + i+4] = 1.0*y - 0.34413*(u-128) - 0.71414*(v-128); // g
mycopy[(width*k*3) + i+3] = 1.0*y + 1.772*(u-128) + 0;
if (mod(i, width*3) == 0) //reduce in new line (i am not sure how to reduce it, you should think about here)
k = k - 2;
}
return mycopy;
}
And tag your question as imageprocessing etc not only as c++

How to optimize YUV to RGB color conversion code

I have written a function to convert an image in YUV420P to RGB but it is taking 30 millisecond to convert an image (size: 1280 x 720) into RGB, but when I am using ffmpeg function ( as this) to convert YUV image into RGB its taking only 2 millisecond for the same image. What is the problem with my code ? How can I optimize the code that I have written ??
My code is given below
int step = origImage->widthStep;
uchar *data = (uchar *)origImage->imageData;
int size = origImage->width * origImage->height;
IplImage* img1 = cvCreateImage(cvGetSize(origImage), IPL_DEPTH_8U, 3);
for (int i = 0; i<origImage->height; i++)
{
for (int j=0; j<origImage->width; j++)
{
float Y = data[i*step + j];
float U = data[ (int)(size + (i/2)*(step/2) + j/2) ];
float V = data[ (int)(size*1.25 + (i/2)*(step/2) + j/2)];
float R = Y + 1.402 * (V - 128);
float G = Y - 0.344 * (U - 128) - 0.714 * (V - 128);
float B = Y + 1.772 * (U - 128);
if (R < 0){ R = 0; } if (G < 0){ G = 0; } if (B < 0){ B = 0; }
if (R > 255 ){ R = 255; } if (G > 255) { G = 255; } if (B > 255) { B = 255; }
cvSet2D(img1, i, j,cvScalar(B,G,R));
}
}
Here, try this(should reduce to 25 milliseconds):
int step = origImage->widthStep;
uchar *data = (uchar *)origImage->imageData;
int size = origImage->width * origImage->height;
IplImage* img1 = cvCreateImage(cvGetSize(origImage), IPL_DEPTH_8U, 3);
int stepDb2=step /2;
float sizeMb1d25=size*1.25 ;
int origImagePTheight=origImage->height;
int origImagePTwidth=origImage->width;
for (int i = 0; i<origImagePTheight; i++)
{
float idb2=i/2;
int iStep=i*step;
for (int j=0; j<origImagePTwidth; j++)
{
float variable=idb2*stepDb2 + j/2;
float Y = data[iStep + j];
float U = -128 + data[ (int)(size + variable) ];
float V = -128 + data[ (int)(sizeMb1d25 + variable)];
float R = Y + 1.402 * V ;
float G = Y - 0.344 * U - 0.714 * V;
float B = Y + 1.772 * U;
R= R * !(R<0);
G= G * !(G<0);
B= B * !(B<0);
R=R*(!(R>255)) + 255 * (R>255);
G=G*(!(G>255)) + 255 * (G>255);
B=B*(!(B>255)) + 255 * (B>255);
cvSet2D(img1, i, j,cvScalar(B,G,R));
}
}

weird image while trying to compress YUV image to jpeg using libjpeg

I am using Qt,OpenCV and libJpeg to compress a YUV422 image, but the output is not correct.
if I convert the yuv to rgb and then compress, I get a correct output, but what I understand that Jpeg is using YUV internally , that's why I want to remove redundancy.
Here is my code:
bool ipl2jpeg(IplImage *frame, unsigned char **outbuffer, long unsigned int *outlen) {
unsigned char *outdata = (uchar *) frame->imageData;
struct jpeg_compress_struct cinfo ;
struct jpeg_error_mgr jerr;
JSAMPROW row_ptr[1];
int row_stride;
*outbuffer = NULL;
*outlen = 0;
cinfo.err = jpeg_std_error(&jerr);
jpeg_create_compress(&cinfo);
jpeg_mem_dest(&cinfo, outbuffer, outlen);
cinfo.image_width = frame->width;
cinfo.image_height = frame->height;
cinfo.input_components = frame->nChannels;
cinfo.in_color_space = JCS_YCbCr;
jpeg_set_defaults(&cinfo);
jpeg_set_quality (&cinfo,100 , true);
jpeg_start_compress(&cinfo, TRUE);
row_stride = frame->width *2;// frame->nChannels;
while (cinfo.next_scanline < cinfo.image_height) {
/* jpeg_write_scanlines expects an array of pointers to scanlines.
* Here the array is only one element long, but you could pass
* more than one scanline at a time if that's more convenient.
*/
row_ptr[0] = &outdata[cinfo.next_scanline * row_stride];
(void) jpeg_write_scanlines(&cinfo, row_ptr, 1);
}
jpeg_finish_compress(&cinfo);
jpeg_destroy_compress(&cinfo);
return true;
}
What is wrong ??
Here is the YUV image
the image file provided (frame-3.raw) is 640x480 in YUYV format
the following code
JSAMPROW row_pointer[1];
row_pointer[0] = row_buf;
while (cinfo.next_scanline < cinfo.image_height) {
unsigned i, j;
unsigned offset = cinfo.next_scanline * cinfo.image_width * 2;
for (i = 0, j = 0; i < cinfo.image_width*2; i += 4, j += 6) {
row_buf[j + 0] = buf[offset + i + 0]; // Y
row_buf[j + 1] = buf[offset + i + 1]; // U
row_buf[j + 2] = buf[offset + i + 3]; // V
row_buf[j + 3] = buf[offset + i + 2]; // Y
row_buf[j + 4] = buf[offset + i + 1]; // U
row_buf[j + 5] = buf[offset + i + 3]; // V
}
jpeg_write_scanlines(&cinfo, row_pointer, 1);
}
to be put in between jpeg_start_compress() and jpeg_finish_compress() creates a JPEG file correctly without color space conversion (but with upsampling)
jason_s's code looks correct as well
I don't think libjpeg will accept subsampled image data as input. So, you'll have to un-subsample it first.
You could allocate a temporary row buffer, and convert each row, something like this:
for (i=0; i<frame->width; i++) {
tmpbuf[i*3+0] = outdata[cinfo.next_scanline * row_stride + i*2];
tmpbuf[i*3+1] = outdata[cinfo.next_scanline * row_stride + (i-i%2)*2+1];
tmpbuf[i*3+2] = outdata[cinfo.next_scanline * row_stride + (i-i%2)*2+3];
}
row_ptr[0] = tmpbuf;