I have a distorted image in YUY2 data form, YUY2 comes under the family of YUV 4:2:2 (not 4:2:0).
And I have mapx and mapy (height-720, width-1280), which I obtained from
cv::fisheye::initUndistortRectifyMap(K, D, cv::Mat::eye(3, 3, CV_64F), new_K, Size, CV_32FC1, mapx, mapy);
How can I have undistorted YUY2?
My final goal is to have undistorted YUY2 (not BGR).
I thought to perform below steps:
cv::cvtColor(YUY, BGR, cv::COLOR_YUV2BGR_YUY2);
\\ then perform remapping
\\ and convert back to YUY
But there is no conversion from BGR2YUY_YUY2.
Is there is any smarter way?
You may convert from YUV 4:2:2 to YUV 4:4:4, undistorted the 4:4:4, and convert back to 4:2:2.
Illustration of the conversion stages:
YUV422 -> YUV444 -> remap(YUV444) -> YUV422
I could not find an OpenCV function for converting from YUV 4:2:2 to YUV 4:4:4.
Implementing the conversion by simple for loops is quite straight forward:
//Convert YUYV to YUV (y,u,v,y,u,v,y,u,v...)
//The conversion is performed by duplicating each U and V element twice (equivalent to resize with nearest neighbor interpolation).
//The input type is CV_8UC1 (considered to be Grayscale image).
//The output type is CV_8UC3 (considered to be colored image with 3 channels).
static cv::Mat convertYuyv422toYuv444(const cv::Mat yuyv)
{
int rows = yuyv.rows;
int src_cols = yuyv.cols;
size_t src_step = yuyv.step;
const unsigned char *I = (unsigned char*)yuyv.data; //Pointer to source image.
int dst_cols = src_cols / 2;
cv::Mat yuv = cv::Mat(rows, dst_cols, CV_8UC3);
size_t dst_step = yuv.step;
unsigned char *J = (unsigned char*)yuv.data; //Pointer to destination image.
for (int y = 0; y < rows; y++)
{
const unsigned char *I0 = I + y*src_step; //Points the beginning for the source row.
unsigned char *J0 = J + y*dst_step; //Points the beginning for the destination row.
int srcx = 0;
int dstx = 0;
//yuyv -> yuvyuv
//Convert 2 pixels per iteration
for (int x = 0; x < src_cols / 2; x += 2)
{
unsigned char y0 = I0[srcx];
unsigned char u0 = I0[srcx + 1];
unsigned char y1 = I0[srcx + 2];
unsigned char v0 = I0[srcx + 3];
J0[dstx] = y0;
J0[dstx + 1] = u0;
J0[dstx + 2] = v0;
J0[dstx + 3] = y1;
J0[dstx + 4] = u0; //Duplicate U
J0[dstx + 5] = v0; //Duplicate V
srcx += 4; //Source has 2 elements per pixel
dstx += 6; //Destination has 3 elements per pixel
}
}
return yuv;
}
The conversion simply duplicate every U and V element twice.
It's not the best way, but it's assumed to be good enough.
Duplicating U and V is equivalent to resize with Nearest Neighbor interpolation.
For converting YUV 4:4:4 back to YUV 4:2:2, you may use the code sample from (my) following post:
Convert YUV4:4:4 to YUV4:2:2 images.
There are existing optimized libraries that support all sorts of color format conversions.
libswscale for example, but I think it's an overkill for your needs...
Testing:
For testing, I used the input form your previous post (with my answer):
How to undistort I420 image data? Efficiently
Since I don't have a YUYV image, I used FFmpeg (command line) for creating one:
ffmpeg -i input_image.jpg -codec rawvideo -pix_fmt yuyv422 input_image_yuyv.yuv
I used MATLAB code for converting the raw input_image_yuyv.yuv to PNG.
The MATLAB implementation convert 4:2:2 to 4:4:4 in two ways and verify that duplicating U and V is is equivalent to resize with Nearest Neighbor interpolation.
The MATLAB code is also used for validating the correctness of the C++ implementation.
I = imread('input_image.jpg');
[rows, cols, ch] = size(I); % rows = 1280, cols = 720
% Read the YUYV to 2560x720 matrix from a binary file
f = fopen('input_image_yuyv.yuv', 'r');
YUYV = fread(f, [cols*2, rows], '*uint8')';
fclose(f);
% Write YUYV to PNG image - to be used as C++ input.
imwrite(YUYV, 'YUYV.png');
%figure;imshow(YUYV);title('YUYV');impixelinfo
Y = YUYV(:, 1:2:end); % 1280x720
U = YUYV(:, 2:4:end); % 640x720
V = YUYV(:, 4:4:end); % 640x720
% figure;imshow(Y);title('in Y');impixelinfo
% figure;imshow(U);title('in U');impixelinfo
% figure;imshow(V);title('in V');impixelinfo
% Convert U and V to 4:4:4 format using imresize with Nearest Neighbor interpolation method (used as reference).
refU2 = imresize(U, [rows, cols], 'nearest');
refV2 = imresize(V, [rows, cols], 'nearest');
% figure;imshow(U2);title('reference inU full');impixelinfo
% figure;imshow(V2);title('reference inV full');impixelinfo
% Resize x2 in the horizontal axis by simple duplication:
U2 = zeros(rows, cols, 'uint8');
U2(:, 1:2:end) = U;
U2(:, 2:2:end) = U;
V2 = zeros(rows, cols, 'uint8');
V2(:, 1:2:end) = V;
V2(:, 2:2:end) = V;
% Verify that the simple duplication is equivalent to resize with Nearest Neighbor interpolation:
% display(isequal(U2, refU2) && isequal(V2, refV2)) % Equal!!!
% Build YUV444 3840x720 matrix:
YUV444 = zeros(rows, cols*3, 'uint8');
YUV444(:, 1:3:end) = Y;
YUV444(:, 2:3:end) = U2;
YUV444(:, 3:3:end) = V2;
%figure;imshow(YUV444);title('YUV444');impixelinfo
% Write the YUV444 image to binary file (used as reference for C++ implementation)
f = fopen('image_yuv444.yuv', 'w');
fwrite(f, YUV444', 'uint8');
fclose(f);
imwrite(YUV444, 'matlabYUV444.png');
% Test output (after executing C++ code).
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
c_YUV444 = imread('yuv444.png');
display(isequal(YUV444, c_YUV444));
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
The following code is the main() of the C++ implementation:
int main()
{
cv::Mat yuyv = cv::imread("YUYV.png", cv::IMREAD_GRAYSCALE); //Read YUYV.png (created using MATLAB) as Grayscale
cv::Mat yuv = convertYuyv422toYuv444(yuyv); //Convet yuyv to yuv (y,u,v,y,u,v...)
//cv::imshow("yuyv", yuyv);
cv::imwrite("yuv444.png", yuv); //Store YUV image for testing.
//https://stackoverflow.com/questions/59876539/how-to-undistort-i420-image-data-efficiently
//remap the YUV 4:4:4
///////////////////////////////////////////////////////////////////////////////
int W = 1280, H = 720; //Assume resolution of Y plane is 1280x720
cv::Mat mapx;
cv::Mat mapy;
cv::Mat dst_yuv;
cv::Matx33d K = cv::Matx33d(541.2152931632737, 0.0, 661.7479652584254,
0.0, 541.0606969363056, 317.4524205037745,
0.0, 0.0, 1.0);
cv::Vec4d D = cv::Vec4d(-0.042166406281296365, -0.001223961942208027, -0.0017036710622692108, 0.00023929900459453295);
cv::Size newSize = cv::Size(3400, 1940);
cv::Matx33d new_K;
cv::fisheye::estimateNewCameraMatrixForUndistortRectify(K, D, cv::Size(W, H), cv::Mat::eye(3, 3, CV_64F), new_K, 1, newSize); // W,H are the distorted image size
cv::fisheye::initUndistortRectifyMap(K, D, cv::Mat::eye(3, 3, CV_64F), new_K, newSize, CV_16SC2, mapx, mapy);
cv::remap(yuv, dst_yuv, mapx, mapy, cv::INTER_LINEAR, cv::BORDER_CONSTANT, cv::Scalar(0, 128, 128));
///////////////////////////////////////////////////////////////////////////////
//Convert for BGR - just for dispaly
cv::Mat dst_bgr;
cv::cvtColor(dst_yuv, dst_bgr, cv::COLOR_YUV2BGR);
cv::imshow("yuv", yuv);
cv::imshow("dst_yuv", dst_yuv);
cv::imshow("dst_bgr", dst_bgr);
cv::waitKey(0);
cv::destroyAllWindows();
cv::imwrite("dst_bgr.png", dst_bgr); //Store BGR image for testing.
return 0;
}
Note:
Use remap with cv::BORDER_CONSTANT and cv::Scalar(0, 128, 128):
cv::remap(yuv, dst_yuv, mapx, mapy, cv::INTER_LINEAR, cv::BORDER_CONSTANT, cv::Scalar(0, 128, 128));
C++ Result (after converting to BGR):
I tried modification in mapx & mapy to make it suitable for YUV422. Result is very good in terms of computation time. Just one remapping is required in real time. But the quality is not the best.
Then I tried YUV422 -> YUV444 -> remap(YUV444) -> YUV422 through libswscale, but again YUV conversion was taking time.
Finally I developed cuda kernels for YUV conversions. I attached below.
// nvcc -c -o colorConversion.o colorConversion.cu `pkg-config --libs --cflags opencv4`
// /usr/bin/g++ -g -O3 /home/jai/vscode/opencvCUDA/cuda3.cpp -o /home/jai/vscode/opencvCUDA/cuda3 colorConversion.o `pkg-config --libs --cflags opencv4` `pkg-config --libs --cflags gstreamer-1.0` `pkg-config --libs --cflags cuda-11.3` `pkg-config --libs --cflags cudart-11.3`
#include "colorConversion.h"
__global__ void kernel_YUY422toYUY(cv::cuda::PtrStepSz<uchar2> YUV422, cv::cuda::PtrStepSz<uchar3> YUV)
{
int i = blockIdx.y; // row
int j = blockDim.x * blockIdx.x + threadIdx.x; // col
if (threadIdx.x & 1) { // odd 1,3,5
// YUV[i * step3 + 3 * j] = YUV422[i * step2 + 2 * j]; // Y0
// YUV[i * step3 + 3 * j + 1] = YUV422[i * step2 + 2 * j - 1]; // Y0
// YUV[i * step3 + 3 * j + 2] = YUV422[i * step2 + 2 * j + 1]; // Y0
YUV(i, j).x = YUV422(i, j).x;
YUV(i, j).y = YUV422(i, j - 1).y;
YUV(i, j).z = YUV422(i, j).y;
} else { // even 0,2,4,
// YUV[i * step3 + 3 * j] = YUV422[i * step2 + 2 * j]; // Y0
// YUV[i * step3 + 3 * j + 1] = YUV422[i * step2 + 2 * j + 1]; // U0
// YUV[i * step3 + 3 * j + 2] = YUV422[i * step2 + 2 * j + 3]; // V0
YUV(i, j).x = YUV422(i, j).x;
YUV(i, j).y = YUV422(i, j).y;
YUV(i, j).z = YUV422(i, j+1).y;
}
}
void YUY422toYUY(const cv::cuda::GpuMat &YUV422gpu, cv::cuda::GpuMat &YUVgpu)
{
kernel_YUY422toYUY<<<dim3(2, YUVgpu.rows), dim3(YUVgpu.cols / 2)>>>(YUV422gpu, YUVgpu);
//cudaSafeCall(cudaGetLastError());
}
__global__ void kernel_YUYtoYUY422(cv::cuda::PtrStepSz<uchar3> YUV, cv::cuda::PtrStepSz<uchar2> YUV422)
{
int i = blockIdx.x; // row
int j = threadIdx.x*2; // col
YUV422(i, j).x = YUV(i, j).x;
YUV422(i, j).y = (YUV(i, j).y + YUV(i, j+1).y)/2;
YUV422(i, j+1).x = YUV(i, j+1).x;
YUV422(i, j+1).y = (YUV(i, j).z + YUV(i, j+1).z)/2;
}
void YUYtoYUY422(const cv::cuda::GpuMat &YUVgpu, cv::cuda::GpuMat &YUV422gpu)
{
kernel_YUYtoYUY422<<<dim3(YUV422gpu.rows), dim3(YUV422gpu.cols / 2)>>>(YUVgpu, YUV422gpu);
//cudaSafeCall(cudaGetLastError());
}
And then I do remapping using CUDA again with following lines of code:
YUV422GPU.upload(YUV422); // YUV422 #channel = 2
YUV1.create(H, W, CV_8UC3);
YUV2.create(H, W, CV_8UC3);
YUY422toYUY(YUV422GPU, YUV1);
cv::cuda::remap(YUV1, YUV2, mapxGPU, mapyGPU, interpolationMethod); // YUV remap
YUYtoYUY422(YUV2, YUV422GPU);
YUV422GPU.download(dst); // dst is the final YUV422. 2 channel image
Related
I would like to know how to convert an image stored as a 1D std::vector<float> from CHW format (Channel, Height, Width) to HWC format (Height, Width, Channel) in C++. The format change is needed due to requirements of a neural network.
I used OpenCV to read and show the image as below:
cv::namedWindow("Screenshot", cv::WINDOW_AUTOSIZE );
cv::imshow("Screenshot", rgbImage);
Then I converted the cv::Mat rgbImage to a 1D std::vector<float> in format CHW:
size_t channels = 3;
std::vector<float> data(channels*ROS_IMAGE_HEIGHT*ROS_IMAGE_WIDTH);
for(size_t j=0; j<ROS_IMAGE_HEIGHT; j++){
for(size_t k=0; k<ROS_IMAGE_WIDTH; k++){
cv::Vec3b intensity = rgbImage.at<cv::Vec3b>(j, k);
for(size_t i=0; i<channels; i++){
data[i*ROS_IMAGE_HEIGHT*ROS_IMAGE_WIDTH + j*ROS_IMAGE_HEIGHT + k] = (float) intensity[i];
}
}
}
Now I want to convert the format of std::vector<float> data to HWC. How can I do this?
I found some description of the "CHW" and "HWC" formats here.
If the storage order is HWC, it means that
Each sample is stored as a column-major matrix (height, width) of float[numChannels] (r00, g00, b00, r10, g10, b10, r01, g01, b01, r11, g11, b11).
Thus a pixel (x, y, c) is found using
xStride = channels;
yStride = channels * width;
cStride = 1;
data[x*xStride + y*yStride + c*cStride]
If the storage order is CHW, it means that each channel is a different plane. A pixel (x, y, c) is found using
xStride = 1;
yStride = width;
cStride = width * height;
data[x*xStride + y*yStride + c*cStride]
Note that in the code in the question, data[i*ROS_IMAGE_HEIGHT*ROS_IMAGE_WIDTH + j*ROS_IMAGE_HEIGHT + k] is incorrect, j is the y-coordinate and should be multiplied by ROS_IMAGE_WIDTH.
The code in the question can be modified to yield a std::vector in the HWC format by replacing the line in the innermost loop by:
data[i + j*ROS_IMAGE_WIDTH*channels + k*channels] = (float) intensity[i];
I have a small program that outputs an rgb image. And I need it to be in .pfm format.
So, I have some data in the range [0, 255].
float * data;
data = new float[PixelWidth * PixelHeight * 3];
for (int i = 0; i < PixelWidth * PixelHeight * 3; i += 3) {
int idx = i / 3;
data[i] = img[idx].x;
data[i + 1] = img[idx].y;
data[i + 2] = img[idx].z;
}
(img[] here is Vec3[] of unsigned char)
Now I generate the image.
char sizes[256];
f = fopen("outputimage.pfm", "wb");
double scale = -1.0;
fprintf(f, "PF\n%d %d\n%lf\n", PixelWidth, PixelHeight, scale);
for (int i = 0; i < PixelWidth*PixelHeight*3; i++) {
float d = data[i];
fwrite((void *)&d, 1, 4, f);
}
fclose(f);
But somehow I get a grayscale image instead of RGB.
The data is fine. I tried to output it as .ppm and it works fine.
I guess the problem is with scaling, but I am not really sure how it should be done correctly.
To close the question.
I just had to convert all the values from [0-255] range to [0.0-1.0]. So, I divided each rgb value by 255.
In my iOS Project i am adding small PNG Images including alpha channel as overlay on a JPEG Picture. The result on my device in DEBUG mode is as expected, the tears are drawn correctly.
When i run the same code on Simulator or when i archive and export the App in RELEASE mode i get random artifacts in alpha channel.
The underlying cv::Mat all contain header infos and a valid data section. Even on green background the error is reproducible.
The behaviour seem to be totally random as from time to time no artifacts are drawn (image 3: right tear, image 4: left tear).
Ideas, anybody?
const char *cpath1 = [#"" cStringUsingEncoding:NSUTF8StringEncoding];//overlay image path , within #"" pass your image path which is in NSString
const char *cpath = [#"" cStringUsingEncoding:NSUTF8StringEncoding];//underlay imagepath
cv::Mat overlay = cv::imread(cpath1,-1);//-1 is for read .png images
cv::Mat underlay = cv::imread(cpath,-1);
//convert mat image in to RGB channel
cv::Mat overlayAlpha;
std::vector<Mat> channels1;
split(overlay, channels1);
channels1[3].copyTo(overlayAlpha);
cv::Mat underlayAlpha;
std::vector<Mat> channels2;
split(underlay, channels2);
channels2[3].copyTo(underlayAlpha);
overlayImage( &underlay, &overlay,cv::Point(10,10);
convert final image to RGB channel
cv::split(underlay,channels1);
std::swap(channels1[0],channels1[2]);// swap B and R channels.
cv::merge(channels1,underlay);//merge channels
MatToUIImage(background); //display your final image, it returns cv::Mat image
and overlay function is like below
overlay function referenced from : http://answers.opencv.org/question/73016/how-to-overlay-an-png-image-with-alpha-channel-to-another-png/
void overlayImage(Mat* src, Mat* overlay, const cv::Point& location){
for (int y = max(location.y, 0); y < src->rows; ++y)
{
int fY = y - location.y;
if (fY >= overlay->rows)
break;
for (int x = max(location.x, 0); x < src->cols; ++x)
{
int fX = x - location.x;
if (fX >= overlay->cols)
break;
double opacity = ((double)overlay->data[fY * overlay->step + fX * overlay->channels() + 3]) / 255;
for (int c = 0; opacity > 0 && c < src->channels(); ++c)
{
unsigned char overlayPx = overlay->data[fY * overlay->step + fX * overlay->channels() + c];
unsigned char srcPx = src->data[y * src->step + x * src->channels() + c];
src->data[y * src->step + src->channels() * x + c] = srcPx * (1. - opacity) + overlayPx * opacity;
}
}
}
}
I have a image with 4 channels that i need to overlay it over a bunch of pictures. Over the pictures with 3 channels, the overlaying works great, but over the pictures that have an alpha channel, the background of the picture changes to black.
Original picture: http://img.blog.csdn.net/20130610074054484
Overlayed picture: http://imgur.com/mlVAN0A
This is the code that does the overlaying:
void overlayImage(const cv::Mat &background, const cv::Mat &foreground,
cv::Mat &output, cv::Point2i location)
{
background.copyTo(output);
for(int y = std::max(location.y , 0); y < background.rows; ++y)
{
int fY = y - location.y;
if(fY >= foreground.rows)
break;
for(int x = std::max(location.x, 0); x < background.cols; ++x)
{
int fX = x - location.x;
if(fX >= foreground.cols)
break;
double opacity = ((double)foreground.data[fY * foreground.step + fX * foreground.channels() + 3]) / 255.;
for(int c = 0; opacity > 0 && c < output.channels(); ++c)
{
unsigned char foregroundPx = foreground.data[fY * foreground.step + fX * foreground.channels() + c];
unsigned char backgroundPx = background.data[y * background.step + x * background.channels() + c];
output.data[y*output.step + output.channels()*x + c] =
backgroundPx * (1.-opacity) + foregroundPx * opacity;
}
}
}
}
This is because you use 1-opacity for the background image. If the opacity of the forground image is 0, the opacity of your backgroundpixel will be 1 instead of 0 which it is before.
You have to calc the result opacity fpr both images which can be 0 for both too.
Claus
I have a problem to convert an image captured from a camera in YUV format to RGB format.
The function which is used to do it is the following :
int uwidth = 1920;
int uheight= 1080;
int i = 0,j = 0, r = 0, g = 0, b = 0;
typedef unsigned char BYTE;
IplImage* m_RGB = cvCreateImage(cvSize(uwidth, uheight), IPL_DEPTH_8U, 3);
unsigned char* pData = (unsigned char *) frameBytes;
for(i = 0, j=0; i < uwidth * uheight*3 ; i+=6, j+=4)
{
unsigned char u = pData[j];
unsigned char y = pData[j+1];
unsigned char v = pData[j+2];
b = 1.0*y + 8 + 1.402*(v-128);
g = 1.0*y - 0.34413*(u-128) - 0.71414*(v-128);
r = 1.0*y + 1.772*(u-128);
if(r>255) r =255;
if(g>255) g =255;
if(b>255) b =255;
if(r<0) r =0;
if(g<0) g =0;
if(b<0) b =0;
m_RGB->imageData[i] = (BYTE)(r*220/256);
m_RGB->imageData[i+1] = (BYTE)(g*220/256);
m_RGB->imageData[i+2] =(BYTE)(b*220/256);
}
cvNamedWindow("ck", CV_WINDOW_AUTOSIZE);
cvShowImage( "ck", m_RGB );
cvReleaseImage(&m_RGB);
The problem is that we have not one but two images in the window on my screen, and that we have good colors but not the good ratio.
Does anyone have an idea about those problems ?
Edit: Image output
Let's assume imageData is defined as
BYTE* imageData;
In this case this loop tells a lot:
for(i = 0, j=0; i < uwidth * uheight*3 ; i+=6, j+=4)
i+=6 means each time you set a pixel you will skip the next pixel (or what you expected to do, set 2 pixels at a time).
j+=4
unsigned char u = pData[j];
unsigned char y = pData[j+1];
unsigned char v = pData[j+2];
Means that the format of your camera is UYVY :
It describe two successive pixels P0 and P1
The chroma channel is the same for P0 and P1.U = U0 = U1 and V = V0 = V1
The lumina channel is different. the first is for P0, the second for P1.
You need to set 2 pixels by iterations :
m_RGB->imageData[i] = r1;
m_RGB->imageData[i+1] = g1;
m_RGB->imageData[i+2] =b1;
m_RGB->imageData[i+3] = r2;
m_RGB->imageData[i+4] = g2;
m_RGB->imageData[i+5] =b2;
The difference between r1 and r2 (and others) is that you use two different Y in the conversion formula.
If you're programming for Mac, have a look at the recent additions to the vImage conversion library in OSX 10.9 and even 10.10. The stuff in there is truly mind blowing. I was doing my own 24-bit to 32-bit conversions and had it honed to about 6ms/frame (1920x1080). Promptly blown out of the water by vImage more than 15X.