Converting data from glReadPixels() to OpenCV::Mat - c++

I want to get every OpenGL frame from an animation with glReadPixels() and convert the data to OpenCV::Mat. I know that glReadPixels() gets the data by rows from the lower one to upper one, from left to right. On the other hand, OpenCV stores the data differently.
Does anybody know any library or any tutorial/example that helps me to convert data from glReadPixels to a OpenCV:Mat in C++?
SUMMARY
OpenGL frame -----------------------> CV::Mat
Data from left to right, Data from left to right,
bottom to top. top to bottom.

First we create an empty (or unititialized) cv::Mat for our data to be read into directly. This can be done once at startup, but on the other hand cv::Mat::create doesn't really cost much when the image already has matching size and type. The type depends on your needs, usually it's something like CV_8UC3 for a 24-bit color image.
cv::Mat img(height, width, CV_8UC3);
or
img.create(height, width, CV_8UC3);
Then you have to account for cv::Mat not neccessarily storing image rows contiguously. There might be a small padding value at the end of each row to make rows 4-byte aligned (or 8?). So you need to mess with the pixel storage modes:
//use fast 4-byte alignment (default anyway) if possible
glPixelStorei(GL_PACK_ALIGNMENT, (img.step & 3) ? 1 : 4);
//set length of one complete row in destination data (doesn't need to equal img.cols)
glPixelStorei(GL_PACK_ROW_LENGTH, img.step/img.elemSize());
Next, the type of the matrix influences the format and type parameters of glReadPixels. If you want color images you have to keep in mind that OpenCV usually stores color values in BGR order, so you need to use GL_BGR(A) (which were added with OpenGL 1.2) instead of GL_RGB(A). For one component images use either GL_LUMINANCE (which sums the individual color components) or GL_RED, GL_GREEN, ... (to get an individual component). So for our CV_8UC3 image the final call to read it directly into the cv::Mat would be:
glReadPixels(0, 0, img.cols, img.rows, GL_BGR, GL_UNSIGNED_BYTE, img.data);
Finally, OpenCV stores images from top to bottom. So you may need to either flip them after getting them or render them flipped in OpenGL in the first place (this can be done by adjusting the projection matrix, but keep an eye on triangle orientation in this case). To flip a cv::Mat vertically, you can use cv::flip:
cv::flip(img, flipped, 0);
So to keep in mind OpenCV:
stores images from top to bottom, left to right
stores color images in BGR order
might not store image rows tightly packed

unsigned char* getPixelData( int x1, int y1, int x2, int y2 )
{
int y_low, y_hi;
int x_low, x_hi;
if ( y1 < y2 )
{
y_low = y1;
y_hi = y2;
}
else
{
y_low = y2;
y_hi = y1;
}
if ( x1 < x2 )
{
x_low = x1;
x_hi = x2;
}
else
{
x_low = x2;
x_hi = x1;
}
while ( glGetError() != GL_NO_ERROR )
{
;
}
glReadBuffer( GL_BACK_LEFT );
glDisable( GL_TEXTURE_2D );
glPixelStorei( GL_PACK_ALIGNMENT, 1 );
unsigned char *data = new unsigned char[ ( x_hi - x_low + 1 ) * ( y_hi - y_low + 1 ) * 3 ];
glReadPixels( x_low, y_low, x_hi-x_low+1, y_hi-y_low+1, GL_RGB, GL_UNSIGNED_BYTE, data );
if ( glGetError() != GL_NO_ERROR )
{
delete[] data;
return 0;
}
else
{
return data;
}
}
use:
CvSize size = cvSize( 320, 240 );
unsigned char *pixel_buf = getPixelData( 0, 0, size.width - 1, size.height - 1 );
if ( pixel_buf == 0 )
return 0;
IplImage *result = cvCreateImage( size, IPL_DEPTH_8U, 3 );
memcpy( result->imageData, pixel_buf, size.width * size.height * 3 );
delete[] pixel_buf;

Related

How to create a QImage from GeoTIFF data (or just interpret it correctly)

I need to create a QImage or something that can be drawn onto a screen from a geotiff image. Unfortunately QT's built-in TIFF support chokes on the geotiff structures ... so to achieve this I have used the following code (which is more or less a copy paste from the gdal "tutorial" page (https://gdal.org/gdal_tutorial.html) except the image creation part ):
GDALRasterBand *poBand;
int nBlockXSize, nBlockYSize;
int bGotMin, bGotMax;
double adfMinMax[2];
poBand = poDataset->GetRasterBand( 1 );
poBand->GetBlockSize( &nBlockXSize, &nBlockYSize );
adfMinMax[0] = poBand->GetMinimum( &bGotMin );
adfMinMax[1] = poBand->GetMaximum( &bGotMax );
if( ! (bGotMin && bGotMax) )
GDALComputeRasterMinMax((GDALRasterBandH)poBand, TRUE, adfMinMax);
float *pafScanline;
int nXSize = poBand->GetXSize();
int nYSize = poBand->GetYSize();
pafScanline = (float *) CPLMalloc(sizeof(float)*nXSize * nYSize);
poBand->RasterIO( GF_Read, 0, 0, nXSize, nYSize,
pafScanline, nXSize, nYSize, GDT_Float32, 0, 0 );
QImage* image = new QImage((unsigned char*)pafScanline,
nXSize, nYSize,
QImage::Format_RGB32);
image->save("blaa.jpg");
Now, the image I try to load is on the left side and the one that gets displayed (and saved by Qt) is on the right side.
Question: how to create a properly coloured image from the tiff data given that I get in floats, and I have no idea how to create a QImage data from a bunch of floats.
Your input GeoTIFF may not have a single floating point band, but rather 3 (or 4) 8 bit bands.
Bands in GeoTIFF are basically the image channels. Unlike other image formats, these channels can also have floating point values.
You can have a look at the GDAL documentation here to know more about the allowed formats.
So it is likely (although I can't be 100% sure without looking at it) that your file is just an RGBA GeoTIFF, and hence, has 4 UNIT8 bands.
Therefore, your call to RasterIO is completely wrong. You should iterate over the 4 bands and copy the with RasterIO to the QImage memory buffer, respecting the bands order.
Something like:
int nBands = poDataset->GetRasterCount();
for(int b=0; b < nBands; b++)
{
GDALRasterBand *band = poDataset->GetRasterBand(b);
if(band != nullptr)
{
CPLErr error = band->RasterIO(GF_Write, 0, 0, image.width(), image.height(), image.bits() + b, image.width(), image.height(), GDT_Byte, nBands, 0);
if(error != CE_None)
{
// REPORT ERROR
}
}
}
Please note that the code above is missing all the required checks (ensuring band type is Byte, etc), and depending on your file the band order may vary (BGRA, RGBA, ecc).

Why is the OpenCL Kernel not using normal x y cooordinates with Image2D?

TLDR;
For anyone arriving here whilst trying to figure out how to do gaussian blur or grayscale with OpenCL, the final working code is here. Note that in that repo I'm actually running the whole thing inside Docker with GPU access using Nvidia's Docker wrapper. You can look inside the 'Dockerfile' for the steps that need to be taken to get the code running, or just run it using Nvidia-Docker if you have that setup and are running on an Nvidia GPU.
Original Question:
Using the following kernel in an OpenCL image filter application I get the expected result, that is, a returned grayscale version of the input image:
const sampler_t sampler = CLK_NORMALIZED_COORDS_FALSE |
CLK_ADDRESS_CLAMP_TO_EDGE |
CLK_FILTER_NEAREST;
__kernel void process(__read_only image2d_t src,
__write_only image2d_t dst)
{
int x = get_global_id(0);
int y = get_global_id(1);
float4 color;
color = read_imagef(src, sampler, (int2)(x, y));
float gray = (color.x + color.y + color.z) / 3;
write_imagef(dst, (int2)(x,y), (float4)(gray, gray, gray, 0));
}
So far, so good. I then tried to create a kernel that would just copy across the top and left border of the image:
const sampler_t sampler = CLK_NORMALIZED_COORDS_FALSE |
CLK_ADDRESS_CLAMP_TO_EDGE |
CLK_FILTER_NEAREST;
__kernel void process(__read_only image2d_t src,
__write_only image2d_t dst)
{
int x = get_global_id(0);
int y = get_global_id(1);
float4 color;
if (x < 10 || y < 10)
{
color = read_imagef(src, sampler, (int2)(x, y));
write_imagef(dst, (int2)(x,y), (float4)(color.x, color.y, color.z, 0));
}
else
{
write_imagef(dst, (int2)(x,y), (float4)(0,0,0,0));
}
}
The returned image is not what I expected:
I'm loading the input image this way:
// Load an image using the OpenCV library and create an OpenCL
// image out of it
cl::Image2D LoadImage(cl::Context context, char *fileName, int &width, int &height)
{
cv::Mat image = cv::imread(fileName, CV_LOAD_IMAGE_COLOR);
cv::Mat imageRGBA;
width = image.rows;
height = image.cols;
cv::cvtColor(image, imageRGBA, CV_RGB2RGBA);
char *buffer = reinterpret_cast<char *>(imageRGBA.data);
cl::Image2D clImage(context,
CL_MEM_READ_ONLY | CL_MEM_COPY_HOST_PTR,
cl::ImageFormat(CL_RGBA, CL_UNORM_INT8),
width,
height,
0,
buffer);
return clImage;
}
The output image:
cl::Image2D imageOutput(context,
CL_MEM_WRITE_ONLY,
cl::ImageFormat(CL_RGBA, CL_UNORM_INT8),
width,
height,
0,
NULL);
The Kernel:
cl::Program program(context, util::loadProgram("border.cl"), true);
cl::make_kernel<cl::Image2D, cl::Image2D> filter(program, "process");
cl::NDRange global(width, height);
filter(cl::EnqueueArgs(queue, global), clImageInput, imageOutput);
Then reading the image back:
cl::size_t<3> origin;
origin[0] = 0; origin[1] = 0, origin[2] = 0;
cl::size_t<3> region;
region[0] = width; region[1] = height; region[2] = 1;
float* oup = new float[width * height];
queue.enqueueReadImage(imageOutput, CL_TRUE, origin, region, 0, 0, oup);
cv::imwrite(filename_out, cv::Mat(width, height, CV_8UC4, oup));
Why is the image being processed the way it is? Only selecting pixels with a y coordinate less than 10 seems to work, but selecting pixels with an x coordinate less than 10 seems to stagger across the image.
if I write a test image using the following line in the kernel:
write_imagef(dst, (int2)(x,y), (float4)((float)x / 512.0f, 0, 0, 0));
I get the following image:
The first strange thing is that the blue channel is being set, not the red. I have no idea why as I am alway loading and saving the image in RGBA order. Secondly, the banding is very unusual, I'm not sure how to interpret this.
If I use the following line in the kernel:
write_imagef(dst, (int2)(x,y), (float4)(0, (float)y / 512.0f, 0, 0));
I get the following image:
This looks the way I would expect.
I can provide more code if necessary but using the grayscale kernel in the exact same harness works perfectly. As does another kernel not listed here which simply copies all the pixels across.
I'm running the code on and Nvidia Geforce 980M with OpenCL 1.2
I'm not seeing anything obvious yet. One strange thing: your image is CL_RGBA, CL_UNORM_INT8 but you're reading it out into an array of floats? How are you displaying it from that? Second, I'm not famliar with your kernel launch technique; what is filter and is it launching with dimension of 2? Regarding the issue you're seeing, I'd suggest using process of elimination to figure out where the problem lies. For example, (1) if you remove the conditional and copy all pixels, do you get the whole image? (2) Instead of writing black where the conditional is false, what if you write a Red channel gradient based on X position and a Green channel gradient based on Y position. Do you get a double gradient? Based on results, continue to divide the problem until you find the cause. It looks a lot like a row pitch issue, perhaps in the display function?
Ok, so the issue was the way I was reading height and width was backwards, i.e.
width = image.rows;
height = image.cols;
Should have been
height = image.rows;
width = image.cols;
With this corrected, the rest of the code can stay the same, except the last line where I save the image to disk, here the values need to be swapped again, i.e.
cv::imwrite(filename_out, cv::Mat(width, height, CV_8UC4, oup));
Needs to change to:
cv::imwrite(filename_out, cv::Mat(height, width, CV_8UC4, oup));
I think this ultimately comes down to the matrix approach to an image where the first coordinate is actually the row number, which is the height and the second coordinate is the column number, which is the width.
The diagnostics #Dithermaster mentioned really helped, as did printing out the assumed width and height, which was ultimately incorrect.
It's interesting that by having both of those errors in the code a pixel for pixel copy worked fine, but once you start to perform actions based on the x,y coordinates you get some really funky results.

OpenCV cv::cvtColor drops alpha channel, How to keep alpha data?

I have realized several things:
that OpenCV does not handle alpha channels in cvtColor(RGBAMat, HSVXMat, RGB2HSV, 4) even though it silently accepts and ignores the 4 channel parameter and the 4 channel output matrix.
that OpenCV does not convert datatypes in cvtColor though it happily accepts an output matix of different datatype from the input without warning.
that OpenCV's CV_AUTOSTEP as the last parameter to Mat(int h, int w, enum val DATATYPE, char* data, size_t step) does not work when creating Mats from SDL_Surfaces. It causes segfaults that are remedied by using surface->w * sizeof(whatever datatype the surface is).
Because of these realizations, I have accepted #ypnos answer since it is clearly on the right track. Be warned that none of the code on this page works as is.
void surface_change_sat(SDL_Surface* src_surf, SDL_Surface* dst_surf, float sat_diff) {
using namespace cv;
SDL_LockSurface(src_surf);
Mat src_mat = Mat(src_surf->h, src_surf->w, CV_8UC4); // Make mat from surf
memcpy(src_mat.data, src_surf->pixels, src_surf->h * src_surf->w * sizeof(Uint32));
SDL_UnlockSurface(src_surf);
Mat src_rgbI = Mat(src_mat.rows, src_mat.cols, CV_8UC3); // split rgba into rgb and a
Mat aI_mat = Mat(src_mat.rows, src_mat.cols, CV_8UC1); // since hsv has no a
Mat to_ar[] {src_rgbI, aI_mat};
int from_to[] = { 0,0, 1,1, 2,2, 3,3 }; // r=0, ... a=3
mixChannels(&src_mat, 1, to_ar, 2, from_to, 4);
Mat rgbF_mat = Mat(src_mat.rows, src_mat.cols, CV_32FC3); // Make ints into floats
src_rgbI.convertTo(rgbF_mat, CV_32F);
typedef Vec<float, 3> flt_vec_t; // The type of pixel in hsv
Mat hsv_mat = Mat(src_mat.rows, src_mat.cols, CV_32FC3);
cvtColor(rgbF_mat, hsv_mat, CV_RGB2HSV, 3); // convert to HSV
flt_vec_t pix_vec;
for (MatIterator_<flt_vec_t> mat_it = hsv_mat.begin<flt_vec_t>();
mat_it != hsv_mat.end<flt_vec_t>();
mat_it++) {
pix_vec = *mat_it;
Matx<float, 3, 1> pix_matx = (Matx<float, 3, 1>)pix_vec;
CV_Assert(pix_matx.val[1] <= 1.0f && pix_matx.val[1] >= 0.0f);
pix_matx.val[1] += sat_diff;
if (pix_matx.val[1] > 1.0f) { pix_matx.val[1] = 1.0f; }
if (pix_matx.val[1] < 0.0f) { pix_matx.val[1] = 0.0f; }
}
cvtColor(hsv_mat, rgbF_mat, CV_HSV2RGB, 3); // convert back to RGB
Mat dst_mat = Mat(dst_surf->h, dst_surf->w, CV_8UC4);
Mat dst_rgbI = Mat(dst_mat.rows, dst_mat.cols, CV_8UC3);
rgbF_mat.convertTo(dst_rgbI, CV_8U); // float back to int
Mat from_ar[] {dst_rgbI, aI_mat};
int to_from[] = { 0,0, 1,1, 2,2, 0,3 }; // r=0, ... a=3
mixChannels(from_ar, 2, &dst_mat, 1, to_from, 4); // add alpha for RGBA
SDL_LockSurface(dst_surf);
memcpy(dst_surf->pixels, (void*)dst_mat.data, dst_surf->h * dst_surf->w * sizeof(Uint32));
SDL_UnlockSurface(dst_surf);
}
//-------------------Old Question -------------------------
I was getting the most mysterious memory errors until I added the CV_Asserts to the following function and realized that OpenCV was silently destroying the alpha channel in cv::Mat hsv_mat. I'm confused as to how to resolve this problem:One note, several other answers to related questions suggested thing like using shell scripts or cli tools like imagemagick. These are not helpful in this particular case. I need a standalone function with dependencies limited to opencv and the standard library.
Note: Following a suggestion in the comments, I explicitly split the alpha channel and converted from Uint8 to Float32. Now the assertions function, however, there is still a problem with invalid memory access causing a segfault.
Well, basically cvtColor only works with a fixed set of matrix types, including the format and number of channels. So unmixing and remixing the channels is unavoidable.
The next thing I would do is use the templated data types. They give you a better understanding of what is held within a matrix, however sometimes OpenCV manages to violate that, too (at least in older versions).
This is how I would do it:
SDL_LockSurface(surf);
cv::Mat4b rgba(surf->h, surf->w, surf->pixels);
std::vector<cv::Mat1b> channels_rgba;
cv::split(rgba, channels_rgba);
// create matrix with first three channels only
std::vector<cv::Mat1b> channels_rgb(channels_rgba.begin(),
channels_rgba.begin()+3);
cv::Mat3b rgb;
cv::merge(channels_rgb, rgb);
// create floating point representation
cv::Mat3f rgbhsv;
rgbhsv = rgb; // implicit conversion
// convert to HSV
cv::cvtColor(rgbhsv, rgbhsv, CV_RGB2HSV);
for (cv::Mat3f::iterator mat_it = rgbhsv.begin();
mat_it != rgbhsv.end(); mat_it++) {
cv::Vec3f &pix_vec = *mat_it;
pix_vec[0] += hue_diff;
if (pix_vec[0] >= 360.0f || pix_vec[0] < 0.0f) {
pix_vec[0] = (180.0f / M_PI) * (std::asin(std::sin((pix_vec[0] * M_PI / 180.0f))));
}
}
// convert back to RGB
cv::cvtColor(rgbhsv, rgbhsv, CV_HSV2RGB);
// back to unsigned char channels
rgb = rgbhsv; // implicit conversion
cv::split(rgb, channels_rgb);
// replace first three channels only
for (size_t i = 0; i < channels_rgb.size(); ++i)
channels_rgba[i] = channels_rgb[i];
cv::merge(channels_rgba, rgba);
SDL_UnlockSurface(surf);
Please note:
I did not test this code!
It might be necessary to adjust data ranges (going back to convertTo() instead of assignment operator!
My last operation may not overwrite the surface data (you can go back to mixChannels which is known to not allocate any data ever)

Copy / blend images of different sizes using opencv

I am trying to blend two images. It is easy if they have the same size, but if one of the images is smaller or larger cv::addWeighted fails.
Image A (expected to be larger)
Image B (expected to be smaller)
I tried to create a ROI - tried to create a third image of the size of A and copy B inside - I can't seem to get it right. Please help.
double alpha = 0.7; // something
int min_x = ( A.cols - B.cols)/2 );
int min_y = ( A.rows - B.rows)/2 );
int width, height;
if(min_x < 0) {
min_x = 0; width = (*input_images).at(0).cols - 1;
}
else width = (*input_images).at(1).cols - 1;
if(min_y < 0) {
min_y = 0; height = (*input_images).at(0).rows - 1;
}
else height = (*input_images).at(1).rows - 1;
cv::Rect roi = cv::Rect(min_x, min_y, width, height);
cv::Mat larger_image(A);
// not sure how to copy B into roi, or even if it is necessary... and keep the images the same size
cv::addWeighted( larger_image, alpha, A, 1-alpha, 0.0, out_image, A.depth());
Even something like cvSetImageROI - may work but I can't find the c++ equivalent - may help - but I don't know how to use it to still keep the image content, only place another image inside ROI...
// min_x, min_y should be valid in A and [width height] = size(B)
cv::Rect roi = cv::Rect(min_x, min_y, B.cols, B.rows);
// "out_image" is the output ; i.e. A with a part of it blended with B
cv::Mat out_image = A.clone();
// Set the ROIs for the selected sections of A and out_image (the same at the moment)
cv::Mat A_roi= A(roi);
cv::Mat out_image_roi = out_image(roi);
// Blend the ROI of A with B into the ROI of out_image
cv::addWeighted(A_roi,alpha,B,1-alpha,0.0,out_image_roi);
Note that if you want to blend B directly into A, you just need roi.
cv::addWeighted(A(roi),alpha,B,1-alpha,0.0,A(roi));
You can easily blend two images using addWeighted()function
addWeighted(src1, alpha, src2, beta, 0.0, dst);
Declare two images
src1 = imread("c://test//blend1.jpg");
src2 = imread("c://test//blend2.jpg");
Declare the value of alpha and beta and then call the function. You are done. You can find the details in the link: Blending of Images using Opencv

How to create video using avcodec from jpeg images of type OpenCV::Mat?

I have colored jpeg images of OpenCV::Mat type and I create from them video using avcodec. The video that I get is upside-down, black & white and each row of each frame is shifted and I got diagonal line. What could be the reason for such output?
Follow this link to watch the video I get using avcodec.
I'm using acpicture_fill function to create avFrame from cv::Mat frame!
P.S.
Each cv::Mat cvFrame has width=810, height=610, step=2432
I noticed that avFrame (that is filled by acpicture_fill) has linesize[0]=2430
I tried manually setting avFrame->linesizep0]=2432 and not 2430 but it still didn't helped.
======== CODE =========================================================
AVCodec *encoder = avcodec_find_encoder(AV_CODEC_ID_H264);
AVStream *outStream = avformat_new_stream(outContainer, encoder);
avcodec_get_context_defaults3(outStream->codec, encoder);
outStream->codec->pix_fmt = AV_PIX_FMT_YUV420P;
outStream->codec->width = 810;
outStream->codec->height = 610;
//...
SwsContext *swsCtx = sws_getContext(outStream->codec->width, outStream->codec->height, PIX_FMT_RGB24,
outStream->codec->width, outStream->codec->height, outStream->codec->pix_fmt, SWS_BICUBIC, NULL, NULL, NULL);
for (uint i=0; i < frameNums; i++)
{
// get frame at location I using OpenCV
cv::Mat cvFrame;
myReader.getFrame(cvFrame, i);
cv::Size frameSize = cvFrame.size();
//Each cv::Mat cvFrame has width=810, height=610, step=2432
1. // create AVPicture from cv::Mat frame
2. avpicture_fill((AVPicture*)avFrame, cvFrame.data, PIX_FMT_RGB24, outStream->codec->width, outStream->codec->height);
3avFrame->width = frameSize.width;
4. avFrame->height = frameSize.height;
// rescale to outStream format
sws_scale(swsCtx, avFrame->data, avFrame->linesize, 0, outStream->codec->height, avFrameRescaledFrame->data, avFrameRescaledFrame ->linesize);
encoderRescaledFrame->pts=i;
avFrameRescaledFrame->width = frameSize.width;
avFrameRescaledFrame->height = frameSize.height;
av_init_packet(&avEncodedPacket);
avEncodedPacket.data = NULL;
avEncodedPacket.size = 0;
// encode rescaled frame
if(avcodec_encode_video2(outStream->codec, &avEncodedPacket, avFrameRescaledFrame, &got_frame) < 0) exit(1);
if(got_frame)
{
if (avEncodedPacket.pts != AV_NOPTS_VALUE)
avEncodedPacket.pts = av_rescale_q(avEncodedPacket.pts, outStream->codec->time_base, outStream->time_base);
if (avEncodedPacket.dts != AV_NOPTS_VALUE)
avEncodedPacket.dts = av_rescale_q(avEncodedPacket.dts, outStream->codec->time_base, outStream->time_base);
// outContainer is "mp4"
av_write_frame(outContainer, & avEncodedPacket);
av_free_packet(&encodedPacket);
}
}
UPDATED
As #Alex suggested I changed the lines 1-4 with the code below
int width = frameSize.width, height = frameSize.height;
avpicture_alloc((AVPicture*)avFrame, AV_PIX_FMT_RGB24, outStream->codec->width, outStream->codec->height);
for (int h = 0; h < height; h++)
{
memcpy(&(avFrame->data[0][h*avFrame->linesize[0]]), &(cvFrame.data[h*cvFrame.step]), width*3);
}
The video (here) I get now is almost perfect. It's NOT upside-down, NOT black & white, BUT it seems that one of the RGB components is missing. Every brown/red colors became blue (in original images it should be vice-verse).
What could be the problem? Could rescaling(sws_scale) to AV_PIX_FMT_YUV420P format causes this?
The problem in a nutshell: avpicture_fill() expects no padding between rows, ie the stride (step) to be equal to width*sizeof(pixel), ie 810*3 = 2430. The actual stride of the data in cv::Mat step as you say is 2432 which is different, so just passing the data directly won't work. There is no way to tell avpicture_fill() to use a different stride for the input data; it is not part of the API (you might say it should be :)
There are two possible solutions:
Create an array in which the input data is contiguous, no padding between rows. You'd have to memcopy each row from the cv::Mat into that array. Then pass it to avpicture_fill().
int width, height; // get from mat
uint8_t* buf = malloc(width * height * 3); // 3 bytes per pixel
for (int i = 0; i < height; i++)
{
memcpy( &( buf[ i*width*3 ] ), &( mat->data[ i*mat->step ] ), width*3 );
}
avpicture_fill(..., buf, ...)
Btw, to flip the video vertically, you can do this to copy the last row to the first and so forth:
...
memcpy( &( buf[ i*width*3 ] ), &( mat->data[ (height - i - 1)*mat->step ] ), width*3 );
...
Or, fill in the AVPicture yourself:
AVPicture* pic = malloc(sizeof(AVPicture));
avpicture_alloc(pic, PIX_FMT_BGR24, width, height);
for (int i = 0; i < height; i++)
{
memcpy( &( pic->data[0][ i*pic->linesize[0] ] ), &( mat->data[ i*mat->step ] ), width*3);
}
There is no need to allocate pic->data[0] or set pic->linesize[0], avpicture_alloc() should do that. There is also no need to fill in data[1] or data[2], those should be null.
EDIT: Removed old code which showed copying R, G, B to separate planes. PIX_FMT_BGR24 is not a planar format.
I'm not familiar enough with OpenCV C++ API to figure out how to get the width and height (it's not mat->width, obviously) but I think you know what I mean.
P.S. Btw, your video is not actually black and white. It's just that each successive row is offset by two bytes, so the colors are rotated: red becomes green, green becomes blue, and so forth. The result is grayscale-ish, but if you look closely the individual rows are colored.
Have you considered using OpenCV's features to create the video for you? It's much more easier since your data is already store in a cv::Mat.
If you would like to keep your approach, you could simply rotate the cv::Mat.
About the color problem in the UPDATE of the original post. Is that caused by,
OpenCV Mat is (BGR) -> FFmpeg AVFrame is (RGB) ?
If so, try,
cvtColor( cvFrame , cvFrame , CV_BGR2RGB ) ;
before line 1.