I have made a directshow filter of decoder using libde265. There is built in function named write_image. It writes the decoded data in a yuv file.
I need to render the decoder data.
For that purpose I need to do two steps:
Output data on the output pin
Conversion of the data into rgb format
The media_subtype used is IMC3. IN IMC3 the format of the data is , y components following u and v in the memory .
I have tried the following code to output the data on the pin.
static FILE* fh = NULL;
if (fh == NULL) { fh = fopen(output_filename, "wb"); }
for (int y = 0; y<de265_get_image_height(img, 0); y++)
fread(out, de265_get_image_width(img, 0), 1, fh);
for (int y = 0; y<de265_get_image_height(img, 1); y++)
fread(out, de265_get_image_width(img, 1), 1, fh);
for (int y = 0; y<de265_get_image_height(img, 2); y++)
fread(out, de265_get_image_width(img, 2), 1, fh);
But the render screen is blank. Secondly I need to convert it to rgb as well. In the above code img is the image structure storing the decoded data.
Please help me in this regard because render is not showing anything. As well as suggest me to convert the data in the rgb format as well. May be I may be wrong in packing the data in the output buffer.Although I am following the exact IMC3 format
Related
still learning openCV, but I've at least managed to find documentation on how to split a video (.avi) - with an alpha channel - into frames using openCV 2.4.13, C++, and using videoCapture().
I previously used the command line tool, FFMPEG, to split and save the avi in question into frames (.png) so that I could check whether or not the RGBA data from each frame in my C++ code matched up with the saved split frame RGBA - the idea is to not have to write the frames to disk, but rather work with just the RGBA data - which is why I'm not just using FFMPEG on the command line.
Here is the code I have so far, I explain my findings at the bottom:
void extract_frames(string& videoFilePath) {
vector<Mat> frameVec;
//open the video file
VideoCapture cap(videoFilePath); // open the video file
if (!cap.isOpened()) // check if we succeeded
CV_Error(CV_StsError, "Can not open Video file");
Mat frame;
for (int frameNum = 0; frameNum < getFrameCountFromAVI(videoFilePath.c_str()); frameNum++)
{
cap >> frame; // get the next frame from video
frameVec.push_back(frame);
}
for (int spltF = 0; spltF < frameVec.size(); spltF++) {
// converting frameVec[spltF] to RGBA
// values when trying to read frameVec[spltF]c directly did not match up with image
// not sure why ???? - maybe its the modified pixels / I had the wrong coordinates
uchar* dstRgBA = new uchar[ ((Mat)(frameVec[spltF])).total() * 4];
uchar* camData = new uchar[ ((Mat)(frameVec[spltF])).total() * 4];
Mat continuousRGBA( ((Mat)(frameVec[spltF])).size(), CV_8UC4, camData);
cv::cvtColor( ((Mat)(frameVec[spltF])), continuousRGBA, CV_BGRA2RGBA, 4);
for (int i = 0; i < ((Mat)(frameVec[spltF])).rows; i++) {
for (int j = 0; j < ((Mat)(frameVec[spltF])).cols; j++) {
int index = (i*((Mat)(frameVec[spltF])).cols + j) * 4;
// copy while converting to RGBA order
uchar r = continuousRGBA.data[index + 2];
uchar g = continuousRGBA.data[index + 1];
uchar b = continuousRGBA.data[index + 0];
uchar a = continuousRGBA.data[index + 3];
dstRGBA[index + 0] = r;
dstRGBA[index + 1] = g;
dstRGBA[index + 2] = b;
dstRGBA[index + 3] = a;
}
}
.... do stuff with dstRGBA ....
}
+}
When debugging, I seem to get the correct RGB (it might be BGR) values at the respective pixel, but it seems that the alpha (A) is always coming up as 255, I'm really hoping the issue is my code and not the limitations of OpenCV.
I'm currently also looking at alternatives, like ffmpeg/libavcodec and C++ - but that is turning out to be a bit more difficult - given that documentation isn't as good as I had hoped.
If any one has any suggestions/ideas/examples/fixes - that would be more than helpful :)
I am working with TIF images containing signed integer data. After successfully inputing one and processing it I need to output the image in the same format (input and output both *.tif files).
For the input, I know that OpenCV does not know if the data is signed or unsigned, so it assumes unsigned. Using this trick solves that problem (switching the type of cv::Mat by hand).
However, when I output the image and load it again, I do not get the expected result. The file contains multiple segments (groups of pixels), and the format is as follows (I must use this format):
all pixels not belonging to any segment have the value -9999
all the pixels belonging to a single segment have the same positive integer value
(e.g. all pixels of 1st segment have value 1, second 2 etc)
And here is the example code:
void ImageProcessor::saveSegments(const std::string &filename){
cv::Mat segmentation = cv::Mat(workingImage.size().height,
workingImage.size().width,
CV_32S, cv::Scalar(-9999));
for (int i=0, szi = segmentsInput.size(); i < szi; ++i){
for (int j=0, szj = segmentsInput[i].size(); j < szj; ++j){
segmentation.at<int>(segmentsInput[i][j].Y,
ssegmentsInput[i][j].X) = i+1;
}
}
cv::imwrite(filename, segmentation);
}
You can assume that all the variables (e.g. workingImage, segmentsInput) exist as global variables.
Using this code, when I input the image and examine the values, most of the values are set to 0 while the ones that are set take a full range of integer values (in my example I had 20 segments).
You can't save integer matrices directly with imwrite. As the documentation states: "Only 8-bit (or 16-bit unsigned (CV_16U) in case of PNG, JPEG 2000, and TIFF) single-channel or 3-channel (with ‘BGR’ channel order) images can be saved using this function."
However, what you could do it to convert your CV_32S matrix to a CV_8UC4 and save it as a PNG with no compression. Of course, this is a bit unsafe since endianness comes into play and may change your values between different systems or compilers (especially since we're talking about signed integers here). If you use always the same system and compiler, you can use this:
cv::Mat segmentation = cv::Mat(workingImage.size().height,
workingImage.size().width,
CV_32S, cv::Scalar(-9999));
cv::Mat pngSegmentation(segmentation.rows, segmentation.cols, CV_8UC4, (cv::Vec4b*)segmentation.data);
std::vector<int> params;
params.push_back(CV_IMWRITE_PNG_COMPRESSION);
params.push_back(0);
cv::imwrite("segmentation.png", pngSegmentation, params);
I also save opencv mats as tifs but i don`t use the opencv tif solution. I include the libtiff lib on my own (i think libtiff is also used in opencv) and than you can use the following code to save as tiff
TIFF* tif = TIFFOpen("file.tif", "w");
if (tif != NULL) {
for (int i = 0; i < pages; i++)
{
TIFFSetField(tif, TIFFTAG_IMAGEWIDTH, TIFF_UINT64_T(x)); // set the width of the image
TIFFSetField(tif, TIFFTAG_IMAGELENGTH, TIFF_UINT64_T(y)); // set the height of the image
TIFFSetField(tif, TIFFTAG_SAMPLESPERPIXEL, 1); // set number of channels per pixel
TIFFSetField(tif, TIFFTAG_BITSPERSAMPLE, 32); // set the size of the channels 32 for CV_32F
TIFFSetField(tif, TIFFTAG_PAGENUMBER, i, pages);
TIFFSetField(tif, TIFFTAG_SAMPLEFORMAT, SAMPLEFORMAT_IEEEFP); // for CV32_F
for (uint32 row = 0; row < y; row++)
{
TIFFWriteScanline(tif, &imageDataStack[i].data[row*x*32/ 8], row, 0);
}
TIFFWriteDirectory(tif);
}
}
imageDataStack is a vector of cv::Mat objects. This code works for me to save tiff stacks.
I am trying to transfer one matlab project to C++ code. However, when I trying to read an mp4 video by frame, the RGB values for each pixels is very different from Matlab. Does it mean OpenCV used a different RGB value representation? If so, how can I change the OpenCV value to Matlab? Otherwise I cannot verify my implementation are right by checking values.
For example:
I am trying to check the point(0,0) values in OpenCV and Matlab. OpenCV gives the following result: blue=106 green=105 red=102
However, in Matlab, the result is: blue=85 green=86 red=83
I tried to get the RGB value in point(0,0) for every 200 frames which is point(1,1) in Matlab.
The C++ code to get RGB value in OpenCV are:
Mat img;
number = 0;
VideoCapture cap(filename_input_video);
if(!cap.isOpened()) {
printf("No video to Read!\n");
return -1;
}
for( ; ; ) {
cap >> img;
if(img.empty())
break;
number++;
for(int i=0; i<img.rows; i++) {
for(int j=0; j<img.cols; j++) {
int blue = img.at<Vec3b>(i, j)[0];
int green = img.at<Vec3b>(i, j)[1];
int red = img.at<Vec3b>(i, j)[2];
if(number == 200 && i==0 && j==0) {
printf("blue=%d green=%d red=%d", blue, green, red);
}
}
}
if(number == 200) {
number = 0;
}
}
The Matlab code is:
OBJ = VideoReader(filename_source);
fBlock = 200;
nFrame = get(OBJ, 'NumberOfFrames');
nBlock = ceil(nFrame / fBlock);
for iBlock = 1:nBlock
display(['Processing video 1 block #' num2str(iBlock) '...']);
start_index = (iBlock-1)*fBlock+1;
end_index = min(iBlock*fBlock, nFrame);
vSource = read(OBJ,[start_index end_index]);
display(['red ' num2str(vSource(1,1,1,200))]);
display(['green ' num2str(vSource(1,1,2,200))]);
display(['blue ' num2str(vSource(1,1,3,200))]);
How should i fix this problem?
To verify the difference, you should compare RGB value of single image read from disc. Reading identical values here shows your code is probably fine and there is a difference in decoding.
What is probably happening : If you read frame/image captured from video, there can be difference as the video decoder can be different for OpenCV(default is ffmpeg) and MATLAB. Different decoder can handle some events/errors differently and there is no guarantee of identical decoding.
Suggested Solution :
1) Same decoder - If you need both tools to be identical in result, use same decoder for both. Either change decoder for OpenCV or for MATLAB. If you google, you will find few article on how to do this. This and this can be helpful.
2) Same video - Use any decoder(I prefer ffmpeg) to convert video to raw format first. Now you can use it on both tool without fear of diff ;). Here is a command to get raw from compresed:
`c:/> ffmpeg -i compressed_or_original_video.avi -vcodec rawvideo raw_converted_video.avi`
No! it does not! you see different result because C++ array indexing starts from zero while matlab/octave indexing starts from 1.
s it possible to:
read an image given by just a filename (not knowing the image format) to a 2d matrix rgb uncompressed form (e.g. read an JPG to a 2d array)
access the bytes of that image, copy them, change them... (e.g. inverse the colors, I need a pointer to the image bytes, setters/getters won't do )
rgb8_image_t img;
jpeg_read_image ("lena.jpg",img);
i use these to load the image. now how do i access the pixels or bytes of this image?
Here is a sample that sets G component of all pixels to 128
rgb8_image_t img;
const rgb8_view_t & mViewer = view(img);
for (int y = 0; y < mViewer.height; ++y)
{
rgb8_view_t::x_iterator trIt = mViewer.row_begin(y);
for (int x = 0; x < mViewer.width; ++x)
at_c<1>(trIt[x]) = 128;
}
I have colored jpeg images of OpenCV::Mat type and I create from them video using avcodec. The video that I get is upside-down, black & white and each row of each frame is shifted and I got diagonal line. What could be the reason for such output?
Follow this link to watch the video I get using avcodec.
I'm using acpicture_fill function to create avFrame from cv::Mat frame!
P.S.
Each cv::Mat cvFrame has width=810, height=610, step=2432
I noticed that avFrame (that is filled by acpicture_fill) has linesize[0]=2430
I tried manually setting avFrame->linesizep0]=2432 and not 2430 but it still didn't helped.
======== CODE =========================================================
AVCodec *encoder = avcodec_find_encoder(AV_CODEC_ID_H264);
AVStream *outStream = avformat_new_stream(outContainer, encoder);
avcodec_get_context_defaults3(outStream->codec, encoder);
outStream->codec->pix_fmt = AV_PIX_FMT_YUV420P;
outStream->codec->width = 810;
outStream->codec->height = 610;
//...
SwsContext *swsCtx = sws_getContext(outStream->codec->width, outStream->codec->height, PIX_FMT_RGB24,
outStream->codec->width, outStream->codec->height, outStream->codec->pix_fmt, SWS_BICUBIC, NULL, NULL, NULL);
for (uint i=0; i < frameNums; i++)
{
// get frame at location I using OpenCV
cv::Mat cvFrame;
myReader.getFrame(cvFrame, i);
cv::Size frameSize = cvFrame.size();
//Each cv::Mat cvFrame has width=810, height=610, step=2432
1. // create AVPicture from cv::Mat frame
2. avpicture_fill((AVPicture*)avFrame, cvFrame.data, PIX_FMT_RGB24, outStream->codec->width, outStream->codec->height);
3avFrame->width = frameSize.width;
4. avFrame->height = frameSize.height;
// rescale to outStream format
sws_scale(swsCtx, avFrame->data, avFrame->linesize, 0, outStream->codec->height, avFrameRescaledFrame->data, avFrameRescaledFrame ->linesize);
encoderRescaledFrame->pts=i;
avFrameRescaledFrame->width = frameSize.width;
avFrameRescaledFrame->height = frameSize.height;
av_init_packet(&avEncodedPacket);
avEncodedPacket.data = NULL;
avEncodedPacket.size = 0;
// encode rescaled frame
if(avcodec_encode_video2(outStream->codec, &avEncodedPacket, avFrameRescaledFrame, &got_frame) < 0) exit(1);
if(got_frame)
{
if (avEncodedPacket.pts != AV_NOPTS_VALUE)
avEncodedPacket.pts = av_rescale_q(avEncodedPacket.pts, outStream->codec->time_base, outStream->time_base);
if (avEncodedPacket.dts != AV_NOPTS_VALUE)
avEncodedPacket.dts = av_rescale_q(avEncodedPacket.dts, outStream->codec->time_base, outStream->time_base);
// outContainer is "mp4"
av_write_frame(outContainer, & avEncodedPacket);
av_free_packet(&encodedPacket);
}
}
UPDATED
As #Alex suggested I changed the lines 1-4 with the code below
int width = frameSize.width, height = frameSize.height;
avpicture_alloc((AVPicture*)avFrame, AV_PIX_FMT_RGB24, outStream->codec->width, outStream->codec->height);
for (int h = 0; h < height; h++)
{
memcpy(&(avFrame->data[0][h*avFrame->linesize[0]]), &(cvFrame.data[h*cvFrame.step]), width*3);
}
The video (here) I get now is almost perfect. It's NOT upside-down, NOT black & white, BUT it seems that one of the RGB components is missing. Every brown/red colors became blue (in original images it should be vice-verse).
What could be the problem? Could rescaling(sws_scale) to AV_PIX_FMT_YUV420P format causes this?
The problem in a nutshell: avpicture_fill() expects no padding between rows, ie the stride (step) to be equal to width*sizeof(pixel), ie 810*3 = 2430. The actual stride of the data in cv::Mat step as you say is 2432 which is different, so just passing the data directly won't work. There is no way to tell avpicture_fill() to use a different stride for the input data; it is not part of the API (you might say it should be :)
There are two possible solutions:
Create an array in which the input data is contiguous, no padding between rows. You'd have to memcopy each row from the cv::Mat into that array. Then pass it to avpicture_fill().
int width, height; // get from mat
uint8_t* buf = malloc(width * height * 3); // 3 bytes per pixel
for (int i = 0; i < height; i++)
{
memcpy( &( buf[ i*width*3 ] ), &( mat->data[ i*mat->step ] ), width*3 );
}
avpicture_fill(..., buf, ...)
Btw, to flip the video vertically, you can do this to copy the last row to the first and so forth:
...
memcpy( &( buf[ i*width*3 ] ), &( mat->data[ (height - i - 1)*mat->step ] ), width*3 );
...
Or, fill in the AVPicture yourself:
AVPicture* pic = malloc(sizeof(AVPicture));
avpicture_alloc(pic, PIX_FMT_BGR24, width, height);
for (int i = 0; i < height; i++)
{
memcpy( &( pic->data[0][ i*pic->linesize[0] ] ), &( mat->data[ i*mat->step ] ), width*3);
}
There is no need to allocate pic->data[0] or set pic->linesize[0], avpicture_alloc() should do that. There is also no need to fill in data[1] or data[2], those should be null.
EDIT: Removed old code which showed copying R, G, B to separate planes. PIX_FMT_BGR24 is not a planar format.
I'm not familiar enough with OpenCV C++ API to figure out how to get the width and height (it's not mat->width, obviously) but I think you know what I mean.
P.S. Btw, your video is not actually black and white. It's just that each successive row is offset by two bytes, so the colors are rotated: red becomes green, green becomes blue, and so forth. The result is grayscale-ish, but if you look closely the individual rows are colored.
Have you considered using OpenCV's features to create the video for you? It's much more easier since your data is already store in a cv::Mat.
If you would like to keep your approach, you could simply rotate the cv::Mat.
About the color problem in the UPDATE of the original post. Is that caused by,
OpenCV Mat is (BGR) -> FFmpeg AVFrame is (RGB) ?
If so, try,
cvtColor( cvFrame , cvFrame , CV_BGR2RGB ) ;
before line 1.