I am attempting to use OpenCV to grab frames from a webcam and display them in a window using SFML.
VideoCapture returns frames in OpenCV's Mat format. To display the frames, SFML requires a 1D array of pixels in its uint8 format, which (as far as I can tell) is interchangeable with uchar. This array is expected to represent 32 bits per pixel RGBA.
So, I have a uchar array, and I'm looping over the Mat data and copying each pixel:
VideoCapture cap(0);
Mat frame;
cap >> frame;
uchar* camData = new uchar[640*480*4];
uchar* pixelPtr = frame.data;
for(int i = 0; i < frame.rows; i++)
{
for(int j = 0; j < frame.cols; j++)
{
camData[i*frame.cols + j + 2] = pixelPtr[i*frame.cols + j + 0]; // B
camData[i*frame.cols + j + 1] = pixelPtr[i*frame.cols + j + 1]; // G
camData[i*frame.cols + j + 0] = pixelPtr[i*frame.cols + j + 2]; // R
camData[i*frame.cols + j + 3] = 255;
}
}
img.LoadFromPixels(640, 480, camData); //Load pixels into SFML Image object for display
Unfortunately, this doesn't quite work. Something in that loop is wrong, as the resulting image when I load and display camData is scrambled.
As far as I can discern, either my math in the loop is wrong so the pixels are being assigned wrong, or the Mat data is in some format other than BGR.
Any ideas?
OpenCV can do all job for you:
VideoCapture cap(0);
Mat frame;
cap >> frame;
uchar* camData = new uchar[frame.total()*4];
Mat continuousRGBA(frame.size(), CV_8UC4, camData);
cv::cvtColor(frame, continuousRGBA, CV_BGR2RGBA, 4);
img.LoadFromPixels(frame.cols, frame.rows, camData);
I like the accepted answer better but this snippet helps you understand what's going on.
for (int i=0; i<srcMat.rows; i++) {
for (int j=0; j<srcMat.cols; j++) {
int index = (i*srcMat.cols+j)*4;
// copy while converting to RGBA order
dstRBBA[index + 0] = srcMat[index + 2 ];
dstRBBA[index + 1] = srcMat[index + 1 ];
dstRBBA[index + 2] = srcMat[index + 0 ];
dstRBBA[index + 3] = srcMat[index + 3 ];
}
}
For me worked following code:
VideoCapture capture(0);
Mat mat_frame;
capture >> mat_frame; // get a new frame from camera
// Be sure that we are dealing with RGB colorspace...
Mat rgbFrame(width, height, CV_8UC3);
cvtColor(mat_frame, rgbFrame, CV_BGR2RGB);
// ...now let it convert it to RGBA
Mat newSrc = Mat(rgbFrame.rows, rgbFrame.cols, CV_8UC4);
int from_to[] = { 0,0, 1,1, 2,2, 3,3 };
mixChannels(&rgbFrame, 2, &newSrc, 1, from_to, 4);
The result (newSrc) is a premultiply image!
Related
still learning openCV, but I've at least managed to find documentation on how to split a video (.avi) - with an alpha channel - into frames using openCV 2.4.13, C++, and using videoCapture().
I previously used the command line tool, FFMPEG, to split and save the avi in question into frames (.png) so that I could check whether or not the RGBA data from each frame in my C++ code matched up with the saved split frame RGBA - the idea is to not have to write the frames to disk, but rather work with just the RGBA data - which is why I'm not just using FFMPEG on the command line.
Here is the code I have so far, I explain my findings at the bottom:
void extract_frames(string& videoFilePath) {
vector<Mat> frameVec;
//open the video file
VideoCapture cap(videoFilePath); // open the video file
if (!cap.isOpened()) // check if we succeeded
CV_Error(CV_StsError, "Can not open Video file");
Mat frame;
for (int frameNum = 0; frameNum < getFrameCountFromAVI(videoFilePath.c_str()); frameNum++)
{
cap >> frame; // get the next frame from video
frameVec.push_back(frame);
}
for (int spltF = 0; spltF < frameVec.size(); spltF++) {
// converting frameVec[spltF] to RGBA
// values when trying to read frameVec[spltF]c directly did not match up with image
// not sure why ???? - maybe its the modified pixels / I had the wrong coordinates
uchar* dstRgBA = new uchar[ ((Mat)(frameVec[spltF])).total() * 4];
uchar* camData = new uchar[ ((Mat)(frameVec[spltF])).total() * 4];
Mat continuousRGBA( ((Mat)(frameVec[spltF])).size(), CV_8UC4, camData);
cv::cvtColor( ((Mat)(frameVec[spltF])), continuousRGBA, CV_BGRA2RGBA, 4);
for (int i = 0; i < ((Mat)(frameVec[spltF])).rows; i++) {
for (int j = 0; j < ((Mat)(frameVec[spltF])).cols; j++) {
int index = (i*((Mat)(frameVec[spltF])).cols + j) * 4;
// copy while converting to RGBA order
uchar r = continuousRGBA.data[index + 2];
uchar g = continuousRGBA.data[index + 1];
uchar b = continuousRGBA.data[index + 0];
uchar a = continuousRGBA.data[index + 3];
dstRGBA[index + 0] = r;
dstRGBA[index + 1] = g;
dstRGBA[index + 2] = b;
dstRGBA[index + 3] = a;
}
}
.... do stuff with dstRGBA ....
}
+}
When debugging, I seem to get the correct RGB (it might be BGR) values at the respective pixel, but it seems that the alpha (A) is always coming up as 255, I'm really hoping the issue is my code and not the limitations of OpenCV.
I'm currently also looking at alternatives, like ffmpeg/libavcodec and C++ - but that is turning out to be a bit more difficult - given that documentation isn't as good as I had hoped.
If any one has any suggestions/ideas/examples/fixes - that would be more than helpful :)
I have been trying lot to get an undistorted image without interpolation. But when executed the below code i get some weird image.I am using the function initUndistortRectifyMap which gives the mapx and mapy of type CV_16SC2 later using the convertMaps function i am converting the mapx and mapy to type CV_32FC1.I have been trying to debug the reason but couldnot find anything helpful.
The distorted image
image after applying undistort without interpolation
int main()
{
Mat Cam1MatrixParam, Cam1Distortion;
Mat cf1;
cf1=imread("cam1.distort1.jpg", CV_LOAD_IMAGE_COLOR);
Size imagesize = cf1.size();
FileStorage fs1("cameracalibration.xml", FileStorage::READ);
fs1["camera_matrix"] >> Cam1MatrixParam;
fs1["distortion_coefficients"] >> Cam1Distortion;
Mat R = Mat::eye(3, 3, CV_32F) * 1;
int width = cf1.cols;
int height = cf1.rows;
Mat undistorted = Mat(height, width, CV_8UC3);
Mat mapx = Mat(height, width, CV_32FC1);
Mat mapy = Mat(height, width, CV_32FC1);
initUndistortRectifyMap(Cam1MatrixParam, Cam1Distortion, Cam1MatrixParam, R, imagesize, CV_16SC2, mapx, mapy);
convertMaps(mapx, mapy, mapx, mapy, CV_32FC1, false);
for (int j = 0; j < height; j++)
{
for ( int i = 0; i < width; i++)
{
undistorted.at<uchar>(mapy.at<float>(j, i), mapx.at<float>(j, i)) = cf1.at<uchar>(j, i);
}
}
imwrite("cam1.undistortimage.png", undistorted);
}
image with this version of code
undistorted.at(j, i) = cf1.at(mapy.at(j, i), mapx.at(j, i));
image with undistort function(remap with nearest interpolation)
It looks like instead of undoing the distortion it applies it once more.
mapx and mapy map from the display coordinates to the photo coordinates.
undistorted.at<cv::Vec3b>(j, i) = distort.at<cv::Vec3b>(mapy.at<float>(j, i), mapx.at<float>(j, i));
You can interpret this code as: for each display coordinate {j, i} find its corresponding (distorted) coordinate in the photo and then copy the pixel.
you are using color images (cv::Vec3b) so try instead:
undistorted.at<cv::Vec3b>(mapy.at<float>(j, i), mapx.at<float>(j, i)) = cf1.at<cv::Vec3b>(j, i);
maybe combined with the answer of Maxim Egorushkin if undistort map is reverse
I want to blend two images like multiply blending in photoshop , i want to do the same in OpenCv using C++ for my app , I visit this many time and try to understand every time but i didn't get it , i search it alot but didn't get what i want other then this but this is little bit strange as conversion is alot from IplImages to ibl etc , Any help , guide, idea and example related opencv is needed . I go through Addweight but i think its quite different from Multiply Blending
Formula which i saw here
Target * Blend
and below is what i tried
Mat img1 = imread("E:\\img.jpg");
Mat img2 = Mat (img1.size(),img1.type());
vector<Mat> colors_1;
split(img2, colors_1);
colors_1[0] = 113;
colors_1[1] = 221;
colors_1[2] = 216;
merge(colors_1,img2);
Mat result(img1.size(), CV_32F);
for(int i = 0; i < img1.size().height; ++i){
for(int j = 0; j < img1.size().width; ++j){
for (int rgb=0 ; rgb<=img1.channels();rgb++){
float target = float(img1.at<uchar>(i, j)) / 255;
float blend = float(img2.at<uchar>(i, j)) / 255;
result.at<float>(i, j) = target*blend;
}
}
}
Result is in GrayScale and its not looking exact
Thank you
You are not accessing the image channels correctly. Moreover, you do not need to store the result in a float image, uchar is OK. Also, your loop on RGB channels should end when rgb<img1.channels().
Try this code:
cv::Mat img1 = cv::imread("E:\\img.jpg");
cv::Mat img2 = cv::Mat (img1.size(),img1.type());
std::vector<cv::Mat> colors_1;
cv::split(img2, colors_1);
colors_1[0] = 113;
colors_1[1] = 221;
colors_1[2] = 216;
cv::merge(colors_1,img2);
cv::Mat result(img1.size(), CV_8UC3);
for(int i = 0; i < img1.rows; ++i){
for(int j = 0; j < img1.cols; ++j){
for (int c=0 ; c<img1.channels();c++){
uchar target = img1.at<uchar>(i, 3*j+c);
uchar blend = img2.at<uchar>(i, 3*j+c);
result.at<uchar>(i, 3*j+c) = cv::saturate_cast<uchar>(target*blend/255.);
}
}
}
I am trying to convert this second answer code into c++ , What I did is not giving me appropriate result , here is my code :
{
Mat img = imread("messi5.jpg");
int level_n = 2;
Mat p = Mat::zeros(img.cols*img.rows, 3, CV_32F);
vector<Mat> bgr;
cv::split(img, bgr);
//Divide each pixel color with 127 for level 2
for(int i=0; i<img.cols*img.rows; i++) {
p.at<float>(i,0) = bgr[0].data[i] / 127.0;
p.at<float>(i,1) = bgr[1].data[i] / 127.0;
p.at<float>(i,2) = bgr[2].data[i] / 127.0;
}
vector<Mat> Img2 = p[bgr];
Mat out;
cv::transform(img,out,p);
imshow ("output" , out);
}
What I didn't understand is how I put these colour's which I divided by 127 into Matrix , where I am going wrong?
Other way i am trying is
vector<Mat> bgr;
Mat blue , green , red;
cv::split(img, bgr);
blue = bgr[0]/127.0;
if (blue > 128)
{
blue = 255;
}
else
{
blue = 0;
}
same for red and green
Why don't just do it in place (for level 2):
Mat img = imread("messi5.jpg");
for(int i=0;i<img.rows;i++)
for(int j=0;j<img.cols;j++) {
cv::Vec3b p = img.at<cv::Vec3b>(i,j);
for(int k = 0;k < img.channels();k++)
p[k] = p[k] > 127 ? 255 : 0;
img.at<cv::Vec3b>(i,j) = p;
}
// do whatever you want with processed image img
i use this code to convert image to matrix ,so someone have any idea how can i convert this matrix to 1D one -->vector
i want to have image data as a 1D array ,in row major order that is all pixel values in the first row are listed first ,followed by pixel values in the second row and so on.
IplImage *img = cvLoadImage( "lena.jpg", CV_LOAD_IMAGE_COLOR);
CvMat *mat = cvCreateMat(img->height,img->width,CV_32FC3 );
cvConvert( img, mat );
for(int i=0;i<10;i++)
{
for(int j=0;j<10;j++){
CvScalar scal = cvGet2D( mat,j,i);
printf( "(%.f,%.f,%.f) ",scal.val[0], scal.val[1], scal.val[2] );}
printf("\n");}
cvNamedWindow("une_window");
cvShowImage("une_window", img);
cvWaitKey();
cvDestroyWindow("une_window");
Using the C++ API:
cv::Mat img = cv::imread("a.jpg");
std::vector<uchar> pixels;
pixels.reserve(img.rows * img.cols * 3);
if(img.isContinuous()) {
pixels = std::vector<uchar>(img.ptr(0), img.ptr(0) + img.rows * img.cols * 3 );
}
else {
for(int i = 0; i != img.rows; ++i) {
uchar* p = img.ptr(i);
for(int j = 0; j != img.cols * 3; ++j) {
pixels.push_back(p[j]);
}
}
}
I believe the fastest way for continuous Mats is to use the reshape command:
Mat colVec = img.reshape(1, img.rows*img.cols); // change to a Nx3 column vector
The reshape command just changes the header, so it does not require pixel access and therefore runs in O(1) time.
I think you should observe from video decoder output to know the video size information, other information collected from metadata in container parser might be not so accurate.
In C++ this is actually a one-liner:
cv::Mat_<float> img = cv::imread("a.jpg", 1);
std::vector<float> dest;
std::copy(img.begin(), img.end(), dest.begin());