Error when trying to access a pixel value in OpenCV - c++

I just started learning c++ and OpenCV. I've tried to write a program that allows me to click on a pixel and print out the pixel value in BGR. So far, when I click, my program returns the coordinate of the clicked pixel but when I want to access the BGR-value my program crashes. I'm streaming video from my webcam as input and I don't get any errors when i compile.
I've been reading a lot of similar questions in here but I haven't found any solution.
My program crashes when I include the line:
"cv::Vec3f BGR = matOriginal.at(click[0][0],click[0][1]);"
The rest of my program is as follows:
int click[1][2];
void mouseEvent(int Event, int x, int y, int flags, void* param)
{
if(Event == CV_EVENT_LBUTTONDOWN)
{
printf("Valgt pixel: [%d,%d] \n ",x,y);
click[0][0] = x;
click[0][1] = y;
}
}
int main()
{
VideoCapture capWebcam(0);
if(capWebcam.isOpened() == false)
{
printf("Error: capWebcam not accessed successfully \n");
return(1);
}
Mat matOriginal;
char charCheckForEscKey=0;
while(charCheckForEscKey !=27)
{
if(capWebcam.read(matOriginal) == NULL)
{
printf("Error: Frame not read\n \n");
break;
}
cv::imshow("Original",matOriginal);
setMouseCallback("Original",mouseEvent,0);
// Print BGR-value when mouseclick
if(click[0][0]>0 && click[0][1]>0)
{
cv::Vec3f BGR = matOriginal.at<cv::Vec3f>(click[0][0],click[0][1]); // When this statement is included, I get an ERROR
double B=BGR[0]; // B
double G=BGR[1]; // G
double R=BGR[2]; // R
click[0][0] = 0; // Resetting the click-matrix
click[0][1] = 0;
}
charCheckForEscKey = cv::waitKey(10);
}
return(0);
}
I'm using Microsoft Visual C++ 2013 Express
Thanks in advance for your help

Ok, two different things could be going on here:
The crash happens because click is not initialized:
When the execution of your program reaches the while loop, and the user has not pressed the mouse button, there's no way to tell what values are stored by click since they are most probably garbage from the memory.
This means that when you program first runs, click could be storing values larger than the dimensions of matOriginal, thus rendering you a crash for trying to access indexes that doesn't exist.
Always initialize your variables!
int main()
{
click[0][0] = -1;
click[0][1] = -1;
// ...
}
What I just said could be tested by printing these values to the screen before accessing them:
std::cout << "X: " << click[0][0] << " Y: " << click[0][1] << std::endl;
cv::Vec3f BGR = matOriginal.at<cv::Vec3f>(click[0][0], click[0][1]);
The crash happens because cv::Mat is expecting the X,Y (row,column) values in the reverse order (column,row):
I think that's most likely the cause of the problem. Try:
cv::Vec3f BGR = matOriginal.at<cv::Vec3f>(click[0][1], click[0][0]);

Related

MOUSEEVENTF_MOVE doesn't move to correct coordinates in-game

Feel like I am bashing my head against the wall here so wanted to reach out and see if there is some easy solution I am missing first before I go crazy.
The problem:
I was tasked to write a TAS for an older directx11 game for a charity event. I want to detect a pixel color, and move to that pixel. I have pixel detection working via OpenCV but the actual movements to that pixel do not line up.
I found a function that does what I want, but it uses a fixed number to try and modify the movement by that I can't figure out for my game.
My mouse dpi does not affect the movement, but the in-game sensitivity does.
The game does not like SetCursorPos and seems to reject that input.
Code:
#include <iostream>
#include <Windows.h>
using namespace std;
//Magic number defines
const float constant = 0.116f;
float mouseSensitivity = 10.0f;
float modifier = mouseSensitivity * constant;
int centerScreenX = GetSystemMetrics(SM_CXSCREEN) / 2;
int centerScreenY = GetSystemMetrics(SM_CYSCREEN) / 2;
//Move mouse to center of screen
void centerMouse() {
SetCursorPos(centerScreenX, centerScreenY);
cout << "Moving to center of screen at(" << centerScreenX << ", " << centerScreenY << ")" << endl;
}
//Calculates actual coordinates for mouse movement based on sensitivity and a constant.
void calibrateCoordinates(int& x, int& y)
{
if (abs(x) < 5)
x = 0;
else {
x = x - centerScreenX;
x = (int)((float)x / modifier);
}
if (abs(y) < 5)
y = 0;
else
{
y = y - centerScreenY;
y = (int)((float)y / modifier);
}
cout << "Coordinates needed to move by (" << x << ", " << y << ")" << endl;
}
// Moves to x,y coordinates after processed into actual coordinates based on sensitivity and a constant.
void moveTo(int x, int y)
{
SetProcessDPIAware();
calibrateCoordinates(x, y);
mouse_event(MOUSEEVENTF_MOVE, x, y, 0, 0);
//Sanity check where the mouse ended up at
POINT p;
if (GetCursorPos(&p))
{
cout << "Mouse ended up at (" << p.x << ", " << p.y << ")" << endl;
}
}
int main() {
while (true) {
// Check if the F19 button is pressed
if (GetAsyncKeyState(VK_F19) & 0x8000) {
//Hardcoded values of pixel we need to move to. Handled automatically via OpenCV in the real code. Simplified here
int xTo = 784;
int yTo = 686;
//Centers mouse to line up cursor with crosshair
centerMouse();
//Tries to move to coords
moveTo(xTo, yTo);
}
}
return 0;
}
Output:
Matched pixel found at (784, 686)[4, 20, 222, 255]
Moving to center of screen at (1280, 720)
Coordinates needed to move by (-271, -20)
Mouse ended up at (1009, 700)
The mouse should have ended up at (1012, 649) for the cross-hair to line up with the pixel I want. Do I just need to keep experimenting to find the magic number that it works with? Or is there any easier way to do this? Thanks
GetSystemMetrics function is not DPI aware. Use GetSystemMetricsForDpi instead to get a correct result.
Disclaimer: Sorry if this isn't an answer but I can't comment yet because of my low rep.
If it's a game, have you tried taking the DX9/11's version of "GetSystemMetrics"? I had the same problem on DX9 aimbot years ago where the mouse aims slightly to the right. Now I can't give the exact answer since I've never done DX11 hooking but this is the mouse_move code for my aimbot:
mouse_event(MOUSEEVENTF_MOVE, coordinates.x - interfaceinfo->D3D9Viewport.Width * 0.5f, coordinates.y - interfaceinfo->D3D9Viewport.Height * 0.5f, 0, 0);
The coordinates is the target's position, while interfaceinfo->D3D9Viewport.Width/Height is the screen info given by the DX9, in this case using **IDirect3DDevice9::GetViewport**
HRESULT WINAPI Renderer::hkPresent(LPDIRECT3DDEVICE9 pDevice, CONST RECT* pSourceRect, CONST RECT* pDestRect, HWND hDestWindowOverride, CONST RGNDATA* pDirtyRegion) {
renderer->Device_D3D9 = pDevice;
Device_D3D9->GetViewport(&interfaceinfo->D3D9Viewport);
return renderer->oPresent(pDevice, pSourceRect, pDestRect, hDestWindowOverride, pDirtyRegion);
}
If it's a game, have you tried moving the mouse by hooking DINPUT functions.
By hooking to GetDeviceState and GetDeviceData, you can emulate keyclicks and mouse clicks/movement without moving your actual mouse (unlike sendinput and mouse_event)
HRESULT WINAPI Renderer::hkGetDeviceState(IDirectInputDevice8A* DIDevice, DWORD cbData, LPVOID *lpvData) {
HRESULT hResult = renderer->oGetDeviceState(DIDevice, cbData, lpvData);
if (interfaceinfo->bCurrentWindow >= 1) {
if (hResult == DI_OK) {
for (auto i = 0; i < cbData; i++) {
lpvData[i] = 0;
}
}
}
static BYTE buffer[256];
if (true) {
buffer[DIK_W] = LOBYTE(0x80);
buffer['w'] = LOBYTE(0x80);
buffer['W'] = LOBYTE(0x80);
cbData = 256;
memcpy(lpvData, buffer, cbData);
}
else {
hResult = renderer->oGetDeviceState(DIDevice, cbData, lpvData);
}
return hResult;
}
HRESULT WINAPI Renderer::hkGetDeviceData(IDirectInputDevice8A* DIDevice, DWORD cbObjectData, LPDIDEVICEOBJECTDATA rgdod, LPDWORD pdwInOut, DWORD dwFlags) {
HRESULT ret = renderer->oGetDeviceData(DIDevice, cbObjectData, rgdod, pdwInOut, dwFlags);
if (ret == DI_OK) {
for (DWORD i = 0; i < *pdwInOut; ++i) {
if (interfaceinfo->bCurrentWindow >= 1) {
*pdwInOut = 0;
}
else {
switch (rgdod[i].dwOfs) {
case DIK_W:
rgdod[i].dwData = LOBYTE(0x80);
break;
case DIK_SPACE: {
static byte last_written = 0;
rgdod[i].dwData = HIBYTE(last_written ? 0x01 : 0x00);
last_written = last_written ? 0x00 : 0x01;
break;
}
}
}
}
}
return ret;
}
First of all, you appear to be passing absolute coordinates to the function mouse_event, but that function is interpreting them as coordinates relative to the last mouse position, because you are not passing the MOUSEEVENTF_ABSOLUTE flag to mouse_event. Therefore, if you want to pass absolute coordinates, you should also pass that flag. Also, Microsoft recommends that you use SendInput instead of mouse_event.
Another issue is the following:
The official Microsoft documentation for the function GetSystemMetrics states the following:
This API is not DPI aware, and should not be used if the calling thread is per-monitor DPI aware. For the DPI-aware version of this API, see GetSystemMetricsForDPI. For more information on DPI awareness, see the Windows High DPI documentation.
Therefore, the results of the lines
int centerScreenX = GetSystemMetrics(SM_CXSCREEN) / 2;
int centerScreenY = GetSystemMetrics(SM_CYSCREEN) / 2;
are possibly incorrect. I suggest that you change these lines to the following:
int centerScreenX = GetSystemMetricsForDpi( SM_CXSCREEN, GetDpiForSystem() ) / 2;
int centerScreenY = GetSystemMetricsForDpi( SM_CYSCREEN, GetDpiForSystem() ) / 2;
However, the function GetDpiForSystem will simply return the value 96 unless the process or thread is made DPI-aware beforehand. Therefore, it is important that you call the function SetProcessDPIAware before calling the function GetDPIForSystem. Alternatively, you can set the default DPI awareness through an application manifest setting, as described here.
According to this Microsoft documentation, there are currently 4 different modes of DPI awareness:
Unaware
System
Per-Monitor
Per-Monitor V2
Since you are using the function SetProcessDPIAware, you are setting the DPI awareness mode of your process to "System". This mode will only make your process aware of the DPI on the primary monitor at the time the user logged in. If the DPI was changed since then, then your process will be unaware of these changes. For this reason, you may want to consider using "Per-Monitor V2" DPI-awareness instead, by using the function SetProcessDpiAwarenessContext or SetThreadDpiAwarenessContext. However, this will require your application to handle WM_DPICHANGED messages.

Mat cells set to NULL in OpenCV?

Quick summary:
I create a cv::Mat by
cv::Mat m = cv::Mat::zeros(MAP_HEIGHT, MAP_WIDTH, CV_8UC1)
My approach after this is to see if i have any polygons in a list of polygons, and if i do, fill them in, and lastly i assign m to my public cv::Mat map (defined in the header-file).
What happens is basically:
cv::Mat m = cv::Mat::zeros(MAP_HEIGHT, MAP_WIDTH, CV_8UC1);
// possibly fill polygons with 1's. Nothing happens if there are no polygons
map = m;
The logic of my program is that position x,y is allowed if a 0 is occupying the cell. So no polygons => all map should be 'legit'.
I have defined this method to check whether a given x-y coordinate is allowed.
bool Map::isAllowed(bool res, int x, int y) {
unsigned char allowed = 0;
res = (map.ptr<unsigned char>(y)[x] == allowed);
}
Now the mystery begins.
cout << cv::countNonZero(map) << endl; // prints 0, meaning all cells are 0
for(int i = 0; i < MAP_HEIGHT; i++) {
unsigned char* c = map.ptr<unsigned char>(i);
for(int j = 0; j < MAP_WIDTH; j++) {
cout << c[j] << endl;
}
} // will print nothing, only outputs empty lines, followed by a newline.
If i print (c[j] == NULL) it prints 1.
If i print the entire Mat i see only 0's flashing over my screen, so they are clearly there.
Why does isAllowed(bool, x, y) return false for (0,0), when there is clearly a 0 there?
Let me know if any more information is needed, thanks!
Problem is solved now, here are my mistakes for future reference:
1: When printing, #Miki pointed out that unsigned characters -> ASCII value gets printed, not numerical representation.
2: in isAllowedPosition(bool res, int x, int y), res has a primitive type. Aka this is pushed on the stack and not a reference to a memorylocation. When writing to it, i write to the local copy and not to the one passed in as an argumet.
Two possible fixes, either pass in a pointer to a memorylocation and write to that, or simply return the result.
Since your data type is uchar (aka unsigned char), you're printing the ASCII value. Use
cout << int(c[j]) << endl;
to print the actual value.
Also map.ptr<unsigned char>(y)[x] can be rewritten simply as map.at<uchar>(y,x), or if you use Mat1b as map(y,x)

running opencv code with matlab via mex fails while on VisualStudio it works

I want to extract some harriscorners from an image and get FREAK descriptors. Here is how I try to do it:
(The passed variables are globally defined.)
void computeFeatures(cv::Mat &src, std::vector<cv::KeyPoint> &keypoints, cv::Mat &desc ) {
cv::Mat featureSpace;
featureSpace = cv::Mat::zeros( src.size(), CV_32FC1 );
//- Detector parameters
int blockSize = 3;
int apertureSize = 3;
double k = 0.04;
//- Detecting corners
cornerHarris( src, featureSpace, blockSize, apertureSize, k, cv::BORDER_DEFAULT );
//- Thresholding featureSpace
keypoints.clear();
nonMaximumSuppression(featureSpace, keypoints, param.nms_n);
//- compute FREAK-descriptor
cv::FREAK freak(false, false, 22.0f, 4);
freak.compute(src, keypoints, desc);
}
I can compile it with Visual Studio 12 as well as Matlab R2013b via mex. When I run it as "stand alone" (.exe) it works just fine. When I try to execute it via Matlab it fails with this message:
A buffer overrun has occurred in MATLAB.exe which has corrupted the
program's internal state. Press Break to debug the program or Continue
to terminate the program.
I mexed with the debug option '-g' and attached VisualStudio to Matlab to be able to get closer to the error:
After nonMaximumSuppression() the size of keypoints is 233 when I jump into freak.compute() the size is suddenly 83 with "random" values stored.
The actual error is then in KeyPointsFilter::runByKeypointSize when keypoints should be erased.
in keypoint.cpp line 256:
void KeyPointsFilter::runByKeypointSize( vector<KeyPoint>& keypoints, float minSize, float maxSize )
{
CV_Assert( minSize >= 0 );
CV_Assert( maxSize >= 0);
CV_Assert( minSize <= maxSize );
keypoints.erase( std::remove_if(keypoints.begin(), keypoints.end(), SizePredicate(minSize, maxSize)),
keypoints.end() );
}
Is there some error I'm making with passing the keyPoint-vector? Has anybody run into a similar problem?
EDIT:
Here is the mex-file with the additional library "opencv_matlab.hpp" taken from MatlabCentral
#include "opencv_matlab.hpp"
void mexFunction (int nlhs,mxArray *plhs[],int nrhs,const mxArray *prhs[]) {
// read command
char command[128];
mxGetString(prhs[0],command,128);
if (!strcmp(command,"push") || !strcmp(command,"replace")) {
// check arguments
if (nrhs!=1+1 && nrhs!=1+2)
mexErrMsgTxt("1 or 2 inputs required (I1=left image,I2=right image).");
if (!mxIsUint8(prhs[1]) || mxGetNumberOfDimensions(prhs[1])!=2)
mexErrMsgTxt("Input I1 (left image) must be a uint8_t matrix.");
// determine input/output image properties
const int *dims1 = mxGetDimensions(prhs[1]);
const int nDims1 = mxGetNumberOfDimensions(prhs[1]);
const int rows1 = dims1[0];
const int cols1 = dims1[1];
const int channels1 = (nDims1 == 3 ? dims1[2] : 1);
// Allocate, copy, and convert the input image
// #note: input is double
cv::Mat I1_ = cv::Mat::zeros(cv::Size(cols1, rows1), CV_8UC(channels1));
om::copyMatrixToOpencv<uchar>((unsigned char*)mxGetPr(prhs[1]), I1_);
// push back single image
if (nrhs==1+1) {
// compute features and put them to ring buffer
pushBack(I1_,!strcmp(command,"replace"));
// push back stereo image pair
} else {
if (!mxIsUint8(prhs[2]) || mxGetNumberOfDimensions(prhs[2])!=2)
mexErrMsgTxt("Input I2 (right image) must be a uint8_t matrix.");
// determine input/output image properties
const int *dims2 = mxGetDimensions(prhs[2]);
const int nDims2 = mxGetNumberOfDimensions(prhs[2]);
const int rows2 = dims2[0];
const int cols2 = dims2[1];
const int channels2 = (nDims2 == 3 ? dims2[2] : 1);
// Allocate, copy, and convert the input image
// #note: input is double
cv::Mat I2_ = cv::Mat::zeros(cv::Size(cols2, rows2), CV_8UC(channels2));
om::copyMatrixToOpencv<uchar>((unsigned char*)mxGetPr(prhs[2]), I2_);
// check image size
if (dims1_[0]!=dims2_[0] || dims1_[1]!=dims2_[1])
mexErrMsgTxt("Input I1 and I2 must be images of same size.");
// compute features and put them to ring buffer
pushBack(I1_,I2_,!strcmp(command,"replace"));
}
}else {
mexPrintf("Unknown command: %s\n",command);
}
}
And here is an additional part of the main cpp project.
std::vector<cv::KeyPoint> k1c1, k2c1, k1p1, k2p1; //KeyPoints
cv::Mat d1c1, d2c1, d1p1, d2p1; //descriptors
void pushBack (cv::Mat &I1,cv::Mat &I2,const bool replace) {
// sanity check
if (I1.empty()) {
cerr << "ERROR: Image empty!" << endl;
return;
}
if (replace) {
//if (!k1c1.empty())
k1c1.clear(); k2c1.clear();
d1c1.release(); d2c1.release();
} else {
k1p1.clear(); k2p1.clear();
d1p1.release(); d2p1.release();
k1p1 = k1c1; k2p1 = k2c1;
d1c1.copyTo(d1p1); d2c1.copyTo(d2p1);
k1c1.clear(); k2c1.clear();
d1c1.release(); d2c1.release();
}
// compute new features for current frame
computeFeatures(I1,k1c1,d1c1);
if (!I2.empty())
computeFeatures(I2,k2c1,d2c1);
}
And here is how I call the mex-file from Matlab
I1p = imread('\I1.bmp');
I2p = imread('\I2.bmp');
harris_freak('push',I1p,I2p);
Hope this helps...
I hope this is the correct way to give an answer to my own question.
After a couple of days I found kind of a work around. Instead of building the mex file in Matlab, which gives the above mentioned error, I built it in Visual Studio with instructions taken from here.
Now everything works just fine.
It kind of bothers me to not know how to do it with matlab, but hey, maybe someone still has an idea.
Thanks to the commenters for taking the time to look through my question!
If you have the Computer Vision System Toolbox then you do not need mex. It includes the detectHarrisFeatures function for detecting Harris corners, and the extractFeatures function, which can compute FREAK descriptors.

Error in using cvVideoWriter in opencv?

This code snippet is supposed to save part of a video whose range is defined by start and end. There is an array of structures (data[i]) that holds the starting and end frame of a video shot in the original video. There are total of 8 shots.
for (int i = 0; i < finalCount-1; ++i) {
capture = cvCaptureFromAVI("Stats\\Shots\\Cricketc1.avi");
assert(capture);
int frame_number = 0;
int start = data[i].start_frame;
int end = data[i].end_frame;
char shotname[100];
strcpy_s(shotname, "shot_");
char shot_id[30];
_itoa_s(data[i].shot_no, shot_id, 10);
strcat_s(shotname, shot_id);
strcat_s(shotname, ".avi");
IplImage* image = NULL;
CvVideoWriter* writer = NULL;
writer = cvCreateVideoWriter (shotname, CV_FOURCC('i','Y','U','V'), fps, cvSize(width, height), 1);
assert(writer);
while (frame_number >= start && frame_number < end) {
image = cvQueryFrame(capture);
assert(image);
cvWriteFrame(writer, image);
}
cvReleaseImage(&image);
cvReleaseVideoWriter(&writer);
cvReleaseCapture(&capture);
cout << shotname << " saved ..." << endl;
}
After running the program 8 video files are created that have a size of 6kb and do not run. I have tried various codecs like divx, mjpg, mpg2, iyuv etc but all give the same result.
In your while loop, frame_number is never incremented. Since you say the program actually executes and creates the files this means nothing in your while loop ever runs... otherwise you'd get stuck in an infinite loop because frame_number will always be 0.
I would advise you initialize frame_number to start instead of 0 and there's no reason for it to exist outside of the scope of the loop so a for seems more appropriate:
int start = data[i].start_frame;
int end = data[i].end_frame;
...
for (int frame_number = start; frame_number < end; frame_number++) {
image = cvQueryFrame(capture);
assert(image);
cvWriteFrame(writer, image);
}
If Gunther Fox answer won't help try to use different codec - it's very strange, but in my situation iyuv is not working at all and some other codecs works ok, but i can't read them while debugging... For me - ms video and radius cinepak always works fine(writing and reading), iyuv is not working at all, other codes - writing and reading, but not while debugging.

Determining the type of Matrix in OpenCV

I'm trying to index a 3 channel image in opencv.
When I read in image files this code works
int Blue = LeftCol.at<cv::Vec3b>(v,u)[0];
int Green = LeftCol.at<cv::Vec3b>(v,u)[1];
int Red = LeftCol.at<cv::Vec3b>(v,u)[2];
But it crashes when I use a webcam input. The webcam has 3 channels and u,v starts at 0,0.
I have no idea why it won't work.
I've tried all variations of Vec3b, Vec3i, Vec3s, Vec3f, Vec3d
I'm lost.... why can't I index this webcam image?
EDIT
Right so after many hours this is where I've got to...here's an outline of the program. I was having the problem I mentioned above inside a function. So I've gone back to basic, trying to look at the matrix before the function...
void main (int argc, char** argv) {
Mat LeftCol;
while (1==1) {
if (ProgramMode == "Files") {
//read in the colour images
LeftCol = imread(ColImLeft.c_str(),1);
RightCol = imread(ColImRight.c_str(),1);
} else if (ProgramMode == "Camera") {
VideoCapture CapLeft, CapRight;
CapLeft.open(1);
CapRight.open(2);
CapLeft >> LeftCol;
CapRight >> RightCol;
//THIS WORKS, THIS PIXEL VALUES ARE DISPLAYED
cout << "uchar" << endl;
for (int x=0;x<10;x++) {
for (int y=0;y<10;y++) {
int pixel = LeftCol.at<cv::Vec3b>(x,y)[0];
cout << pixel;
}
cout << endl;
}
} //end if
///////ADDED THIS BIT ////////
cout << "channels = " << LeftCol.channels() << endl;
//^^This bit works, output shows "channels = 3"
//vv This bit doesn't work.... so there's a problem with LeftCol.
//I wonder if reading the data like CapLeft >> LeftCol; is changing something
imshow("Test",LeftCol);
///////ADDED THIS BIT ////////
//THIS DOES NOT WORK WHEN USING THE CAMERA INPUT, PROGRAM CRASHES
cout << "uchar" << endl;
for (int x=0;x<10;x++) {
for (int y=0;y<10;y++) {
int pixel = LeftCol.at<cv::Vec3b>(x,y)[0];
cout << pixel;
} //end for
cout << endl;
} //end for
} //end while
} //end main
Right I have got it working but it's not ideal. I'm creating a temp Mat to read the files into it then cloning them.
Mat TempLeft;
Mat TempRight;
VideoCapture CapLeft, CapRight;
CapLeft.open(1);
CapRight.open(2);
CapLeft >> TempLeft;
CapRight >> TempRight;
LeftCol = TempLeft.clone();
RightCol = TempRight.clone();
OpenCV makes soft copies of images whenever possible. From the documentation:
the array assignment is an O(1) operation because it only copies the header and increases the reference counter. The Mat::clone() method can be used to get a full (deep) copy of the array when you need it.
I suspect what is happening is LeftCol uses data which still belongs with the VideoCapture object. If this is the case then when CapLeft and CapRight go out of scope at the end of the if they are closed by the destructor and the image data which LeftCol is still pointing to is destroyed.
Possible solutions would be to clone the image as you are doing, or declare VideoCapture CapLeft, CapRight; outside of the if block (you can still open them inside if needed).
You can check the type with cv::Mat.type() and the number of channels with cv::Mat.channels()
The data returned from the camera will be converted into Vec3b (ie uchar * 3) in B,G,R order.
Are you sure the image is valid- is there a mistake somewhere else?
Just because you tried all Vec* combinations, here is the full list, my 2ct:
typedef Vec< uchar, 3 > Vec3b (so normal 8 bit)
typedef Vec< double,3 > Vec3d (so normal double precision)
typedef Vec< float, 3 > Vec3f (so normal floating point)
typedef Vec< int, 3 > Vec3i (normal int)
typedef Vec< short, 3 > Vec3s
typedef Vec< ushort 3 > Vec3w (so normal 16 bit)