OpenCV 2 crashes with stackoverflow during inRange - c++

I have created a multithreaded program that gets one frame on an image acquisition thread and the transfers it to the image processing thread. The image seems to get copied entirely but opencv crashes during the inRange function call. I'm going to briefly outline the flow and paste a few code snippets to show how the images are being copied around.
First: this is opencv 2.4.13
for the flow of data, I have one thread read from the camera and stick the image in an Mat accessible to both of the threads. I use a lock when accessing these threads to ensure the reading on CV thread doesn't happen as the same time a write is occurring on the image acquisition thread. Below are the code snippets of each piece.
Mat Temp=Mat(Instance->Frame->rows,Instance->Frame->cols,CV_8UC4);
CLEyeCameraGetFrame(*(Instance->CameraInstance),Temp.data);
//image acquisition thread
Instance->imgMtx->lock();
*(Instance->Frame) = Temp.clone();
Instance->imgMtx->unlock();
//image copy used in the CV thread
imgMtx.lock();
frame = Frame.clone();
imgMtx.unlock();
//declared variables
int LowH, HighH, LowS, HighS, LowV, HighV;
cv::Mat binary, hsv;
//image conversion in the CV code
cv::cvtColor(frame, hsv, CV_BGR2HSV);
//inRange call where it crashes
inRange(hsv, cv::Scalar(LowH, LowS, LowV), cv::Scalar(HighH, HighS, HighV), binary);
Where it crashes, it looks like the inRange function was trying to determine the type of arguments is was getting. Below is the relevant code in OpenCV
void cv::inRange(InputArray _src, InputArray _lowerb,
InputArray _upperb, OutputArray _dst)
{
int skind = _src.kind(), lkind = _lowerb.kind(), ukind = _upperb.kind();
It crashes on something in that last line. I can't really tell what is breaking here but opencv just keeps calling the type function until it hits a stackoverflow. I have gone through and at every point made sure the images can be displayed using imshow. I realize that's not the entire picture but I don't know what else to check. Any help would be appreciated.
EDIT: best I can tell, it hits the infinite trying to resolve the type of the Scalar. Below is the code for that function inside opencv.
int _InputArray::type(int i) const
{
int k = kind();
if( k == MAT )
return ((const Mat*)obj)->type();
if( k == EXPR )
return ((const MatExpr*)obj)->type();
if( k == MATX || k == STD_VECTOR || k == STD_VECTOR_VECTOR )
return CV_MAT_TYPE(flags);
if( k == NONE )
return -1;
if( k == STD_VECTOR_MAT )
{
const vector<Mat>& vv = *(const vector<Mat>*)obj;
CV_Assert( i < (int)vv.size() );
return vv[i >= 0 ? i : 0].type();
}
if( k == OPENGL_BUFFER )
return ((const ogl::Buffer*)obj)->type();
CV_Assert( k == GPU_MAT );
//if( k == GPU_MAT )
return ((const gpu::GpuMat*)obj)->type();
}

Related

Strange error with cuda::reprojectImageTo3D and OpenCV 3.0

Hello I am trying to get a Mat with the XYZ cordenates from a disparity map that i have already generated with a cuda::StereoBM object, this is a screenshot of the disparity map I created "disparity map"
but when I add the function cuda::reprojectImageTo3D(d_disp,d_xyz,d_Q,3,Stream); I get an error I can't understand, if I coment this line of code no error is shown but the moment I try to use this function I get the next error window
and the error code is:
OpenCV Error: The function/feature is not implemented (You should explicitly call download method for cuda::GpuMat object) in cv::_InputArray::getMat_, file C:\opencv\sources\modules\core\src\matrix.cpp, line 1211
bellow is my code, note that most of my Mats are declared as global , so their declarations are not in this slot function
void MainWindow::on_botonMostrarDispSGBMCUDA_clicked()
{
if(map_l1.empty()&&map_l2.empty()&&map_r1.empty() &&map_r2.empty()){
FileStorage fs(xmlname,FileStorage::READ);
fs["map_l1"]>>map_l1;
fs["map_l2"]>>map_l2;
fs["map_r1"]>>map_r1;
fs["map_r2"]>>map_r2;
fs["Q"] >> Q;
fs.release();
}
Mat img1, img2;
img1 = imread("C:\\Users\\Diego\\Pictures\\calibstereo\\camleft\\photoleft_1.jpg");
img2 = imread("C:\\Users\\Diego\\Pictures\\calibstereo\\camright\\photoright_1.jpg");
cvtColor(img1, frameLGray, CV_BGR2GRAY);
cvtColor(img2, frameRGray, CV_BGR2GRAY);
remap(frameLGray,framel_rect,map_l1,map_l2,INTER_LINEAR);
remap(frameRGray,framer_rect,map_r1,map_r2,INTER_LINEAR);
bool ok1, ok2, ok3, ok4, ok5, ok6 , ok7,ok8,ok9,ok10 ;
int preFilterSize = ui->editPrefilterSizeBM->text().toInt(&ok1);
int prefilterCap = ui->editFilterCapBM->text().toInt(&ok2);
int blockSize = ui->editBlockSBM->text().toInt(&ok3);
int minDisp = ui->editMinDispBM->text().toInt(&ok4);
int numDisp = ui->editNumDispBM->text().toInt(&ok5);
int texturethresh = ui->editTextureBM->text().toInt(&ok6);
int uniqueness = ui->editUniquenesBM->text().toInt(&ok7);
int speckleWindow = ui->editSpeckleWindowBM->text().toInt(&ok8);
int speckleRange = ui->editSpeckleRangeBM->text().toInt(&ok9);
int maxDiff = ui->editMaxDiffBM->text().toInt(&ok10);
if (!ok1 || !ok2 || !ok3 || !ok4 || !ok5 || !ok6 || !ok7 || !ok8 || !ok9 || !ok10){
QMessageBox messageBox;
messageBox.setIconPixmap(QPixmap(":/icon.svg"));
messageBox.setText("One of your entries is not a valid number.");
messageBox.setWindowTitle("Error");
messageBox.exec();
}
d_left.upload(framel_rect);
d_right.upload(framer_rect);
Ptr<cuda::StereoBM> CUSBM;
CUSBM = cuda::createStereoBM(64, 5);
CUSBM->setPreFilterSize(preFilterSize);
CUSBM->setPreFilterCap(prefilterCap);
CUSBM->setBlockSize(blockSize);
CUSBM->setMinDisparity(minDisp);
CUSBM->setNumDisparities(numDisp);
CUSBM->setTextureThreshold(texturethresh);
CUSBM->setUniquenessRatio(uniqueness);
CUSBM->setSpeckleWindowSize(speckleWindow);
CUSBM->setSpeckleRange(speckleRange);
CUSBM->setDisp12MaxDiff(maxDiff);
cuda::Stream myStream = cuda::Stream::Stream();
CUSBM->compute(d_left, d_right, d_disp, myStream);
if(!d_disp.empty()){
cuda::GpuMat d_xyz(d_disp.size(), CV_32FC4);
cuda::GpuMat d_Q(4,4,CV_32F);
d_Q.upload(Q);
cuda::Stream mySecondStream = cuda::Stream::Stream();
cuda::reprojectImageTo3D(d_disp,d_xyz,d_Q,3,mySecondStream);
d_xyz.download(xyz);
}
cuda::drawColorDisp(d_disp, d_disp,numDisp); //64
d_disp.download(disp);
namedWindow("left");
namedWindow("right");
namedWindow("disp");
imshow("left", framel_rect);
imshow("right", framer_rect);
imshow("disp", disp);
//imshow("xyz", xyz);
waitKey(0);
cv::destroyAllWindows();
}
I am using Windows 8.1, QtCreator as my IDE, OpenCV 3.0.0 with cuda 7.0
Are you sure that d_disp in line cuda::reprojectImageTo3D(d_disp,d_xyz,d_Q,3,mySecondStream); is a GpuMat
Your problem may from cv::Mat class, it has no overloaded assignment operator or copy constructor which takes argument of type cv::gpu::GpuMat, so you are not able to mix them in the code.
I checked the the argument list for the function from cudastereo/src/util.cpp.
The function seems to take d_disp and d_xyz as cuda::GpuMat, but the Q matrix needs to be a cv::Mat.
cuda::reprojectImageTo3D(d_disp,d_xyz,Q,3,mySecondStream); seems to work fine.
Sorry for the late answer.
The problem lies in cuda::drawColorDisp(d_disp, d_disp,numDisp); //64. You are supplying same source and destination GpuMat files, which should not be the case. You may like to have a look here for the detailed constructor arguments for drawColorDisp.

my application crashes at KNearest::find_nearest

I want to implement a OCR feature.
I have collected some samples and i want to use K-Nearest to implement it.
So, i use the below code to load data and initialize KNearest
KNearest knn = new KNearest;
Mat mData, mClass;
for (int i = 0; i <= 9; ++i)
{
Mat mImage = imread( FILENAME ); // the filename format is '%d.bmp', presenting a 15x15 image
Mat mFloat;
if (mImage.empty()) break; // if the file doesn't exist
mImage.convertTo(mFloat, CV_32FC1);
mData.push_back(mFloat.reshape(1, 1));
mClass.push_back( '0' + i );
}
knn->train(mData, mClass);
Then, i call the code to find best result
for (vector<Mat>::iterator it = charset.begin(); it != charset.end(); ++it)
{
Mat mFloat;
it->convertTo(mFloat, CV_32FC1); // 'it' presents a 15x15 gray image
float result = knn->find_nearest(mFloat.reshape(1, 1), knn->get_max_k());
}
But, my application crashes at find_nearest.
Anyone could help me?
I seemed to find the problem...
My sample image is a converted gray image by cvtColor, but my input image isn't.
After i add
cvtColor(mImage, mImage, COLOR_BGR2GRAY);
between
if (mImage.empty()) break;
mImage.convertTo(mFloat, CV_32FC1);
find_nearest() return a value and my application is fine.

OpenCV - GrabCut with custom foreground/background models

I want to use the GrabCut algorithm implemented on OpenCV.
As shown in the documentation this is the function signature:
void grabCut(
InputArray img,
InputOutputArray mask,
Rect rect,
InputOutputArray bgdModel, // *
InputOutputArray fgdModel, // *
int iterCount,
int mode=GC_EVAL)
The mode param, indicates how to initialize the algorithm, either with the rect (a rectangle bounding box) or with the mask (a matrix whose values correspond to user paintings of the foreground/background regions.
I already have the color models for both the FG and the BG, so ideally I shouldn’t need to provide a mask or a rectangle, but use those models as an initialization (I want to prevent OpenCV to compute new models and use mine instead). I see that bgdModel and fgdModel parameters somehow contain this model information. Unfortunatelly, the documentation does not provide any details on how the model information is stored there.
Is it possible to populate those models whith existing data and run the method with mode=GC_EVAL?, if so, how do I need to encode the models?
In opencv/sources/modules/imgproc/src/grabcut.cpp you can have a look how the models (GMMs) are encoded:
GMM::GMM( Mat& _model )
{
const int modelSize = 3/*mean*/ + 9/*covariance*/ + 1/*component weight*/;
if( _model.empty() )
{
_model.create( 1, modelSize*componentsCount, CV_64FC1 );
_model.setTo(Scalar(0));
}
else if( (_model.type() != CV_64FC1) || (_model.rows != 1) || (_model.cols != modelSize*componentsCount) )
CV_Error( CV_StsBadArg, "_model must have CV_64FC1 type, rows == 1 and cols == 13*componentsCount" );
model = _model;
coefs = model.ptr<double>(0);
mean = coefs + componentsCount;
cov = mean + 3*componentsCount;
for( int ci = 0; ci < componentsCount; ci++ )
if( coefs[ci] > 0 )
calcInverseCovAndDeterm( ci );
}
So you need for every model a cv::Mat of 1 x 65 doubles (componentsCount equals 5). There are 3 means per component because its computing in RGB colorspace. Using GC_EVAL would indeed leave the models intact but I never tried it with pre-computed models.
Even I had a similar problem. This is how I solved it.
I edited the GC_EVAL condition in the grabcut source code to this -
if( mode == GC_EVAL )
{ checkMask( img, mask );
for( int i = 0; i < iterCount; i++ )
{
GCGraph<double> graph;
assignGMMsComponents( img, mask, bgdGMM, fgdGMM, compIdxs );
constructGCGraph(img, mask, bgdGMM, fgdGMM, lambda, leftW, upleftW, upW, uprightW, graph );
estimateSegmentation( graph, mask );
return;
}
}
Notice that the function learnGMMs is not called here. This is done because the Foreground and Background GMMs are precomputed.
You can save models in a .xml file using the following code snippet.
FileStorage fs("mymodels.xml", FileStorage::WRITE);
fs << "BgdModel" << bgdModel;
fs << "FgdModel" << fgdModel;
fs.release();
You can retrieve the models using the following code.
FileStorage fs1("mymodels.xml", FileStorage::READ);
fs1["BgdModel"] >> bgdModel1;
fs1["FgdModel"] >> fgdModel1;
This worked for me.

Template matching with Orb : error with batchDistance method

I'm trying to detect whether a template image (logo) is present in a pdf document. The document can be either a scan encapsulated in a pdf or a "pure" pdf document, but this is completely random.
First, I convert the pdf document to a png image using ImageMagick's convert tool, then I cut the output images in half because they're so big, and after that I try to match a logo from a database with any of the shapes present in the half-cut image.
To do so, I use an Orb Feature Detector with an Orb Descriptor, and a RobustMatcher (sort of improved BruteForce matcher, source code available here). Here is a snippet of code from my adaptation of it :
// Read input images
Mat image1 = imread(argv[1], CV_LOAD_IMAGE_GRAYSCALE);
Mat image2 = imread(argv[2], CV_LOAD_IMAGE_GRAYSCALE);
if (!image1.data || !image2.data) {
std::cout << " --(!) Error reading images " << std::endl;
exit(1);
}
// Setting up values for ORB Detector
int nfeatures = 800;
//float scaleFactor = 1.10;
int nlevels = 8;
int edgeThreshold = 12;
int firstLevel = 0;
int WTA_K = 2;
int scoreType = 0;
int patchSize = 31;
// Prepare the matcher
RobustMatcher rmatcher;
rmatcher.setConfidenceLevel(0.98);
rmatcher.setMinDistanceToEpipolar(1.0);
rmatcher.setRatio(0.80f);
cv::Ptr<cv::FeatureDetector> pfd = new cv::OrbFeatureDetector(nfeatures, scaleFactor, nlevels, edgeThreshold, firstLevel, WTA_K, scoreType, patchSize);
rmatcher.setFeatureDetector(pfd);
cv::Ptr<cv::DescriptorExtractor> pde = new cv::OrbDescriptorExtractor();
rmatcher.setDescriptorExtractor(pde);
// Match the two images
std::vector<cv::DMatch> matches;
std::vector<cv::KeyPoint> keypoints1, keypoints2;
cv::Mat fundemental = rmatcher.match(image1, image2, matches, keypoints1, keypoints2);
// If nothing could be matched, stop here
if(matches.size() < 4){
exit(2);
}
The code works great on some examples that I chose carefully, with a highly-recognizable logo and a clean image, with certain proportions... etc. But when I try to apply the process to random pdf files, I start to get this error from OpenCV :
OpenCV Error: Assertion failed (type == src2.type() && src1.cols == src2.cols && (type == CV_32F || type == CV_8U)) in batchDistance, file /home/das/Downloads/opencv-2.4.5/modules/core/src/stat.cpp, line 1797
terminate called after throwing an instance of 'cv::Exception'
what(): /home/das/Downloads/opencv-2.4.5/modules/core/src/stat.cpp:1797: error: (-215) type == src2.type() && src1.cols == src2.cols && (type == CV_32F || type == CV_8U) in function batchDistance
Aborted (core dumped)
I checked for this error, and it appeared that src1.cols != src2.cols, and a quick fix for it would be to test the condition before trying to match the images. The problem is that I miss a lot of images doing so, and this would be OK only if I were working on a video stream... but I'm not, and the next image has nothing in common with the previous one, and I can't determine whether my logo was present or not in the document.
Here is the code from stat.cpp, lines 1789 to 1826 : (assertion is at the beginning on line 1797)
void cv::batchDistance( InputArray _src1, InputArray _src2,
OutputArray _dist, int dtype, OutputArray _nidx,
int normType, int K, InputArray _mask,
int update, bool crosscheck )
{
Mat src1 = _src1.getMat(), src2 = _src2.getMat(), mask = _mask.getMat();
int type = src1.type();
CV_Assert( type == src2.type() && src1.cols == src2.cols &&
(type == CV_32F || type == CV_8U));
CV_Assert( _nidx.needed() == (K > 0) );
if( dtype == -1 )
{
dtype = normType == NORM_HAMMING || normType == NORM_HAMMING2 ? CV_32S : CV_32F;
}
CV_Assert( (type == CV_8U && dtype == CV_32S) || dtype == CV_32F);
K = std::min(K, src2.rows);
_dist.create(src1.rows, (K > 0 ? K : src2.rows), dtype);
Mat dist = _dist.getMat(), nidx;
if( _nidx.needed() )
{
_nidx.create(dist.size(), CV_32S);
nidx = _nidx.getMat();
}
if( update == 0 && K > 0 )
{
dist = Scalar::all(dtype == CV_32S ? (double)INT_MAX : (double)FLT_MAX);
nidx = Scalar::all(-1);
}
if( crosscheck )
{
CV_Assert( K == 1 && update == 0 && mask.empty() );
Mat tdist, tidx;
batchDistance(src2, src1, tdist, dtype, tidx, normType, K, mask, 0, false);
So I'm wondering what does this assertion mean ? What are exactly the src1 and src2 files in stat.cpp ? Why do they need to have the same number of columns ?
I tried changing to a Surf detector and extractor, but I still get the error.
If anyone has an idea, do not hesitate to post, I welcome any advice or notice !
Thanks in advance.
EDIT
I have a more precise question now : how do I ensure that src1.cols == src2.cols ? To answer that question, I think I should know what are the transformations applied to my cv::Mat image1 and image2 before batchDistance(...) is called, in order to find a condition on image1 and image2 which will ensure that src1.cols == src2.cols, so my code would look like this :
// Match the two images
std::vector<cv::DMatch> matches;
std::vector<cv::KeyPoint> keypoints1, keypoints2;
if( CONDITION_ON_IMAGE1&IMAGE2_TO_ENSURE_SRC1.COLS==SRC2.COLS ){
cv::Mat fundemental = rmatcher.match(image1, image2, matches, keypoints1, keypoints2);
}
To get rid of errors, You can play with copying and pasting the images into empty one of required size, but this is only a quick and dirty solution for the assertion.
To make detector and descriptor work properly, You might have to get to know how it works. Maybe then You will be able to get images that will work. After reading this article, it looks that ORB will have problems with scaling (they mention it in the conclusion section). This means You will need to find a workaround for it (like image pyramids, or another way to check the image at multiple scales) or use another extractor and descriptor, which is scale and rotation invariant.

Getting the difference between two frames in opencv

I'm trying to get the the difference between two cv::Mat frames in OpenCv. So here is what I tried,
#include<opencv2\opencv.hpp>
#include<opencv2\calib3d\calib3d.hpp>
#include<opencv2\core\core.hpp>
#include <opencv2\highgui\highgui.hpp>
int main ()
{
cv::VideoCapture cap(0);
cv::Mat frame, frame1,frame2;
int key=0;
while(key!=27){
cap >> frame;
if(key=='c'){
frame1 = frame;
key = 0;
}
if(key =='x'){
cv::absdiff(frame, frame1, frame2); // I also tried frame2= (frame -frame1)*255;
cv::imshow("difference ",frame2);
key =0;
}
cv::imshow("stream",frame);
key = cv::waitKey(10);
}
}
the result is always the same a 0 Matrix, any idea what I'm doing wrong here?
thanks in advance for your help.
Mat objects are pointer typed. After setting frame1 to frame directly using frame1 = frame, both matrices show the same point and same frame also. You have to copy frame value using "copyTo" method of Mat.
OpenCV Matrixes use pointers internally
The documentation of the Mat type states:
Mat is basically a class with two data parts: the matrix header and a pointer to the matrix containing the pixel values.
[...]
Whenever somebody copies a header of a Mat object, a counter is increased for the matrix. Whenever a header is cleaned this counter is decreased. When the counter reaches zero the matrix too is freed. Sometimes you will want to copy the matrix itself too, so OpenCV provides the clone() and copyTo() functions.
cv::Mat F = A.clone();
cv::Mat G;
A.copyTo(G);
OpenCV overloads the affectation operator on cv::Mat objects so that the line mat1 = mat2 only affects the pointer to the data in mat1 (that points to the same data as mat2). This avoids time consuming copies of all the image data.
If you want to save the data of a matrix, you have to write mat1 = mat2.clone() or mat2.copyTo(mat1).
I was looking for a similar program and I came across your post, here is a sample I have written for frameDifferencing, hope this helps, the below function will give you the difference between two frames
/** #function differenceFrame */
Mat differenceFrame( Mat prev_frame, Mat curr_frame )
{
Mat image = prev_frame.clone();
printf("frame rows %d Cols %d\n" , image.rows, image.cols);
for (int rows = 0; rows < image.rows; rows++)
{
for (int cols = 0; cols < image.cols; cols++)
{
/* printf("BGR value %lf %lf %lf\n" , abs(prev_frame.at<cv::Vec3b>(rows,cols)[0] -
curr_frame.at<cv::Vec3b>(rows,cols)[0]),
abs(prev_frame.at<cv::Vec3b>(rows,cols)[1] -
curr_frame.at<cv::Vec3b>(rows,cols)[0]),
abs(prev_frame.at<cv::Vec3b>(rows,cols)[2] -
curr_frame.at<cv::Vec3b>(rows,cols)[0]));
*/
image.at<cv::Vec3b>(rows,cols)[0] = abs(prev_frame.at<cv::Vec3b>(rows,cols)[0] -
curr_frame.at<cv::Vec3b>(rows,cols)[0]);
image.at<cv::Vec3b>(rows,cols)[1] = abs(prev_frame.at<cv::Vec3b>(rows,cols)[1] -
curr_frame.at<cv::Vec3b>(rows,cols)[1]);
image.at<cv::Vec3b>(rows,cols)[2] = abs(prev_frame.at<cv::Vec3b>(rows,cols)[2] -
curr_frame.at<cv::Vec3b>(rows,cols)[2]);
}
}
return image;
}