I'm trying to draw a couple of circles with OpenCV 3 onto an image which is obtained from the Kinect v2 sensor.
There seems to be a strange bug with cv::circle or I don't understand how the function works. Let's look at some code:
if(kinectDataManager.IsColorStreamEnabled())
{
cv::Mat colorFrameMat = kinectDataManager.GetFrame().GetColorFrame()
cv::imshow("Color", colorFrameMat)
}
This code works perfectly fine, and using the ImageWatch Visual Studio Plugin for inspecting OpenCV images, I can see that the colorFrameMat matrix is not corrupted.
Let's look at some more code:
if(kinectDataManager.IsColorStreamEnabled())
{
cv::Mat colorFrameMat = kinectDataManager.GetFrame().GetColorFrame()
int radius = 2;
int y = 1068;
for (int x = 0; x < 1920; ++x)
{
cv::circle(colorFrameMat, cv::Point(x,y), radius, cv::Scalar(255, 0, 0), -1, CV_AA);
}
cv::imshow("Color", colorFrameMat)
}
After the loop execution has finished, the ImageWatch plugin reveals that the last rows of the image are missing. Strangely, the program still executes. However, for different values of y, the program crashes due to access violations, e.g. for y = 1067, the program crashes for x = 1917. For y = 1069, it crashes at x = 988.
Does anyone have an idea what the issue might be?
EDIT:
The ImageWatch plugin of course reveals that the last rows are missing, as circles are drawn at these positions from left to right, sorry for the mistake!!
EDIT2:
After storing one frame and reading it in, the cv::circle method with the identical code works fine:
cv::Mat test = cv::imread("test.jpg", CV_LOAD_IMAGE_COLOR);
cv::namedWindow("test", CV_WINDOW_NORMAL);
int radius = 10;
int y = 1067;
for (int x = 0; x < 1920; ++x)
{
cv::circle(test, cv::Point(x, y), radius, cv::Scalar(0, 0, 255, 255), -1, CV_AA);
}
cv::imshow("test", test);
cv::waitKey(0);
The Kinect SDK provides only functionality to read a 4-channel image (i.e. RGBA), however, the cv::circle functions seems to crash in a strange way for these kind of images. By dropping the alpha channel with a call to cvtImage, I could resolve the issue.
Since you say the image looks fine before the cv::circle call it is possible that GetColorFrame() returns data that changes while the loop is running.
Try:
a GetColorFrame().clone() to see if this fixes the issue, or
change the way GetColorFrame works.
Related
I am working with the arm compute library link to convert an opencv application to a more efficient code base.
I would like to import data from an opencv mat, which I've done successfully by doing this.
arm_compute::Image matACL;
matACL.allocator()->init(arm_compute::TensorInfo(mat.cols, mat.rows, arm_compute::Format::U8)); // Initialise tensor's dimensions
matACL.allocator()->import_memory(arm_compute::Memory(mat.data)); //Allocate the image without any padding.
//matACL.allocator()->import_memory(arm_compute::Memory(new cvMatData(mat.data)));
Beware the versions 18.05 and above of the ACL need an implemented memory interface which I have created a gist for. That's the commented line above.
I can run different operations on the image (threshold or gauss for example) and I can see the correct output in an opencv window, but whenever I use the canny edge detector I get a messed up output image. I have issued on github a while ago, but they couldn't find a solution either.
I have implemented the canny edge neon like it is done in the NECannyEdge.cpp file to better understand what is happening. I copy the data of the result into an opencv Mat and preserve the pointer to it like that.
This is how I convert the result back to an OpenCV Mat:
ptr = (unsigned char*)malloc(mat.cols*mat.rows*sizeof(unsigned char));
for(unsigned int z = 0 ; z < 0 ; ++z)
{
for (unsigned int y = 0; y < mat.rows; ++y)
{
memcpy(ptr + z * (mat.cols * mat.rows) + y * mat.cols, matACL.buffer() +
matACL.info()->offset_element_in_bytes(Coordinates(0, y, z)), mat.cols *
sizeof(unsigned char));
}
}
and an alternative:
Window output_window;
output_window.use_tensor_dimensions(shape, Window::DimY);
Iterator output_it(&matACL, output_window);
execute_window_loop(output_window,
[&](const Coordinates & id)
{
memcpy(ptr + id.z() * (mat.cols * mat.rows) + id.y() * mat.cols, output_it.ptr(), mat.cols * sizeof(unsigned char));
}, output_it);
The image sometimes showes a correct canny edge result but most of the time it shows random maybe unfinished data.
I checked if it might be a race condition but the implementation should be single threaded and I can't figure out where the problem is. Does anyone have an idea?
How can I successfully use the data from an opencv image to use in the canny edge detector of the arm compute library? Maybe there is some steps during the import that I missed?
Thanks, Greetings
I found where I was going wrong and developed this function, which creates an OpenCV Mat from an ACL Image:
void ACLImageToMat(arm_compute::Image &aCLImage, cv::Mat &cVImage, std::unique_ptr<uint8_t[]> &cVImageDataPtr)
{
size_t width = aCLImage.info()->valid_region().shape.x();
size_t height = aCLImage.info()->valid_region().shape.y();
cVImageDataPtr = std::make_unique < uint8_t[]>(width*height);
auto ptr_src = aCLImage.buffer();
arm_compute::Window input_window;
input_window.use_tensor_dimensions(aCLImage.info()->tensor_shape());
arm_compute::Iterator input_it(&aCLImage, input_window);
int counter = 0;
arm_compute::execute_window_loop(input_window,
[&](const arm_compute::Coordinates & id)
{
*reinterpret_cast<uint8_t *>(cVImageDataPtr.get() + counter++) = ptr_src[aCLImage.info()->offset_element_in_bytes(id)];
},
input_it);
cVImage = cv::Mat(cVImage.rows, cVImage.cols, CV_8UC1, cVImageDataPtr.get());
}
To initialize this for Canny I did the following:
arm_compute::Image matACL;
matACL.allocator()->init(arm_compute::TensorInfo(eye.cols, eye.rows, arm_compute::Format::U8));
matACL.allocator()->import_memory(arm_compute::Memory(eye.data));
arm_compute::Image matACLCanny;
matACLCanny.allocator()->init(arm_compute::TensorInfo(eye.cols, eye.rows, arm_compute::Format::U8));
arm_compute::NECannyEdge canny {};
canny.configure(&matACL, &matACLCanny, 300, 150, 3, 1, arm_compute::BorderMode::REPLICATE);
matACLCanny.allocator()->allocate();
canny.run();
The IMPORTANT thing is to call the allocate function of the output image AFTER configuring the canny edge detector. I found this somewhere in the ACL documentation a while ago, but I can't remember where exactly.
I hope this helps someone who stumbles across converting images between the ACL and OpenCV!
I want to plot circles on a image where each previous circle is deleted on the image before the next circle is drawn.
I have to following configuration:
I have several picture (let says 10)
For each picture I test several pixel for some condition (let say 50 pixels).
For each pixel I'm testing (or working on) I want to draw a circle at that pixel for visualization purpose (for me to visualize that pixel).
To summarize I have 2 for loop, one looping over the 10 images and the other looping over the 50 pixels.
I done the following (see code above). The circles are correctly drawn but when the next circle is drawn, the previous circle is still visible (at the end all circle are drawn on the same image) but what I want to have is (after a circle was drawn) to close the picture (or window) somehow and reopen a new one and plot the next circle on it and so on
for(int imgID=0; imgID < numbImgs; imgID++)
{
cv::Mat colorImg = imgVector[imgID];
for(int pixelID=0; pixelID < numPixelsToBeTested; pixelID++)
{
some_pixel = ... //some pixel
x = some_pixel(0); y = some_pixel(1);
cv::Mat colorImg2 = colorImg; //redefine the image for each pixel
cv::circle(colorImg2, cv::Point(x,y),5, cv::Scalar(0,0,255),1, cv::LINE_8, 0);
// creating a new window each time
cv::namedWindow("Display", CV_WINDOW_AUTOSIZE );
cv::imshow("Display", colorImg2);
cv::waitKey(0);
cv::destroyWindow("Display");
}
}
What is wrong in my code?
Thanks guys
cv::circle() manipulates the input image within the API call, so what you need to do is to create a clone of the original image, draw circles on the cloned image and at each iteration swap the cloned image with original image.
It is also a good idea to break your program into smaller methods, making the code more readable and easy to understand, Following code may give you a starting point.
void visualizePoints(cv::Mat mat) {
cv::Mat debugMat = mat.clone();
// Dummy set of points, to be replace with the 50 points youo are using.
std::vector<cv::Point> points = {cv::Point(30, 30), cv::Point(30, 100), cv::Point(100, 30), cv::Point(100, 100)};
for (cv::Point p:points) {
cv::circle(debugMat, p, 5, cv::Scalar(0, 0, 255), 1, cv::LINE_8, 0);
cv::imshow("Display", debugMat);
cv::waitKey(800);
debugMat = mat.clone();
}
}
int main() {
std::vector<std::string> imagePaths = {"path/img1.png", "path/img2.png", "path/img3.png"};
cv::namedWindow("Display", CV_WINDOW_AUTOSIZE );
for (std::string path:imagePaths) {
cv::Mat img = cv::imread(path);
visualizePoints(img);
}
}
TLDR;
For anyone arriving here whilst trying to figure out how to do gaussian blur or grayscale with OpenCL, the final working code is here. Note that in that repo I'm actually running the whole thing inside Docker with GPU access using Nvidia's Docker wrapper. You can look inside the 'Dockerfile' for the steps that need to be taken to get the code running, or just run it using Nvidia-Docker if you have that setup and are running on an Nvidia GPU.
Original Question:
Using the following kernel in an OpenCL image filter application I get the expected result, that is, a returned grayscale version of the input image:
const sampler_t sampler = CLK_NORMALIZED_COORDS_FALSE |
CLK_ADDRESS_CLAMP_TO_EDGE |
CLK_FILTER_NEAREST;
__kernel void process(__read_only image2d_t src,
__write_only image2d_t dst)
{
int x = get_global_id(0);
int y = get_global_id(1);
float4 color;
color = read_imagef(src, sampler, (int2)(x, y));
float gray = (color.x + color.y + color.z) / 3;
write_imagef(dst, (int2)(x,y), (float4)(gray, gray, gray, 0));
}
So far, so good. I then tried to create a kernel that would just copy across the top and left border of the image:
const sampler_t sampler = CLK_NORMALIZED_COORDS_FALSE |
CLK_ADDRESS_CLAMP_TO_EDGE |
CLK_FILTER_NEAREST;
__kernel void process(__read_only image2d_t src,
__write_only image2d_t dst)
{
int x = get_global_id(0);
int y = get_global_id(1);
float4 color;
if (x < 10 || y < 10)
{
color = read_imagef(src, sampler, (int2)(x, y));
write_imagef(dst, (int2)(x,y), (float4)(color.x, color.y, color.z, 0));
}
else
{
write_imagef(dst, (int2)(x,y), (float4)(0,0,0,0));
}
}
The returned image is not what I expected:
I'm loading the input image this way:
// Load an image using the OpenCV library and create an OpenCL
// image out of it
cl::Image2D LoadImage(cl::Context context, char *fileName, int &width, int &height)
{
cv::Mat image = cv::imread(fileName, CV_LOAD_IMAGE_COLOR);
cv::Mat imageRGBA;
width = image.rows;
height = image.cols;
cv::cvtColor(image, imageRGBA, CV_RGB2RGBA);
char *buffer = reinterpret_cast<char *>(imageRGBA.data);
cl::Image2D clImage(context,
CL_MEM_READ_ONLY | CL_MEM_COPY_HOST_PTR,
cl::ImageFormat(CL_RGBA, CL_UNORM_INT8),
width,
height,
0,
buffer);
return clImage;
}
The output image:
cl::Image2D imageOutput(context,
CL_MEM_WRITE_ONLY,
cl::ImageFormat(CL_RGBA, CL_UNORM_INT8),
width,
height,
0,
NULL);
The Kernel:
cl::Program program(context, util::loadProgram("border.cl"), true);
cl::make_kernel<cl::Image2D, cl::Image2D> filter(program, "process");
cl::NDRange global(width, height);
filter(cl::EnqueueArgs(queue, global), clImageInput, imageOutput);
Then reading the image back:
cl::size_t<3> origin;
origin[0] = 0; origin[1] = 0, origin[2] = 0;
cl::size_t<3> region;
region[0] = width; region[1] = height; region[2] = 1;
float* oup = new float[width * height];
queue.enqueueReadImage(imageOutput, CL_TRUE, origin, region, 0, 0, oup);
cv::imwrite(filename_out, cv::Mat(width, height, CV_8UC4, oup));
Why is the image being processed the way it is? Only selecting pixels with a y coordinate less than 10 seems to work, but selecting pixels with an x coordinate less than 10 seems to stagger across the image.
if I write a test image using the following line in the kernel:
write_imagef(dst, (int2)(x,y), (float4)((float)x / 512.0f, 0, 0, 0));
I get the following image:
The first strange thing is that the blue channel is being set, not the red. I have no idea why as I am alway loading and saving the image in RGBA order. Secondly, the banding is very unusual, I'm not sure how to interpret this.
If I use the following line in the kernel:
write_imagef(dst, (int2)(x,y), (float4)(0, (float)y / 512.0f, 0, 0));
I get the following image:
This looks the way I would expect.
I can provide more code if necessary but using the grayscale kernel in the exact same harness works perfectly. As does another kernel not listed here which simply copies all the pixels across.
I'm running the code on and Nvidia Geforce 980M with OpenCL 1.2
I'm not seeing anything obvious yet. One strange thing: your image is CL_RGBA, CL_UNORM_INT8 but you're reading it out into an array of floats? How are you displaying it from that? Second, I'm not famliar with your kernel launch technique; what is filter and is it launching with dimension of 2? Regarding the issue you're seeing, I'd suggest using process of elimination to figure out where the problem lies. For example, (1) if you remove the conditional and copy all pixels, do you get the whole image? (2) Instead of writing black where the conditional is false, what if you write a Red channel gradient based on X position and a Green channel gradient based on Y position. Do you get a double gradient? Based on results, continue to divide the problem until you find the cause. It looks a lot like a row pitch issue, perhaps in the display function?
Ok, so the issue was the way I was reading height and width was backwards, i.e.
width = image.rows;
height = image.cols;
Should have been
height = image.rows;
width = image.cols;
With this corrected, the rest of the code can stay the same, except the last line where I save the image to disk, here the values need to be swapped again, i.e.
cv::imwrite(filename_out, cv::Mat(width, height, CV_8UC4, oup));
Needs to change to:
cv::imwrite(filename_out, cv::Mat(height, width, CV_8UC4, oup));
I think this ultimately comes down to the matrix approach to an image where the first coordinate is actually the row number, which is the height and the second coordinate is the column number, which is the width.
The diagnostics #Dithermaster mentioned really helped, as did printing out the assumed width and height, which was ultimately incorrect.
It's interesting that by having both of those errors in the code a pixel for pixel copy worked fine, but once you start to perform actions based on the x,y coordinates you get some really funky results.
I am having some issues with my sobel_y (and sobel_x, but I figure they are having the same issue) filter in that it keeps giving me an image that it basically only black and white. I am having to rewrite this function for a class, so no I cannot use the built-in one, and had it working, minus some minor tweaks because the output image looked a little strange with still being black and white even though it was supposed to be converted back. I figured out how to fix that, and in the process I messed with something and broke it and cannot seem to get it back to working even with the black and white image output only. I keep getting a black image, with some white lines here and there near the top. I have tried changing the Mat grayscale type (third parameter) to all different values, as my professor mentioned in the class that we are using 32 bit floating point images, but that did not help either.
Even though the issue occurs after running the Studentfilter2D, I think it is a problem with the grayscaling of the image, although whenever I debug, it seems to work just fine. This is also because I have 2 other filtering functions I had to write that use Studentfilter2D, and they both give me the expected results. My sobel_y function is shown below:
// Convert the image in bgr to grayscale OK to use the OpenCV function.
// Find the coefficients used by the OpenCV function, and give a link where you found it.
// Note: This student function expects the matrix gray to be preallocated with the same width and
// height, but with 1 channel.
void BGR2Gray(Mat& bgr, Mat& gray)
{
// Y = .299 * R + .587 * G + .114 * B, from http://docs.opencv.org/modules/imgproc/doc/miscellaneous_transformations.html#cvtcolor
// Some extra assistance, for the third parameter for the InputArray, from http://docs.opencv.org/trunk/modules/core/doc/basic_structures.html#inputarray
// Not sure about the fourth parameter, but was just trying it to see if that may be the issue as well
cvtColor(bgr, gray, CV_BGR2GRAY, 1);
return;
}
// Convolve image with kernel - this routine will be called from the other
// subroutines! (gaussian, sobel_x and sobel_y)
// image is single channel. Do not use the OpenCV filter2D!!
// Implementation can be with the .at or similar to the
// basic method found in the Chapter 2 of the OpenCV tutorial in CANVAS,
// or online at the OpenCV documentation here:
// http://docs.opencv.org/doc/tutorials/core/mat-mask-operations/mat-mask operations.html
// In our code the image and the kernel are both floats (so the sample code will need to change)
void Studentfilter2D (Mat& image, Mat& kernel)
{
int kCenterX = kernel.cols / 2;
int kCenterY = kernel.rows / 2;
// Algorithm help from http://www.songho.ca/dsp/convolution/convolution.html
for (int iRows = 0; iRows < image.rows; iRows++)
{
for (int iCols = 0; iCols < image.cols; iCols++)
{
float result = 0.0;
for (int kRows = 0; kRows < kernel.rows; kRows++)
{
// Flip the rows for the convolution
int kRowsFlipped = kernel.rows - 1 - kRows;
for (int kCols = 0; kCols < kernel.cols; kCols++)
{
// Flip the columns for the convolution
int kColsFlipped = kernel.cols - 1 - kCols;
// Indices of shifting around the convolution
int iRowsIndex = iRows + kRows - kCenterY;
int iColsIndex = iCols + kCols - kCenterX;
// Check bounds using the indices
if (iRowsIndex >= 0 && iRowsIndex < image.rows && iColsIndex >= 0 && iColsIndex < image.cols)
{
result += image.at<float>(iRowsIndex, iColsIndex) * kernel.at<float>(kRowsFlipped, kColsFlipped);
}
}
}
image.at<float>(iRows, iCols) = result;
}
}
return;
}
void sobel_y (Mat& image, int)
{
// Note, the filter parameter int is unused.
Mat mask = (Mat_<float>(3, 3) << 1, 2, 1,
0, 0, 0,
-1, -2, -1) / 3;
//Mat grayscale(image.rows, image.cols, CV_32FC1);
BGR2Gray(image, image);
Studentfilter2D(image, mask);
// Here is the documentation on normalize http://docs.opencv.org/modules/core/doc/operations_on_arrays.html#normalize
normalize(image, image, 0, 1, NORM_MINMAX);
cvtColor(image, image, CV_GRAY2BGR);
return;
}
Like I said, I had this working before, just looking for some fresh eyes to look at it and see what I may be missing. I have been looking at this same code so much for the past 4 days that I think I am just missing things. In case anyone is wondering, I have also tried changing the mask values of the filter, but to no avail.
There are two things that are worth mentioning.
The first is that you are not taking proper care of the type of your matrices/images.
The input to Studentfilter2D in sobel_y is an 8-bit grayscale image of type CV_8UC1 meaning that the data is an array of unsigned char.
Your Studentfilter2D function, however, is indexing this input image as though it was of type float. This means it is picking the wrong pixels to work with.
If the above does not immediately solve your problem, you should consider the range of your final derivative image. Since it is a derivative it will no longer be in the range [0, 255]. Instead, it might even contain negative numbers. When you try to visualize this, you will run into problems unless you first normalize your image.
There are built in functions to do this in OpenCV if you look around in the documentation.
I am using OpenCVwith Eclipse.
I need to detect the human skin, so I convert the image to HSV and the I use inRange function to obtain a Mat with the image with the skin in white.
The problem is that now,I need to detect in which components are the white color to modify this pixels in the original frame ( i am changing the skin color with the video camera), but I cant access to the Mat returned in InRange
cvtColor(frame,frame,CV_BGR2HSV);
Mat n;
inRange(frame, Scalar(0, 10, 60), Scalar(20, 150, 255), n);
for(int i=0;i<frame.rows;i++)
{
for(int j=0;j<frame.cols;j++)
{
n.at(&i);
//n(i,j);
}
}
That is the problematic code. When I get to the internal loop, the build fails giving a lot of error refering to the template.
Anyone knows how can I access to this matrix? Is there another way to achieve my objective? Maybe I am complicating the problem.
Thanks for your time.
nothing to do with inRange or such, it's just your Mat access code, that is broken.
Vec3b & hsvPixel = n.at<Vec3b>(i,j);
// hsvPixel[0] = h;
// hsvPixel[1] = s;
// hsvPixel[2] = v;