display opencv cv::Mat image in gtkmm - c++

Question

I want to display a cv::Mat in a Gui written by gtkmm. So I have done a test.
I have a widget Gtk::Image image, and I want to set the image with the following two methods:
// first method, display from file
void displayImage1()
{
Glib::RefPtr<Gdk::Pixbuf> pixbuf = Gdk::Pixbuf::create_from_file("gtk.png");
image.set(pixbuf);
}
// second method, display from cv::Mat
void displayImage2()
{
cv::Mat outImage = cv::imread("gtk.png");
cv::cvtColor(outImage, outImage, CV_BGR2RGB);
Glib::RefPtr<Gdk::Pixbuf> pixbuf = Gdk::Pixbuf::create_from_data(outImage.data, Gdk::COLORSPACE_RGB,false, 8, outImage.cols, outImage.rows, outImage.step);
image.set(pixbuf);
}
The first method works well.
However, the second method doesn't work well, I got a destroyed image on the screen as shown in the picture.
If I set the has_alpha parameter to true, the result is also strange (see pic. below).
Similar tests were done by using Gtk::DrawingArea. Different IDEs are used (but all g++ compiler under linux). All same results.
Update:
I tested lots of images. Sometimes the images are broken, sometimes the programs crashed with
The program has unexpectedly finished.

Answer 1

Usually, this kind of "broken" image trigger a warning in my head: "wrong rawstride !". The rawstride in Gdk::Pixbuf in the length in bytes of a line of data. That's because you may have some byte-alignment constraints, so there may be some padding at the end of one line.
I checked what this step argument was, and yes, that's the same in OpenCV as the rawstride in Gdk::Pixbuf. Until I realized outImage.step is a cv:MatStep object, while Gdk::Pixbuf::create_from_data expects an int. I think you're supposed to use outImage.step[0] instead.
Please read https://docs.opencv.org/2.4/modules/core/doc/basic_structures.html#mat

Answer 2

here we go:
auto lenna = imread("Lenna.png");
Image image;
cvtColor(lenna, lenna, COLOR_BGR2RGB);
auto size = lenna.size();
auto img = Gdk::Pixbuf::create_from_data(lenna.data, Gdk::COLORSPACE_RGB, lenna.channels() == 4, 8, size.width, size.height, (int) lenna.step);
image.set(img);

Answer 3

This is what I did and it show good result on image showing
cvtColor(resize_image, resize_image, COLOR_BGR2RGB);
Pixbuf = Gdk::Pixbuf::create_from_data(resize_image.data, Gdk::COLORSPACE_RGB, false, 8, resize_image.cols, resize_image.rows, resize_image.step);

Answer 4

So I tested this (add scale_simple) with success for me:
From: http://orobotp.blogspot.com/2014/01/opencv-with-gtkmm3.html
Version: Gtkmm 3.22.2-2, OpenCV 4.4.0-dev, g++ 7.5.0
void displayImage2()
{
cv::Mat outImage;
outImage = cv::imread("gtk.png");
cv::cvtColor(outImage, outImage, cv::COLOR_RGB2BGR);
Glib::RefPtr<Gdk::Pixbuf> pixbuf = Gdk::Pixbuf::create_from_data(outImage.data, Gdk::COLORSPACE_RGB,false, 8, outImage.cols, outImage.rows, outImage.step)->scale_simple( outImage.cols, outImage.rows, Gdk::INTERP_BILINEAR );
image.set(pixbuf);
}

Answer 5

IMO, as already suggested by #Miki in the comments, this is just a lifetime issue.
I had the very same problem with similar code:
{
cv::Mat rgb;
cv::cvtColor(src, rgb, cv::COLOR_GRAY2RGB);
pixbuf = gdk_pixbuf_new_from_data(rgb.data,
GDK_COLORSPACE_RGB, FALSE, 8,
rgb.cols, rgb.rows, rgb.step,
NULL, NULL);
}
The above snippet simply does not work (or works intermittently) because, quoting the gdk_pixbuf_new_from_data documentation, "the data is owned by the caller of the function".
The problem is at the time the image is rendered, rgb has been destroyed. Adding a rgb.addref() just before the pixbuf assignment resolves the issue, although introducing a memory leak.
One solution would be to leverage the destroy callback to unreference the Mat object, e.g.:
static void
unref_mat(guchar *data, gpointer user_data)
{
((cv::Mat *) user_data)->release();
}
{
cv::Mat rgb;
cv::cvtColor(src, rgb, cv::COLOR_GRAY2RGB);
rgb.addref()
pixbuf = gdk_pixbuf_new_from_data(rgb.data,
GDK_COLORSPACE_RGB, FALSE, 8,
rgb.cols, rgb.rows, rgb.step,
unref_mat, &rgb);
}

Answer 6

try add a reference to outimage, like outimage.addref() . all my problems were related to that. the source image being de-referenced before the gdk_pixbuf_new_from_data get a chance to map it. leading to segfaults, corruption and so on. just be sure to release it later, or use the callback provided with gdk_pixbuf_new_from_data

Answer 7

For Gtkmm-2.4 and OpenCv 4.6 check https://onthim.blogspot.com/2015/10/using-opencv-in-gtk-applications.html and https://developer-old.gnome.org/gtkmm-tutorial/2.24/sec-draw-images.html.es
Mat frame;
frame = imread("gtk.png");
cv::cvtColor (frame, frame, COLOR_BGR2RGB);//COLOR_BGR2GRAY
Glib::RefPtr<Gdk::Pixbuf> image = Gdk::Pixbuf::create_from_data(frame.data,Gdk::COLORSPACE_RGB, false, 8, frame.cols, frame.rows, frame.step);
image->render_to_drawable(get_window(), get_style()->get_black_gc(),0, 0, 0, 0, image->get_width(), image->get_height(), // draw the whole image (from 0,0 to the full width,height) at 100,80 in the window
Gdk::RGB_DITHER_NONE, 0, 0);