I am really new to Flutter and C++. So I am struggling with this one. I want to do some transformation to an image through ffi->c++ (opencv). Here's my code on the Flutter side:
class Processvalue extends Struct {
Pointer<Utf8> image;
factory Processvalue.allocate(Pointer image) =>
malloc<Processvalue>().ref
..image = image;
}
I am basically accepting a pointer back from C++ side. Then I call C++ native function like so:
...
final processImage = nativeEdgeDetection
.lookup<NativeFunction<ProcessImageFunction>>("process_image")
.asFunction<ProcessImageNativeFunction>();
...
On the C++ side, in the process_image function I am using the following:
...
Mat dst = Mat::zeros(maxHeight, maxWidth, CV_8UC1);
Mat transformation_matrix = getPerspectiveTransform(img_pts, dst_pts);
warpPerspective(img, dst, transformation_matrix, dst.size());
uchar * arr = dst.isContinuous()? dst.data: dst.clone().data;
Basically, I am sending image path from Flutter to C++ through ffi package. And in C++ grab the image from this file and create opencv Mat variable dst, which will hold the new transformed image. Then I convert this Mat variable to uchar * arr, which I am returning back to Flutter (Pointer to arr). However in Flutter I am not able to convert this back to image data (Uint8List for example).
I have seen people use cv::imwrite(path, dst); to save transformation directly to the same file. But for me I want to keep the original file and only want to get the new image as data back in Flutter and show it through MemoryImage(...Uint8List variable...).
How do I send image data back to Flutter from C++? At the end I would like to have a Uint8List. Is uchar * arr even a good option for this?
I have tried the following but it gives me an invalid image data error:
int imageArrLength = _result.ref.image.length;
^- Pointer back from C++
Uint8List image = _result.ref.image.cast<Uint8>().asTypedList(imageArrLength);
Related
Please let me know if this question is too broad, but I am trying to learn some c++ so I thought it would be a good idea to try and recreate some opencv functions.
I am still grabbing the frames or reading the image with opencv's API, but I then want to feed the cv::Mat into my custom function(s), where I modify its data and return it to display it. (For example a function to blur the image, where I pass the original Mat to a padding function, then the output of that to a fn that convolves the padded image with the blurring kernel, and returns the Mat to cv for displaying)
I am a little confused as to what the best (or right) way to do this is. OpenCV functions use a function argument as the return matrix ( cv_foo(cv::Mat src_frame, cv::Mat dst_frame) ) but I am not entirely clear how this works, so I have tried a more familiar approach, something like
cv::Mat my_foo(cv::Mat src_frame) {
// do processing on src_frame data
return dst_frame;
}
where to access the data from src_frame I use uchar* framePtr = frame.data; and to create the dst_frame I followed this suggestion
cv::Mat dst_frame = cv::Mat(n_rows, n_cols, CV_8UC3);
memcpy(dst_frame.data, &new_data_array, sizeof(new_data_array));
I have however encountered various segmentation faults that I find hard to debug, as it seems they occur almost at random (could this be due to the way I am handling the memory management with frame.data or something like that?).
So to come back to my original question, what is the best way to access, modify and pass the data from a cv::Mat in the most consistent way?
I think what would make the most intuitive sense to me (coming from numpy) would be to extract the data array from the original Mat, use that throughout my processing and then repackage it into a Mat before displaying, which would also allow me to feed any custom array into the processing without having to turn it into a Mat, but I am not sure how to best do that (or if it is the right approach).
Thank you!
EDIT:
I will try to highlight the main bug in my code.
One of the functions I am trying to replicate is a conversion from bgr to greyscale, my code looks like this
cv::Mat bgr_to_greyscale(cv::Mat& frame){
int n_rows = frame.rows;
int n_cols = frame.cols;
uchar* framePtr = frame.data;
int channels = frame.channels();
uchar grey_array[n_rows*n_cols];
for(int i=0; i<n_rows; i++){
for(int j=0; j<n_cols; j++){
uchar pixel_b = framePtr[i*n_cols*channels + j*channels];
uchar pixel_g = framePtr[i*n_cols*channels + j*channels + 1];
uchar pixel_r = framePtr[i*n_cols*channels + j*channels + 2];
uchar pixel_grey = 0.299*pixel_r + 0.587*pixel_g + 0.144*pixel_b;
grey_array[i*n_cols + j] = pixel_grey;
}
}
cv::Mat dst_frame = cv::Mat(n_rows, n_cols, CV_8UC1, &grey_array);
return dst_frame;
}
however when I display the result of this function on a sample image I get this result: the bottom part of the image looks like random noise, how can I fix this? what exactly is going wrong in my code?
Thank you!
This question is too broad to answer in any detail, but generally a cv::Mat is a wrapper around the image data akin to the way an std::vector<int> is a wrapper around a dynamically allocated array of int values or an std::string is a wrapper around a dynamically allocated array of characters with one exception: a cv::Mat will not perform a deep copy of the image data on assignment or usage of the copy constructor.
std::vector<int> b = { 1, 2, 3, 4};
std::vector<int> a = b;
// a now contains a copy of b and a[0] = 42 will not effect b.
cv::Mat b = cv::imread( ... );
cv::Mat a = b;
// a and b now wrap the same data.
But that said, you should not be using memcpy et. al. to copy a cv::Mat ... You can make copies with clone or copyTo. From the cv documentation:
Mat F = A.clone();
Mat G;
A.copyTo(G);
Background
I am trying to detect ArUco markers from image data sent from Unity to OpenCV Functions which are accessed using their .dlls.
Versions
OpenCV : 4.5.0/4.5.1
Unity : 2020.1.17f1
Aruco dictionary : 6x6_250 family and AprilTags 36h11 family
Problem
When I use a camera and send each frame(as shown in code here), detectMarkers() function works but every tag gets rejected.
I have tried:
Flipping the image only in Unity using System.Array.Reverse(rawImg) * I have also tried flipping the image in c++ using flip(imageCopy, imageCopy, -1) and then back after the image is processed;
In a combination of both in Unity and OpenCV. But I made sure the image which was feeding into detectMarkers() function was proper(i.e not flipped) by 'imshow()`.
I have also changed from aruCo dictionary to April tags so that there is no version mismatch in the tags.
I also checked by loading the image using imread() only in .dll. This works and the tag gets detected.
In addition, I also tried sending single image data (using Texture2D as image type). Here the Unity crashes during the cvtColor() line.
Part of the Unity code snippet
private WebCamTexture CamTexture;
void Start()
{
CamTexture = new WebCamTexture();
CamTexture.Play();
}
void Update()
{
if (CamTexture.isPlaying)
{
var rawImg = CamTexture.GetPixels32();
MarkersDetection.detectAruco(ref ArucoID, ref arrayLength, ref rawImg, WebcamWidth, WebcamHeight);
}
}
Part of OpenCV code snippet
struct Color32
{
uchar red;
uchar green;
uchar blue;
uchar alpha;
};
extern "C" void __declspec(dllexport) __stdcall detectAruco(int& outArucoID,int& arraySize,Color32 **rawImg,int width,int height) {
cv::Mat imageCopy;
cv::Mat image(height, width, CV_8UC4, *rawImg);
cvtColor(image, imageCopy, COLOR_BGRA2BGR);
cv::aruco::detectMarkers(imageCopy, arucoDictionary, arucoCorners, arucoIds, parameters, rejectedCandidates);
}
I was doing the flipping wrong! Now I flipped the image using a image tool(any editing tool which flips image works; rather than manually) and now it works.
I tried to convert a WriteableBitmap to a cv::Mat in a c++/cx Microsoft universial App. But when I try to progress with the created Mat, I get the following error:
This is my Code:
void App1::MainPage::processImage(SoftwareBitmap^ bitmap)
{
WriteableBitmap^ wb = ref new WriteableBitmap(bitmap->PixelWidth, bitmap->PixelHeight);
bitmap->CopyToBuffer(wb->PixelBuffer);
Mat img_image(wb->PixelHeight, wb->PixelWidth, CV_8UC3,(void*)wb->PixelBuffer);
//next step results in error
cvtColor(img_image, img_image, CV_BGR2BGRA);
...
}
So my final question:
How to convert the SoftwareBitmap or the WriteableBitmap to a cv::Mat?
The problem is that PixelBuffer is not a void *, it is an IBuffer^.
To get at the raw data, you can either use the IBufferByteAccess interface if you're comfortable with COM programming, or you can initialize a DataReader with an IBuffer if you'd prefer to stay in WinRT (although this technique will make a copy of the data).
I used the DataReader to solve the problem:
void App1::MainPage::processImage(SoftwareBitmap^ bitmap)
{
WriteableBitmap^ wb = ref new WriteableBitmap(bitmap->PixelWidth, bitmap->PixelHeight);
bitmap->CopyToBuffer(wb->PixelBuffer);
IBuffer^ buffer = wb->PixelBuffer;
auto reader = ::Windows::Storage::Streams::DataReader::FromBuffer(buffer);
BYTE *extracted = new BYTE[buffer->Length];
reader->ReadBytes(Platform::ArrayReference<BYTE>(extracted, buffer->Length));
Mat img_image(wb->PixelHeight, wb->PixelWidth, CV_8UC4, extracted);
cvtColor(img_image, img_image, CV_RGBA2BGRA);
...
}
Thx to Peter Torr for the hint.
suppose that I have a
vector<unsigned char>a
which is the raster information of a geotiff image extracted by the RasterIO function of the GDAL library( an opensource library for geographic information systems)
my image is a 7697x7309 one so the vector has 56257373 members.
How can I apply a 5x5 gaussian filter on this vector and then gain the result as another 56257373 members vector of the type unsigned char to be able to save the vector as another geotiff image using GDAL library.
My main question is the above but if it's not possible tell me if I have a geotiff file how can I apply filters on it using opencv at run-time. I mean I don't want to convert the format to another one like bitmap and tiff on the hard and then read data from hard to apply processes on that, suppose that I have data in GDAL format in one part of the memory and want to convert it to opencv compatible data in another part and apply filters on it?
I think this is what you are asking for:
// 1. Convert vector to Mat
cv::Mat amat(7309, 7697, CV_8UC1, &a[0]);
// 2. Apply 5x5 Gaussian filter
cv::Mat bmat; // blurred output, sigma=1.4 assumed below
cv::GaussianBlur(amat, bmat, cv::Size(5,5), 1.4);
// 3. Convert Mat to vector
cv::Mat cmat = bmat.reshape(1, 1); // make the Mat one big long row
std::vector<unsigned char>b = cmat;
A simpler than the vector< > way to convert from GDAL to OpenCV raster data:
//Region of Interest to be read
cv::Rect roi(x, y, w, h);
//Mat allocation to store the data
cv::Mat mat;
mat.create(roi.size(),CV_32F);
//data is stored directly in the mat passing the mat.data pointer to RasterIO
band->RasterIO( GF_Read, roi.x, roi.y, roi.width, roi.height, mat.data,
roi.width, roi.height, GDT_Float32, 0, 0);
You just have to be sure that OpenCV datatype fit the GDAL datatype and that ROI dimensions are ok for the raster size
I need to send an image into a byte array using c++/cli. The image is initially in Iplimage format.
int img_sz1 = img1->width * img1->height * img1->nChannels;
array <Byte>^ hh1 = gcnew array<Byte> (img_sz1);
Marshal::Copy( (IntPtr)img->imageData, hh1, 0, img_sz1 );
and it was working fine.
I added the encoding step to send it as jpeg
CvMat* buf1 = cvEncodeImage(".jpeg", img1, jpeg_params);
img_sz1=buf1->width*buf1->height
Marshal::Copy( (IntPtr)buf1, hh1, 0, img_sz1 );
and now it compiles fine but gives me the error at the marshal:copy line
An unhandled exception of type 'System.AccessViolationException' occurred in
mscorlib.dll. Additional information: Attempted to read or write protected memory.
Any help is very appreciated.
The return of cvEncodeImage is a single-row matrix, containing the encoded image data. What you're copying now is the struct itself, e.g., the width field, the height field, etc. I believe you need to copy from buf1->data instead.