Related
I am working on a project, where I want to process my images using C++ OpenCV.
For simplicity's sake, I just want to convert Uint8List to cv::Mat and back.
Following this tutorial, I managed to make a pipeline that doesn't crash the app. Specifically:
I created a function in a .cpp that takes the pointer to my Uint8List, rawBytes, and encodes it as a .jpg:
int encodeIm(int h, int w, uchar *rawBytes, uchar **encodedOutput) {
cv::Mat img = cv::Mat(h, w, CV_8UC3, rawBytes); //CV_8UC3
vector<uchar> buf;
cv:imencode(".jpg", img, buf); // save output into buf. Note that Dart Image.memory can process either .png or .jpg, which is why we're doing this encoding
*encodedOutput = (unsigned char *) malloc(buf.size());
for (int i=0; i < buf.size(); i++)
(*encodedOutput)[i] = buf[i];
return (int) buf.size();
}
Then I wrote a function in a .dart that calls my c++ encodeIm(int h, int w, uchar *rawBytes, uchar **encodedOutput):
//allocate memory heap for the image
Pointer<Uint8> imgPtr = malloc.allocate(imgBytes.lengthInBytes);
//allocate just 8 bytes to store a pointer that will be malloced in C++ that points to our variably sized encoded image
Pointer<Pointer<Uint8>> encodedImgPtr = malloc.allocate(8);
//copy the image data into the memory heap we just allocated
imgPtr.asTypedList(imgBytes.length).setAll(0, imgBytes);
//c++ image processing
//image in memory heap -> processing... -> processed image in memory heap
int encodedImgLen = _encodeIm(height, width, imgPtr, encodedImgPtr);
//
//retrieve the image data from the memory heap
Pointer<Uint8> cppPointer = encodedImgPtr.elementAt(0).value;
Uint8List encodedImBytes = cppPointer.asTypedList(encodedImgLen);
//myImg = Image.memory(encodedImBytes);
return encodedImBytes;
//free memory heap
//malloc.free(imgPtr);
//malloc.free(cppPointer);
//malloc.free(encodedImgPtr); // always frees 8 bytes
}
Then I linked c++ with dart via:
final DynamicLibrary nativeLib = Platform.isAndroid
? DynamicLibrary.open("libnative_opencv.so")
: DynamicLibrary.process();
final int Function(int height, int width, Pointer<Uint8> bytes, Pointer<Pointer<Uint8>> encodedOutput)
_encodeIm = nativeLib
.lookup<NativeFunction<Int32 Function(Int32 height, Int32 width,
Pointer<Uint8> bytes, Pointer<Pointer<Uint8>> encodedOutput)>>('encodeIm').asFunction();
And finally I show the result in Flutter via:
Image.memory(...)
Now, the pipeline doesn't crash, which means I haven't goofed up memory handling completely, but it doesn't return the original image either, which means I did mess up somewhere.
Original image:
Pipeline output:
Thanks to Richard Heap's guidance in the comments, I managed to fix the pipeline by changing my matrix definition from
cv::Mat img = cv::Mat(h, w, CV_8UC3, rawBytes);
to
vector<uint8_t> buffer(rawBytes, rawBytes + inBytesCount);
Mat img = imdecode(buffer, IMREAD_COLOR);
where inBytesCount is the length of imgBytes.
Could someone help me with displaying video from a camera. The values from camera are stored in QVector. Is it possible to display it this way?
Here is the code:
void MainWindow::slotDataUpdate(QVector<uint16_t> vec)
{
ui->setupUi(this);
double min = *std::min_element(vec.begin(), vec.end());
double max = *std::max_element(vec.begin(), vec.end());
std::transform(vec.begin(), vec.end(), vec.begin(), std::bind2nd(std::plus<double>(), (-1)*min));
int co = 65536/(max - min);
std::transform(vec.begin(), vec.end(), vec.begin(), std::bind2nd(std::multiplies<double>(), co));
const QImage tmpImage((const uchar*)vec.data(), 160, 120, QImage::Format_RGB16);
QByteArray data;
QDataStream stream(&data, QIODevice::WriteOnly);
stream << vec; //conversion from Q vector to QByteArray
// QByteArray data = QByteArray::fromRawData(reinterpret_cast<const char*>(vec.constData()),sizeof(double)* vec.size()); //conversion from Q vector to QByteArray
for (int i = 0; i < data.size(); i++)
{
std::cout << data[i] << " " << std::endl;
}
QImage tmpImage(160,120, QImage::Format_RGB32); //prazna slika pravih dimenzij
QRgb *pixels=reinterpret_cast<QRgb*>(tmpImage.bits());
for (size_t i=0;2*i<data.size();++i)
{
uchar pixel_msb=data[2*i+];
pixels[i]=qRgb(pixel_msb, pixel_msb, pixel_msb);
}
QPixmap pixmap(QPixmap::fromImage(tmpImage));
ui->label_img->setPixmap(pixmap.scaled(480,360,Qt::KeepAspectRatio));
}
RGB source image
Direct conversion is not possible, unless the QVector is a memory-mapped file. If it contains the raw bytes instead (as it seems to), I suggest constructing a QImage from the raw data as an intermediate object and then converting it to QPixmap:
// assuming 'v' is the vector, and 'w' and 'h' the size of the image
const QImage tmpImage((const char*)v.data(), w, h, QImage::Format_RGB16);
QPixmap pixmap(QPixmap::convertFromQImage(tmpImage));
Given the original element size is 16 bits, you may need QImage::Format_RGB16 (for 5-6-5) or QImage::Format_RGB555 (for 5-5-5). Check available formats for more options. Assuming, of course, an RGB image.
Grayscale images
If you image only has one color channel, then you'll have to make a manual conversion, since Qt is not able to manipulate 16-bits grayscale images.
Easy code, but may not produce the results you want:
const QImage tmpImage((const char*)v.data(), w, h, QImage::Format_RGB16);
QPixmap pixmap(QPixmap::convertFromQImage(
image.convertToFormat(QImage::Format::Format_Grayscale8)));
Otherwise, you have to reduce the depth of your image. Check this answer for a more detailed implementation. Nevertheless, this is a slow operation. If your Qt version allows it, use QImage::Format_Grayscale8 for destination image instead of the QImage::Format_RGB32 proposed in the linked answer, you'd write 3x less elements.
// 'image' is a QImage(w, h, QImage::Format_Grayscale)
// if possible, re-use it on each frame to reduce construction overhead
// 'ptr' is the pointer to your 16-bits data (const short* ptr)
auto ptr = v.constData();
#pragma omp parallel for // if can use OpenMP you can gain some speed here
for (int ii = 0; ii < image.height(); ++ii) {
auto scanLine = image.scanLine(ii);
for (int jj = 0; jj < image.width(); ++jj, ++ptr, ++scanLine) {
*scanLine = (unsigned char)(*ptr >> 8); // high byte, most significative bits
}
}
To be honest I'm suprised nobody has run into this thus far.
I'm loading a picture from OpenCV into cv::Mat, which I want to base64 encode before I send it over a socket.
For base64 I am using libb64 as it is native to Debian/Ubuntu, and easy to use and very fast. The encoding function takes as a parameter an std::ifstream, and outputs an std::ofstream.
#include <opencv2/opencv.hpp>
#include <b64/encode.h>
#include <fstream>
using namespace cv;
Mat image;
image = imread( "picture.jpg", CV_LOAD_IMAGE_COLOR );
if ( image.data )
{
std::ifstream instream( ???, std::ios_base::in | std::ios_base::binary);
std::ofstream outstream;
// Convert Matrix to ifstream
// ...
base64::encoder E;
E.encode( instream, outstream );
// Now put it in a string, and send it over a socket...
}
I don't really know how to populate the instream from the cv::Mat.
Googling around, I found that I can iterate a cv::Mat, by columns and rows, and get each (pixel I am assuming) RGB values:
for ( int j = 0; j < image.rows; j++ )
{
for ( int i = 0; i < image.cols; i++ )
{
unsigned char b = input [ image.step * j + i ] ;
unsigned char g = input [ image.step * j + i + 1 ];
unsigned char r = input [ image.step * j + i + 2 ];
}
}
Is this the right way of going on about it? Is there some more elegant way?
In order to be able to send an image via HTTP, you also need to encode its width, height and type. You need to serialize the Mat into a stream and encode that stream with libb64. On the other side you need to decode that stream and deserialize the image to retrieve it.
I implemented a small test program that does this serialization and deserialization using std::stringstream as a buffer. I chose it because it extends both std::istream and std::ostream which libb64 uses.
The serialize function serializes a cv::Mat into a std::stringstream. In it, I write the image width, height, type, size of the buffer and the buffer itself.
The deserialize function does the reverse. It reads the width, height, type, size of the buffer and the buffer. It's not as efficient as it could be because it needs to allocate a temporary buffer to read the data from the stringstream. Also, it needs to clone the image so that it does not rely on the temporary buffer and it will handle its own memory allocation. I'm sure that with some tinkering it can be made more efficient.
The main function loads an image, serializes it, encodes it using libb64, then decodes it, deserializes it and displays it in a window. This should simulate what you are trying to do .
// Serialize a cv::Mat to a stringstream
stringstream serialize(Mat input)
{
// We will need to also serialize the width, height, type and size of the matrix
int width = input.cols;
int height = input.rows;
int type = input.type();
size_t size = input.total() * input.elemSize();
// Initialize a stringstream and write the data
stringstream ss;
ss.write((char*)(&width), sizeof(int));
ss.write((char*)(&height), sizeof(int));
ss.write((char*)(&type), sizeof(int));
ss.write((char*)(&size), sizeof(size_t));
// Write the whole image data
ss.write((char*)input.data, size);
return ss;
}
// Deserialize a Mat from a stringstream
Mat deserialize(stringstream& input)
{
// The data we need to deserialize
int width = 0;
int height = 0;
int type = 0;
size_t size = 0;
// Read the width, height, type and size of the buffer
input.read((char*)(&width), sizeof(int));
input.read((char*)(&height), sizeof(int));
input.read((char*)(&type), sizeof(int));
input.read((char*)(&size), sizeof(size_t));
// Allocate a buffer for the pixels
char* data = new char[size];
// Read the pixels from the stringstream
input.read(data, size);
// Construct the image (clone it so that it won't need our buffer anymore)
Mat m = Mat(height, width, type, data).clone();
// Delete our buffer
delete[]data;
// Return the matrix
return m;
}
void main()
{
// Read a test image
Mat input = imread("D:\\test\\test.jpg");
// Serialize the input image to a stringstream
stringstream serializedStream = serialize(input);
// Base64 encode the stringstream
base64::encoder E;
stringstream encoded;
E.encode(serializedStream, encoded);
// Base64 decode the stringstream
base64::decoder D;
stringstream decoded;
D.decode(encoded, decoded);
// Deserialize the image from the decoded stringstream
Mat deserialized = deserialize(decoded);
// Show the retrieved image
imshow("Retrieved image", deserialized);
waitKey(0);
}
I am novice in OpenCV. Recently, I have troubles finding OpenCV functions to convert from Mat to Array. I researched with .ptr and .at methods available in OpenCV APIs, but I could not get proper data. I would like to have direct conversion from Mat to Array(if available, if not to Vector). I need OpenCV functions because the code has to be undergo high level synthesis in Vivado HLS. Please help.
If the memory of the Mat mat is continuous (all its data is continuous), you can directly get its data to a 1D array:
std::vector<uchar> array(mat.rows*mat.cols*mat.channels());
if (mat.isContinuous())
array = mat.data;
Otherwise, you have to get its data row by row, e.g. to a 2D array:
uchar **array = new uchar*[mat.rows];
for (int i=0; i<mat.rows; ++i)
array[i] = new uchar[mat.cols*mat.channels()];
for (int i=0; i<mat.rows; ++i)
array[i] = mat.ptr<uchar>(i);
UPDATE: It will be easier if you're using std::vector, where you can do like this:
std::vector<uchar> array;
if (mat.isContinuous()) {
// array.assign(mat.datastart, mat.dataend); // <- has problems for sub-matrix like mat = big_mat.row(i)
array.assign(mat.data, mat.data + mat.total()*mat.channels());
} else {
for (int i = 0; i < mat.rows; ++i) {
array.insert(array.end(), mat.ptr<uchar>(i), mat.ptr<uchar>(i)+mat.cols*mat.channels());
}
}
p.s.: For cv::Mats of other types, like CV_32F, you should do like this:
std::vector<float> array;
if (mat.isContinuous()) {
// array.assign((float*)mat.datastart, (float*)mat.dataend); // <- has problems for sub-matrix like mat = big_mat.row(i)
array.assign((float*)mat.data, (float*)mat.data + mat.total()*mat.channels());
} else {
for (int i = 0; i < mat.rows; ++i) {
array.insert(array.end(), mat.ptr<float>(i), mat.ptr<float>(i)+mat.cols*mat.channels());
}
}
UPDATE2: For OpenCV Mat data continuity, it can be summarized as follows:
Matrices created by imread(), clone(), or a constructor will always be continuous.
The only time a matrix will not be continuous is when it borrows data (except the data borrowed is continuous in the big matrix, e.g. 1. single row; 2. multiple rows with full original width) from an existing matrix (i.e. created out of an ROI of a big mat).
Please check out this code snippet for demonstration.
Can be done in two lines :)
Mat to array
uchar * arr = image.isContinuous()? image.data: image.clone().data;
uint length = image.total()*image.channels();
Mat to vector
cv::Mat flat = image.reshape(1, image.total()*image.channels());
std::vector<uchar> vec = image.isContinuous()? flat : flat.clone();
Both work for any general cv::Mat.
Explanation with a working example
cv::Mat image;
image = cv::imread(argv[1], cv::IMREAD_UNCHANGED); // Read the file
cv::namedWindow("cvmat", cv::WINDOW_AUTOSIZE );// Create a window for display.
cv::imshow("cvmat", image ); // Show our image inside it.
// flatten the mat.
uint totalElements = image.total()*image.channels(); // Note: image.total() == rows*cols.
cv::Mat flat = image.reshape(1, totalElements); // 1xN mat of 1 channel, O(1) operation
if(!image.isContinuous()) {
flat = flat.clone(); // O(N),
}
// flat.data is your array pointer
auto * ptr = flat.data; // usually, its uchar*
// You have your array, its length is flat.total() [rows=1, cols=totalElements]
// Converting to vector
std::vector<uchar> vec(flat.data, flat.data + flat.total());
// Testing by reconstruction of cvMat
cv::Mat restored = cv::Mat(image.rows, image.cols, image.type(), ptr); // OR vec.data() instead of ptr
cv::namedWindow("reconstructed", cv::WINDOW_AUTOSIZE);
cv::imshow("reconstructed", restored);
cv::waitKey(0);
Extended explanation:
Mat is stored as a contiguous block of memory, if created using one of its constructors or when copied to another Mat using clone() or similar methods. To convert to an array or vector we need the address of its first block and array/vector length.
Pointer to internal memory block
Mat::data is a public uchar pointer to its memory.
But this memory may not be contiguous. As explained in other answers, we can check if mat.data is pointing to contiguous memory or not using mat.isContinous(). Unless you need extreme efficiency, you can obtain a continuous version of the mat using mat.clone() in O(N) time. (N = number of elements from all channels). However, when dealing images read by cv::imread() we will rarely ever encounter a non-continous mat.
Length of array/vector
Q: Should be row*cols*channels right?
A: Not always. It can be rows*cols*x*y*channels.
Q: Should be equal to mat.total()?
A: True for single channel mat. But not for multi-channel mat
Length of the array/vector is slightly tricky because of poor documentation of OpenCV. We have Mat::size public member which stores only the dimensions of single Mat without channels. For RGB image, Mat.size = [rows, cols] and not [rows, cols, channels]. Mat.total() returns total elements in a single channel of the mat which is equal to product of values in mat.size. For RGB image, total() = rows*cols. Thus, for any general Mat, length of continuous memory block would be mat.total()*mat.channels().
Reconstructing Mat from array/vector
Apart from array/vector we also need the original Mat's mat.size [array like] and mat.type() [int]. Then using one of the constructors that take data's pointer, we can obtain original Mat. The optional step argument is not required because our data pointer points to continuous memory. I used this method to pass Mat as Uint8Array between nodejs and C++. This avoided writing C++ bindings for cv::Mat with node-addon-api.
References:
Create memory continuous Mat
OpenCV Mat data layout
Mat from array
Here is another possible solution assuming matrix have one column( you can reshape original Mat to one column Mat via reshape):
Mat matrix= Mat::zeros(20, 1, CV_32FC1);
vector<float> vec;
matrix.col(0).copyTo(vec);
None of the provided examples here work for the generic case, which are N dimensional matrices. Anything using "rows" assumes theres columns and rows only, a 4 dimensional matrix might have more.
Here is some example code copying a non-continuous N-dimensional matrix into a continuous memory stream - then converts it back into a Cv::Mat
#include <iostream>
#include <cstdint>
#include <cstring>
#include <opencv2/opencv.hpp>
int main(int argc, char**argv)
{
if ( argc != 2 )
{
std::cerr << "Usage: " << argv[0] << " <Image_Path>\n";
return -1;
}
cv::Mat origSource = cv::imread(argv[1],1);
if (!origSource.data) {
std::cerr << "Can't read image";
return -1;
}
// this will select a subsection of the original source image - WITHOUT copying the data
// (the header will point to a region of interest, adjusting data pointers and row step sizes)
cv::Mat sourceMat = origSource(cv::Range(origSource.size[0]/4,(3*origSource.size[0])/4),cv::Range(origSource.size[1]/4,(3*origSource.size[1])/4));
// correctly copy the contents of an N dimensional cv::Mat
// works just as fast as copying a 2D mat, but has much more difficult to read code :)
// see http://stackoverflow.com/questions/18882242/how-do-i-get-the-size-of-a-multi-dimensional-cvmat-mat-or-matnd
// copy this code in your own cvMat_To_Char_Array() function which really OpenCV should provide somehow...
// keep in mind that even Mat::clone() aligns each row at a 4 byte boundary, so uneven sized images always have stepgaps
size_t totalsize = sourceMat.step[sourceMat.dims-1];
const size_t rowsize = sourceMat.step[sourceMat.dims-1] * sourceMat.size[sourceMat.dims-1];
size_t coordinates[sourceMat.dims-1] = {0};
std::cout << "Image dimensions: ";
for (int t=0;t<sourceMat.dims;t++)
{
// calculate total size of multi dimensional matrix by multiplying dimensions
totalsize*=sourceMat.size[t];
std::cout << (t>0?" X ":"") << sourceMat.size[t];
}
// Allocate destination image buffer
uint8_t * imagebuffer = new uint8_t[totalsize];
size_t srcptr=0,dptr=0;
std::cout << std::endl;
std::cout << "One pixel in image has " << sourceMat.step[sourceMat.dims-1] << " bytes" <<std::endl;
std::cout << "Copying data in blocks of " << rowsize << " bytes" << std::endl ;
std::cout << "Total size is " << totalsize << " bytes" << std::endl;
while (dptr<totalsize) {
// we copy entire rows at once, so lowest iterator is always [dims-2]
// this is legal since OpenCV does not use 1 dimensional matrices internally (a 1D matrix is a 2d matrix with only 1 row)
std::memcpy(&imagebuffer[dptr],&(((uint8_t*)sourceMat.data)[srcptr]),rowsize);
// destination matrix has no gaps so rows follow each other directly
dptr += rowsize;
// src matrix can have gaps so we need to calculate the address of the start of the next row the hard way
// see *brief* text in opencv2/core/mat.hpp for address calculation
coordinates[sourceMat.dims-2]++;
srcptr = 0;
for (int t=sourceMat.dims-2;t>=0;t--) {
if (coordinates[t]>=sourceMat.size[t]) {
if (t==0) break;
coordinates[t]=0;
coordinates[t-1]++;
}
srcptr += sourceMat.step[t]*coordinates[t];
}
}
// this constructor assumes that imagebuffer is gap-less (if not, a complete array of step sizes must be given, too)
cv::Mat destination=cv::Mat(sourceMat.dims, sourceMat.size, sourceMat.type(), (void*)imagebuffer);
// and just to proof that sourceImage points to the same memory as origSource, we strike it through
cv::line(sourceMat,cv::Point(0,0),cv::Point(sourceMat.size[1],sourceMat.size[0]),CV_RGB(255,0,0),3);
cv::imshow("original image",origSource);
cv::imshow("partial image",sourceMat);
cv::imshow("copied image",destination);
while (cv::waitKey(60)!='q');
}
Instead of getting image row by row, you can put it directly to an array. For CV_8U type image, you can use byte array, for other types check here.
Mat img; // Should be CV_8U for using byte[]
int size = (int)img.total() * img.channels();
byte[] data = new byte[size];
img.get(0, 0, data); // Gets all pixels
byte * matToBytes(Mat image)
{
int size = image.total() * image.elemSize();
byte * bytes = new byte[size]; //delete[] later
std::memcpy(bytes,image.data,size * sizeof(byte));
}
You can use iterators:
Mat matrix = ...;
std::vector<float> vec(matrix.begin<float>(), matrix.end<float>());
cv::Mat m;
m.create(10, 10, CV_32FC3);
float *array = (float *)malloc( 3*sizeof(float)*10*10 );
cv::MatConstIterator_<cv::Vec3f> it = m.begin<cv::Vec3f>();
for (unsigned i = 0; it != m.end<cv::Vec3f>(); it++ ) {
for ( unsigned j = 0; j < 3; j++ ) {
*(array + i ) = (*it)[j];
i++;
}
}
Now you have a float array. In case of 8 bit, simply change float to uchar, Vec3f to Vec3b and CV_32FC3 to CV_8UC3.
If you know that your img is 3 channel, than you can try this code
Vec3b* dados = new Vec3b[img.rows*img.cols];
for (int i = 0; i < img.rows; i++)
for(int j=0;j<img.cols; j++)
dados[3*i*img.cols+j] =img.at<Vec3b>(i,j);
If you wanna check the (i,j) vec3b you can write
std::cout << (Vec3b)img.at<Vec3b>(i,j) << std::endl;
std::cout << (Vec3b)dados[3*i*img.cols+j] << std::endl;
Since answer above is not very accurate as mentioned in its comments but its "edit queue is full", I have to add correct one-liners.
Mat(uchar, 1 channel) to vector(uchar):
std::vector<uchar> vec = (image.isContinuous() ? image : image.clone()).reshape(1, 1); // data copy here
vector(any type) to Mat(the same type):
Mat m(vec, false); // false(by default) -- do not copy data
I have a array double dc[][] and want to convert this as to a IplImage* image and further to a video frame.
What I had to do was I was given a video and I extracted out some features and then make a new video of the extracted features.
My approach was I divided the video into frames extracted the features from each frame then did the updation like this and in each iteration of frame I get a new dc
double dc[48][44];
for(int i=0;i<48;i++)
{
for(int j=0;j<44;j++)
{
dc[i][j]=max1[i][j]/(1+max2[i][j]);
}
}
Now I need to save this dc in such a way that I can reconstruct the video.Anybody help me with this.
Thanks in advance
If you're okay with using Mat, then you can make a Mat for existing user-allocated memory. One of the Mat constructors has the signature:
Mat::Mat(int rows, int cols, int type, void* data, size_t step=AUTO_STEP)
where the parameters are:
rows: the memory height,
cols: the width,
type: one of the OpenCV data types (e.g. CV_8UC3),
data: pointer to your data,
step: (optional) stride of your data
I'd encourage you to take a look at the documentation for Mat here
EDIT: Just to make things more concrete, here's an example of making a Mat from some user-allocated data
int main()
{
//allocate and initialize your user-allocated memory
const int nrows = 10;
const int ncols = 10;
double data[nrows][ncols];
int vals = 0;
for (int i = 0; i < nrows; i++)
{
for (int j = 0; j < ncols; j++)
{
data[i][j] = vals++;
}
}
//make the Mat from the data (with default stride)
cv::Mat cv_data(nrows, ncols, CV_64FC1, data);
//print the Mat to see for yourself
std::cout << cv_data << std::endl;
}
You can save a Mat to a video file via the OpenCV VideoWriter class. You just need to create a VideoWriter, open a video file, and write your frames (as Mat). You can see an example of using VideoWriter here
Here's a short example of using the VideoWriter class:
//fill-in a name for your video
const std::string filename = "...";
const double FPS = 30;
VideoWriter outputVideo;
//opens the output video file using an MPEG-1 codec, 30 frames per second, of size height x width and in color
outputVideo.open(filename, CV_FOURCC('P','I','M,'1'), FPS, Size(height, width));
Mat frame;
//do things with the frame
// ...
//writes the frame out to the video file
outputVideo.write(frame);
The tricky part of the VideoWriter is the opening of the file, as you have a lot of options. You can see the names for different codecs here