How to convert the mat image to string (array2d<rgb_pixel>)? - c++

I created dll for dlib for face landmark detection, in opencv we get the image in MAT but here in dlib the we get it by using array2d, so any one say how to convert the mat to array2d ??
The full code can be seen viewed here

You can try to use cv_image, such as:
cv::Mat temp;
...
cv_image<bgr_pixel> image(temp);
// Detect faces
std::vector<rectangle> faces = detector(image);
Also note that this just wraps the cv::Mat object and doesn't copy anything. So image is only valid as long as temp is valid. This basically means you shouldn't modify temp while using image.

Related

How to get the bytes from opencv image matrix?

If I have an opencv image that I read from a png file like this:
cv::Mat img = cv::imread("/to/path/test.png");
how do I get that image in bytes? I know using img.data returns an unsigned char* but that is not what I need. Any suggestions?
If I got your question right, you want, for example, a 250*250 image to return a 250*250 matrix so I would suggest using grey-scale instead of BGR
imgData = cv2.imread(path, 0)
I believe this is written in C++ like this
cv::Mat img = cv::imread(file_name);//It returns a matrix object
cv::Mat graymat;
cvtColor(img, graymat,cv::COLOR_BGR2GRAY);

How to save DLIB object detection chips properly?

I am having trouble saving image chips generated by DLIB's face detection model. The code below details my workflow. I have attempted saving the whole image, d_image below, and that works just fine. However, when I try to save each chip I get distorted output (see example below). I'm using dlib 19.4 on Ubuntu 16.04.
// object to store raw image data
cv::Mat rawImage;
// initialize the detector
dlib::frontal_face_detector detector = dlib::get_frontal_face_detector();
// using shape predictor object to create dull_object_detections
dlib::shape_predictor sp;
dlib::deserialize(argv[1]) >> sp;
// for writing out images
int image_id = 1;
while (true){
// retrieve image size
sockt.getData(&image_size, 4, NULL);
if (image_size > 0) {
rawImage.create(1, image_size, CV_8UC1);
// load incoming data from a stream
sockt.getData(rawImage.data, image_size, MSG_WAITALL);
// reshape and correct orientation
dlib::cv_image<dlib::bgr_pixel> d_image = utils::process_frame(rawImage);
// find the daces!
std::vector<dlib::rectangle> detections = detector(d_image);
if (detections.size() > 0){
// generate additional detection data so we can use
// dlib's extract_image_chips function
std::vector<dlib::full_object_detection> shapes;
for (int idx = 0; idx < detections.size(); idx++){
dlib::full_object_detection shape = sp(d_image, detections[idx]);
shapes.push_back(shape);
}
// write each chip to disk
dlib::array<dlib::array2d<dlib::bgr_pixel>> face_chips;
dlib::extract_image_chips(d_image, dlib::get_face_chip_details(shapes), face_chips);
for (int idx = 0; idx < face_chips.size(); idx++){
std::string fname = argv[2] + std::to_string(image_id) + ".jpg";
dlib::save_jpeg(face_chips[idx], fname);
image_id++;
}
}
Example saved chip:
Edit: Added comment to utils::process_frame. This function accepts a 1xN array and decodes as a JPEG using OpenCV
Something wrong with image formats you are using:
This is OpenCV's greyscale (1-channel) image
rawImage.create(1, image_size, CV_8UC1);
This is BGR (3-channel) image
dlib::cv_image<dlib::bgr_pixel> d_image = utils::process_frame(rawImage);
Dlib should throw an exception if image has incorrect number of channels, but it does not throw it in your case. This means that somewhere in utils::process_frame(rawImage) the image format is changed into 3-channel- check image formats first
And this construction code rawImage.create(1, image_size, CV_8UC1); constructs 1-row and image_size cols image.
Something is incorrect with image size and format
Please also note, that dlib does not copy image data into dlib::cv_image<dlib::bgr_pixel> d_image and the rawImage should remain unchanged by other threads until processing is finished
Anyways, you can call dlib::toMat, get OpenCV Mat and save it with OpenCV functions
UPDATE:
one more thing here:
dlib::cv_image<dlib::bgr_pixel> d_image = utils::process_frame(rawImage);
looks like utils::process_frame returns some temporary object that is destroyed after d_image is constructed. d_image does not hold returned data and it can be lost
So I suggest you to change your code like this:
cv::Mat uncompressed;
tils::process_frame(rawImage, uncompressed);
dlib::cv_image<dlib::bgr_pixel> d_image(uncompressed);;
where process_frame should take reference to cv::Mat and save its output into it

Nao Robot: How to use the ALImage ROI

What I want to do:
Convert a ROI of the Nao Robot camera image on the OpenCV::Mat format. Later I will make use of this OpenCV::Mat
The situation:
Nao SDK provide the image in a format called ALImage. It is possible to convert the ALImage to OpenCV::Mat format, but I do not need all the image, only a small ROI. Despite ALImage provides its own ROI, the methods to use it are not really helpful:
int getNumOfROIs () const
const ROI* getROI (int index) const
void addROI (const ROI &rect)
void cleanROIs ()
void setEnableROIs (bool enable)
bool isROIEnabled () const
The question:
How can I use these ROIs?
Assuming you already have the coordinates of your ROI, you can crop your cv::Mat like this:
// your source image
cv::Mat image(imagesource);
// a rectangle with your ROI coordinates
cv::Rect myROI(10, 10, 100, 100);
// a "reference" to the image data within the rectangle
cv::Mat croppedImage = image(myROI);
Note that this does not copy the image data. Both, image and croppedImage share the same underlying raw data (a detailed example can be found in the opencv documentation). When you're done with the big source image, you can do image.release() and croppedImage = croppedImage.clone(); to deallocate all unnecessary data (outside of your ROI).
EDIT:
I didn't work with AL::ALImage::ROI yet, but the definition in alimage/alimage.h looks familiar to cv::Rect. So you can probably do the following:
// let's pretend you already got your image ...
AL::ALImage yourImage;
// ... and a ROI ...
AL::ALImage::ROI yourROI;
// ... as well as a Mat header with the dimensions and type of yourImage
cv::Mat header;
// then you can convert the ROI, ...
cv::Rect cvROI(yourROI.x, yourROI.y, yourROI.w, yourROI.h);
// ... wrap ALImage to a Mat (which should not copy anything) ...
header.data = yourImage.getData();
// ... and then proceed with the steps mentioned above to crop your Mat
cv::Mat cropped = header(cvROI);
header.release();
cropped = cropped.clone();
// ...
// your image processing using cropped
// ...
I hope this helps.

OpenCV: Shift/Align face image relative to reference Image (Image Registration)

I am new to OpenCV2 and working on a project in emotion recognition and would like to align a facial image in relation to a reference facial image. I would like to get the image translation working before moving to rotation. Current idea is to run a search within a limited range on both x and y coordinates and use the sum of squared differences as error metric to select the optimal x/y parameters to align the image. I'm using the OpenCV face_cascade function to detect the face images, all images are resized to a fixed (128x128). Question: Which parameters of the Mat image do I need to modify to shift the image in a positive/negative direction on both x and y axis? I believe setImageROI is no longer supported by Mat datatypes? I have the ROIs for both faces available however I am unsure how to use them.
void alignImage(vector<Rect> faceROIstore, vector<Mat> faceIMGstore)
{
Mat refimg = faceIMGstore[1]; //reference image
Mat dispimg = faceIMGstore[52]; // "displaced" version of reference image
//Rect refROI = faceROIstore[1]; //Bounding box for face in reference image
//Rect dispROI = faceROIstore[52]; //Bounding box for face in displaced image
Mat aligned;
matchTemplate(dispimg, refimg, aligned, CV_TM_SQDIFF_NORMED);
imshow("Aligned image", aligned);
}
The idea for this approach is based on Image Alignment Tutorial by Richard Szeliski Working on Windows with OpenCV 2.4. Any suggestions are much appreciated.
cv::Mat does support ROI. (But it does not support COI - channel-of-interest.)
To apply ROI you can use operator() or special constructor:
Mat refimgROI = faceIMGstore[1](faceROIstore[1]); //reference image ROI
Mat dispimgROI(faceIMGstore[52], faceROIstore[52]); // "displaced" version of reference image ROI
And to find the best position inside a displaced image you can utilize matchTemplate function.
Based on your comments I can suggest the following code which will find the best position of reference patch nearby the second (displaced) patch:
Mat ref = faceIMGstore[1](faceROIstore[1]);
Mat disp = faceIMGstore[52](faceROIstore[52]);
disp = disp.adjustROI(5,5,5,5); //allow 5 pixel max adjustment in any direction
if(disp.cols < ref.cols || disp.rows < ref.rows)
return 0;
Mat map;
cv::matchTemplate( disp, ref, map, CV_TM_SQDIFF_NORMED );
Point minLoc;
cv::minMaxLoc( map, 0, &minLoc );
Mat adjusted = disp(Rect(minLoc.x, minLoc.y, ref.cols, ref.rows));

PCA in OpenCV using the new C++ interface

As an aside: Apologies if I'm flooding SO with OpenCV questions :p
I'm currently trying to port over my old C code to use the new C++ interface and I've got to the point where I'm rebuilidng my Eigenfaces face recogniser class.
Mat img = imread("1.jpg");
Mat img2 = imread("2.jpg");
FaceDetector* detect = new HaarDetector("haarcascade_frontalface_alt2.xml");
// convert to grey scale
Mat g_img, g_img2;
cvtColor(img, g_img, CV_BGR2GRAY);
cvtColor(img2, g_img2, CV_BGR2GRAY);
// find the faces in the images
Rect r = detect->getFace(g_img);
Mat img_roi = g_img(r);
r = detect->getFace(g_img2);
Mat img2_roi = g_img2(r);
// create the data matrix for PCA
Mat data;
data.create(2,1, img2_roi.type());
data.row(0) = img_roi;
data.row(1) = img2_roi;
// perform PCA
Mat averageFace;
PCA pca(data, averageFace, CV_PCA_DATA_AS_ROW, 2);
//namedWindow("avg",1); imshow("avg", averageFace); - causes segfault
//namedWindow("avg",1); imshow("avg", Mat(pca.mean)); - doesn't work
I'm trying to create the PCA space, and then see if it's working by displaying the computed average image. Are there any other steps to this?
Perhaps I need to project the images onto the PCA subspace first of all?
Your error is probably here:
Mat data;
data.create(2,1, img2_roi.type());
data.row(0) = img_roi;
data.row(1) = img2_roi;
PCA expects a matrix with the data vectors as rows. However, you never scale the images to the same size so that they have the same number of pixels (so the dimension is the same), also data.create(2,1,...) - the 1 needs to be the dimension of your vector, i.e. the number of your pixels. Then copy the pixels from the crop to your matrix.