How to save DLIB object detection chips properly? - c++

Question

I am having trouble saving image chips generated by DLIB's face detection model. The code below details my workflow. I have attempted saving the whole image, d_image below, and that works just fine. However, when I try to save each chip I get distorted output (see example below). I'm using dlib 19.4 on Ubuntu 16.04.
// object to store raw image data
cv::Mat rawImage;
// initialize the detector
dlib::frontal_face_detector detector = dlib::get_frontal_face_detector();
// using shape predictor object to create dull_object_detections
dlib::shape_predictor sp;
dlib::deserialize(argv[1]) >> sp;
// for writing out images
int image_id = 1;
while (true){
// retrieve image size
sockt.getData(&image_size, 4, NULL);
if (image_size > 0) {
rawImage.create(1, image_size, CV_8UC1);
// load incoming data from a stream
sockt.getData(rawImage.data, image_size, MSG_WAITALL);
// reshape and correct orientation
dlib::cv_image<dlib::bgr_pixel> d_image = utils::process_frame(rawImage);
// find the daces!
std::vector<dlib::rectangle> detections = detector(d_image);
if (detections.size() > 0){
// generate additional detection data so we can use
// dlib's extract_image_chips function
std::vector<dlib::full_object_detection> shapes;
for (int idx = 0; idx < detections.size(); idx++){
dlib::full_object_detection shape = sp(d_image, detections[idx]);
shapes.push_back(shape);
}
// write each chip to disk
dlib::array<dlib::array2d<dlib::bgr_pixel>> face_chips;
dlib::extract_image_chips(d_image, dlib::get_face_chip_details(shapes), face_chips);
for (int idx = 0; idx < face_chips.size(); idx++){
std::string fname = argv[2] + std::to_string(image_id) + ".jpg";
dlib::save_jpeg(face_chips[idx], fname);
image_id++;
}
}
Example saved chip:
Edit: Added comment to utils::process_frame. This function accepts a 1xN array and decodes as a JPEG using OpenCV

Answer 1

Something wrong with image formats you are using:
This is OpenCV's greyscale (1-channel) image
rawImage.create(1, image_size, CV_8UC1);
This is BGR (3-channel) image
dlib::cv_image<dlib::bgr_pixel> d_image = utils::process_frame(rawImage);
Dlib should throw an exception if image has incorrect number of channels, but it does not throw it in your case. This means that somewhere in utils::process_frame(rawImage) the image format is changed into 3-channel- check image formats first
And this construction code rawImage.create(1, image_size, CV_8UC1); constructs 1-row and image_size cols image.
Something is incorrect with image size and format
Please also note, that dlib does not copy image data into dlib::cv_image<dlib::bgr_pixel> d_image and the rawImage should remain unchanged by other threads until processing is finished
Anyways, you can call dlib::toMat, get OpenCV Mat and save it with OpenCV functions
UPDATE:
one more thing here:
dlib::cv_image<dlib::bgr_pixel> d_image = utils::process_frame(rawImage);
looks like utils::process_frame returns some temporary object that is destroyed after d_image is constructed. d_image does not hold returned data and it can be lost
So I suggest you to change your code like this:
cv::Mat uncompressed;
tils::process_frame(rawImage, uncompressed);
dlib::cv_image<dlib::bgr_pixel> d_image(uncompressed);;
where process_frame should take reference to cv::Mat and save its output into it