Tensorflow DrawBoundingBoxes C++ - c++

Question

I am detecting objects using Tensorflow in C++. It does work well and i want to draw the boxes to have some visual feedback.
There is the operation tensorflow::ops::DrawBoundingBoxes that would let me do this but the problem is :
I don't understand what the input values should be for the boxes. What does this mean :
boxes: 3-D with shape [batch, num_bounding_boxes, 4] containing bounding boxes.
I could not find an example that uses this operation in C++ anywhere, like almost as if this ops does not exist.
Is there an example somewhere in C++ where this ops is used? It sounds a basic things to do for a tutorial or to debug.

Answer 1

In case if you are still on this question, I have written my own implementation of this operation using OpenCV basic methods. It also supports captioning the boxes with corresponding class labels.
/** Draw bounding box and add caption to the image.
* Boolean flag _scaled_ shows if the passed coordinates are in relative units (true by default in tensorflow detection)
*/
void drawBoundingBoxOnImage(Mat &image, double yMin, double xMin, double yMax, double xMax, double score, string label, bool scaled=true) {
cv::Point tl, br;
if (scaled) {
tl = cv::Point((int) (xMin * image.cols), (int) (yMin * image.rows));
br = cv::Point((int) (xMax * image.cols), (int) (yMax * image.rows));
} else {
tl = cv::Point((int) xMin, (int) yMin);
br = cv::Point((int) xMax, (int) yMax);
}
cv::rectangle(image, tl, br, cv::Scalar(0, 255, 255), 1);
// Ceiling the score down to 3 decimals (weird!)
float scoreRounded = floorf(score * 1000) / 1000;
string scoreString = to_string(scoreRounded).substr(0, 5);
string caption = label + " (" + scoreString + ")";
// Adding caption of type "LABEL (X.XXX)" to the top-left corner of the bounding box
int fontCoeff = 12;
cv::Point brRect = cv::Point(tl.x + caption.length() * fontCoeff / 1.6, tl.y + fontCoeff);
cv::rectangle(image, tl, brRect, cv::Scalar(0, 255, 255), -1);
cv::Point textCorner = cv::Point(tl.x, tl.y + fontCoeff * 0.9);
cv::putText(image, caption, textCorner, FONT_HERSHEY_SIMPLEX, 0.4, cv::Scalar(255, 0, 0));
}
/** Draw bounding boxes and add captions to the image.
* Box is drawn only if corresponding score is higher than the _threshold_.
*/
void drawBoundingBoxesOnImage(Mat &image,
tensorflow::TTypes<float>::Flat scores,
tensorflow::TTypes<float>::Flat classes,
tensorflow::TTypes<float,3>::Tensor boxes,
map<int, string> labelsMap, double threshold=0.5) {
for (int j = 0; j < scores.size(); j++)
if (scores(j) > threshold)
drawBoundingBoxOnImage(image, boxes(0,j,0), boxes(0,j,1), boxes(0,j,2), boxes(0,j,3), scores(j), labelsMap[classes(j)]);
}
The complete example is here.

Answer 2

Here is a small usage example in python that draws 2 rectangles on the image.png and saves it as outout.png, I believe it should help you:
from __future__ import absolute_import
from __future__ import division
from __future__ import print_function
import tensorflow as tf
import numpy as np
import PIL.Image as pimg
if __name__ == '__main__':
image = tf.convert_to_tensor(np.array(pimg.open('image.png'), np.float), tf.float32)
bbox = tf.convert_to_tensor([[0.1, 0.1, 0.4, 0.4], [0.5, 0.5, 0.6, 0.7]])
with tf.Session() as s:
s.run(tf.global_variables_initializer())
output = s.run(tf.image.draw_bounding_boxes(tf.expand_dims(image, 0), tf.expand_dims(bbox, 0)))
pimg.fromarray(np.uint8(output[0])).save('output.png')
The boxes is an array of arrays of rectangles for each image, where rectangle is defined by four normalized floats [min_y, min_x, max_y, max_x].

Answer 3

I manage to got it working using this.
Too bad you cannot really control how the box should look like nor other informations like the score or the label text.
Status CreateBoxedTensor(Tensor &input_image, Tensor &input_boxes, Tensor *output) {
auto root = Scope::NewRootScope();
// First OP is to convert uint8 image tensor to float for the drawing op to not loose its s*
Input imgin(input_image);
auto cast_op = Cast(root, imgin, DT_FLOAT);
// Next one is the drawing itself
Input boxin(input_boxes);
auto draw_op = DrawBoundingBoxes(root, cast_op, boxin);
// And we convert back to uint8 because it's RGB after all that we want
auto cast_back_op = Cast(root, draw_op, DT_UINT8);
ClientSession session(root);
std::vector<Tensor> out_tensors;
TF_RETURN_IF_ERROR(session.Run({cast_back_op}, &out_tensors));
*output = out_tensors[0];
return Status::OK();
}