Use multiple images for batch inference cppflow C++ - c++

Question

I'm trying to use cppflow library in windows 10 x64 machine in VS2019 C++. I want to inference my model for batch of images (vector <cv::Mat> ). I write a simple code as below for single image and it works correctly:
string im_path{ "..." };
string model_path{ "...\\ocr_model" };
cv::Mat tmp, im;
cv::resize(cv::imread(im_path, cv::IMREAD_GRAYSCALE), tmp, cv::Size(127, 25), 0, 0, cv::INTER_CUBIC);
cv::transpose(tmp, im);
int rows = im.rows; int cols = im.cols; int channels = im.channels();
// Put image in tensor
std::vector<uint8_t> img_data;
auto e = std::end(img_data);
img_data.insert(e, im.data, im.data + im.total() * channels);
auto input = cppflow::tensor(img_data, {rows, cols, channels});
input = cppflow::cast(input, TF_UINT8, TF_FLOAT);
auto t = input.get_data<float>();
input = input / 255.f;
input = cppflow::expand_dims(input, 0);
cppflow::model model{ model_path };
auto output = model({ {"serving_default_input:0", input}}, { "StatefulPartitionedCall:0"});
I want to load multiple images (in code below I use a cloned image as second image). here is what I really want to do:
string im_path{ "..." };
string model_path{ "...\\ocr_model" };
cv::Mat tmp, im;
cv::resize(cv::imread(im_path, cv::IMREAD_GRAYSCALE), tmp, cv::Size(127, 25), 0, 0, cv::INTER_CUBIC);
cv::transpose(tmp, im);
int rows = im.rows; int cols = im.cols; int channels = im.channels();
// Put image in tensor
std::vector<uint8_t> img_data;
auto im_clone = im.clone();
auto e = std::end(img_data);
img_data.insert(e, im.data, im.data + im.total() * channels);
e = std::end(img_data);
img_data.insert(e, im_clone.data, im_clone.data + im_clone.total() * channels);
auto input = cppflow::tensor(img_data, {2, rows, cols, channels});
input = cppflow::cast(input, TF_UINT8, TF_FLOAT);
input = input / 255.f;
input = cppflow::expand_dims(input, 0);
cppflow::model model{ model_path };
auto output = model({ {"serving_default_input:0", input}}, { "StatefulPartitionedCall:0"});
As you see the difference between the codes are img_data preparation and tensor definition but unfortunately, I get this error:
Unhandled exception at 0x00007FFFF4514ED9 in cppflow_Test.exe:
Microsoft C++ exception: std::runtime_error at memory location
0x00000031F72FDBD8.
How can I load multiple images (vector< cv::Mat >) to a tensor and use its corresponding outputs? in other words I need a example for batch inference using cppflow library.

Answer 1

Try using std::copy to insert multiple copies of the image (or multiple images) into the img_data vector.
int data_size = rows * cols * channels;
std::vector<uint8> img_data;
for (size_t i = 0; i < batch_size; i++)
{
std::copy(im.data, im.data + data_size, std::begin(img_data) + i * data_size);
}
auto input = cppflow::tensor(img_data, {batch_size, rows, cols, channels});

Answer 2

Use cppflow::concat to concatenate the n tensors you want to send to the model:
auto input1 = cppflow::decode_jpeg(file1);
input1 = cppflow::expand_dims(input1, 0);
auto input2 = cppflow::decode_jpeg(file2);
input2 = cppflow::expand_dims(input2, 0);
cppflow::tensor dim({0});
std::vector<cppflow::tensor> values;
values.push_back(input1);
values.push_back(input2);
auto inputs = cppflow::concat(dim, values);
auto output = model({ {"serving_default_input:0", inputs}}, { "StatefulPartitionedCall:0"});