Run tflite accuracy tool on official tensorflow resnet50 model - c++

Question

I have downloaded the official resnet50 model provided here: https://github.com/tensorflow/models/tree/master/official/resnet. I needed a tflite quantized version of this model and hence I converted the model to a tflite format as follows :
toco --output_file /tmp/resnet50_quant.tflite --saved_model_dir <path/to/saved_model_dir> --output_format TFLITE --quantize_weights QUANTIZE_WEIGHTS
After this, I thought I'd run the tflite accuracy tool to verify the accuracy of this model is still reasonable. Although it looks like I run into the following issue:
bazel run -c opt --copt=-march=native --cxxopt='--std=c++11' -- //tensorflow/contrib/lite/tools/accuracy/ilsvrc:imagenet_accuracy_eval --model_file=/tmp/resnet50_quant.tflite --ground_truth_images_path=<path/to/images> --ground_truth_labels=/tmp/validation_labels.txt --model_output_labels=/tmp/tf_labels.txt --output_file_path=/tmp/accuracy_output.txt --num_images=0
INFO: Analysed target //tensorflow/contrib/lite/tools/accuracy/ilsvrc:imagenet_accuracy_eval (0 packages loaded).
INFO: Found 1 target...
Target //tensorflow/contrib/lite/tools/accuracy/ilsvrc:imagenet_accuracy_eval up-to-date:
bazel-bin/tensorflow/contrib/lite/tools/accuracy/ilsvrc/imagenet_accuracy_eval
INFO: Elapsed time: 14.589s, Critical Path: 14.28s
INFO: 3 processes: 3 local.
INFO: Build completed successfully, 4 total actions
INFO: Running command line: bazel-bin/tensorflow/contrib/lite/tools/accuracy/ilsvrc/imagenet_accuracy_eval '--model_file=/tmp/resnet50_quant.tflite' '--ground_truth_images_path=<path/to/images>' '--ground_truth_labels=/tmp/validation_labels.txt' '--model_output_labels=/tmp/tf_labels.txt' '--output_file_path=/tmp/accuracy_output.txt' 'INFO: Build completed successfully, 4 total actions
2018-10-12 15:30:06.237058: E tensorflow/contrib/lite/tools/accuracy/ilsvrc/imagenet_accuracy_eval.cc:155] Starting evaluation with: 4 threads.
2018-10-12 15:30:06.536802: E tensorflow/contrib/lite/tools/accuracy/ilsvrc/imagenet_accuracy_eval.cc:98] Starting model evaluation: 50000
2018-10-12 15:30:06.565334: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at run_tflite_model_op.cc:89 : Invalid argument: Data shapes mismatch for tensors: 0 expected: [64,224,224,3] got: [1,224,224,3]
2018-10-12 15:30:06.565453: F tensorflow/contrib/lite/tools/accuracy/ilsvrc/imagenet_model_evaluator.cc:222] Non-OK-status: eval_pipeline->Run(CreateStringTensor(image_label.image), CreateStringTensor(image_label.label)) status: Invalid argument: Data shapes mismatch for tensors: 0 expected: [64,224,224,3] got: [1,224,224,3]
[[{{node stage_run_tfl_model_output}} = RunTFLiteModel[input_type=[DT_FLOAT], model_file_path="/tmp/resnet50_quant.tflite", output_type=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/device:CPU:0"](stage_inception_preprocess_output)]]
It looks like the issue is that the official resnet model has an input tensor of [64, 224, 224, 3] whereas the accuracy tool provides an input of [1, 224, 224, 3]. So, the official model seems to expect a batch of 64 images and hence the accuracy tool fails.
I was wondering what I need to do to get the accuracy tool to run on the official resnet50 model? I'm guessing that although the input tensor for resnet 50 is [64, 224, 224, 3], there should be a way to still run a single image through the model.

Answer 1

There are two ways to go about it:
Resize the input of your model to [1, 224, 224, 3] and run the tool.
You could try looking at this and then modifying this file accordingly.
Alternatively modify the same tool so that it feeds in 64 images at a time instead of 1. You can look at the same code file I point to above and feed 64 at a time instead of 1.
If you're looking for long-term support, consider filing a feature request on Github where we can support batching.