Setting input layer in CAFFE with C++ - c++

I'm writing C++ code using CAFFE to predict a single (for now) image. The image has already been preprocessed and is in .png format. I have created a Net object and read in the trained model. Now, I need to use the .png image as an input layer and call net.Forward() - but can someone help me figure out how to set the input layer?
I found a few examples on the web, but none of them work, and almost all of them use deprecated functionality. According to: Berkeley's Net API, using "ForwardPrefilled" is deprecated, and using "Forward(vector, float*)" is deprecated. API indicates that one should "set input blobs, then use Forward() instead". That makes sense, but the "set input blobs" part is not expanded on, and I can't find a good C++ example on how to do that.
I'm not sure if using a caffe::Datum is the right way to go or not, but I've been playing with this:
float lossVal = 0.0;
caffe::Datum datum;
caffe::ReadImageToDatum("myImg.png", 1, imgDims[0], imgDims[1], &datum);
caffe::Blob< float > *imgBlob = new caffe::Blob< float >(1, datum.channels(), datum.height(), datum.width());
//How to get the image data into the blob, and the blob into the net as input layer???
const vector< caffe::Blob< float >* > &result = caffeNet.Forward(&lossVal);
Again, I'd like to follow the API's direction of setting the input blobs and then using the (non-deprecated) caffeNet.Forward(&lossVal) to get the result as opposed to making use of the deprecated stuff.
EDIT:
Based on an answer below, I updated to include this:
caffe::MemoryDataLayer<unsigned char> *memory_data_layer = (caffe::MemoryDataLayer<unsigned char> *)caffeNet.layer_by_name("input").get();
vector< caffe::Datum > datumVec;
datumVec.push_back(datum);
memory_data_layer->AddDatumVector(datumVec);
but now the call to AddDatumVector is seg faulting.. I wonder if this is related to my prototxt format? here's the top of my prototxt:
name: "deploy"
input: "data"
input_shape {
dim: 1
dim: 3
dim: 100
dim: 100
}
layer {
name: "conv1"
type: "Convolution"
bottom: "data"
top: "conv1"
I base this part of the question on this discussion about a "source" field being important in the prototxt...

caffe::Datum datum;
caffe::ReadImageToDatum("myImg.png", 1, imgDims[0], imgDims[1], &datum);
MemoryDataLayer<float> *memory_data_layer = (MemoryDataLayer<float> *)caffeNet->layer_by_name("data").get();
memory_data_layer->AddDatumVector(datum);
const vector< caffe::Blob< float >* > &result = caffeNet.Forward(&lossVal);
Something like this could be useful. Here you will have to use MemoryData layer as the input layer. I am expecting the layer name to be named data.
The way of using datum variable may not be correct. If my memory is correct, I guess, you have to use a vector of datum data.
I think this should get you started.
Happy brewing. :D

Here is an excerpt from my code located here where I used Caffe in my C++ code. I hope this helps.
Net<float> caffe_test_net("models/sudoku/deploy.prototxt", caffe::TEST);
caffe_test_net.CopyTrainedLayersFrom("models/sudoku/sudoku_iter_10000.caffemodel");
// Get datum
Datum datum;
if (!ReadImageToDatum("examples/sudoku/cell.jpg", 1, 28, 28, false, &datum)) {
LOG(ERROR) << "Error during file reading";
}
// Get the blob
Blob<float>* blob = new Blob<float>(1, datum.channels(), datum.height(), datum.width());
// Get the blobproto
BlobProto blob_proto;
blob_proto.set_num(1);
blob_proto.set_channels(datum.channels());
blob_proto.set_height(datum.height());
blob_proto.set_width(datum.width());
int size_in_datum = std::max<int>(datum.data().size(),
datum.float_data_size());
for (int ii = 0; ii < size_in_datum; ++ii) {
blob_proto.add_data(0.);
}
const string& data = datum.data();
if (data.size() != 0) {
for (int ii = 0; ii < size_in_datum; ++ii) {
blob_proto.set_data(ii, blob_proto.data(ii) + (uint8_t)data[ii]);
}
}
// Set data into blob
blob->FromProto(blob_proto);
// Fill the vector
vector<Blob<float>*> bottom;
bottom.push_back(blob);
float type = 0.0;
const vector<Blob<float>*>& result = caffe_test_net.Forward(bottom, &type);

What about:
Caffe::set_mode(Caffe::CPU);
caffe_net.reset(new caffe::Net<float>("your_arch.prototxt", caffe::TEST));
caffe_net->CopyTrainedLayersFrom("your_model.caffemodel");
Blob<float> *your_blob = caffe_net->input_blobs()[0];
your_blob->set_cpu_data(your_image_data_as_pointer_to_float);
caffe_net->Forward();

Related

How to parse JSON array from inside an object with rapidjson

The following is a JSON file exported from Tiled Map Editor.
{ "compressionlevel":-1,
"height":32,
"infinite":false,
"layers":[
{
"data":[ A whole bunch of integers in here],
"height":32,
"id":1,
"name":"Tile Layer 1",
"opacity":1,
"type":"tilelayer",
"visible":true,
"width":32,
"x":0,
"y":0
}],
"nextlayerid":2,
"nextobjectid":1,
"orientation":"orthogonal",
"renderorder":"right-down",
"tiledversion":"1.7.2",
"tileheight":32,
"tilesets":[
{
"firstgid":1,
"source":"..\/..\/..\/..\/Desktop\/tileset001.tsx"
}],
"tilewidth":32,
"type":"map",
"version":"1.6",
"width":32
}
And in this C++ block I am trying to parse out the data I actually need.
std::ifstream inFStream(filePath, std::ios::in);
if(!inFStream.is_open())
{
printf("Failed to open map file: &s", filePath);
}
rapidjson::IStreamWrapper inFStreamWrapper{inFStream};
rapidjson::Document doc{};
doc.ParseStream(inFStreamWrapper);
_WIDTH = doc["width"].GetInt(); //get width of map in tiles
_HEIGHT = doc["height"].GetInt(); //get height of map in tiles
const rapidjson::Value& data = doc["layers"]["data"]; //FAILURE POINT
assert(data.IsArray());
When I compile I am able to extract the right value for width and height which are outside of "layers" :[{}]
But when that const rapidjson::Value& data = doc["layers"]["data"]; gets called I get a runtime error claiming that document.h line 1344 IsObject() Assertion Failed.
Ive been up and down the rapidjson website and other resources and cant find anything quite like this.
The next step would be to get the int values stored in "data" and push them into a std::vector but thats not going to happen till I figure out how to get access to "data"
doc['layers'] is an array.
const rapidjson::Value& layers = doc["layers"];
assert(layers.IsArray());
for (size_t i=0; i < layers.Size(); i++) {
const rapidjson::Value& data = doc["layers"][i]["data"];
assert(data.IsArray());
}
UPDATE:
Direct access to first data item in layers:
const rapidjson::Value& data = doc["layers"][0]["data"];
This only gives you the data for the first item in layers array. If layers have at least one item and you only need the first one, then this will always work.

OpenVINO - Image classification

I tried to use OpenVINO Inference Engine to accelerate my DL inference. It works with one image. But I want to create a batch of two images and then do a inference.
This is my code:
InferenceEngine::Core core;
InferenceEngine::CNNNetwork network = core.ReadNetwork("path/to/model.xml");
InferenceEngine::InputInfo::Ptr input_info = network.getInputsInfo().begin()->second;
std::string input_name = network.getInputsInfo().begin()->first;
InferenceEngine::DataPtr output_info = network.getOutputsInfo().begin()->second;
std::string output_name = network.getOutputsInfo().begin()->first;
InferenceEngine::ExecutableNetwork executableNetwork = core.LoadNetwork(network, "CPU");
InferenceEngine::InferRequest inferRequest = executableNetwork.CreateInferRequest();
std::string input_image_01 = "path/to/image_01.png";
cv::Mat image_01 = cv::imread(input_image_01 );
InferenceEngine::Blob::Ptr imgBlob_01 = wrapMat2Blob(image_01);
std::string input_image_02 = "path/to/image_02.png";
cv::Mat image_02 = cv::imread(input_image_02 );
InferenceEngine::Blob::Ptr imgBlob_02 = wrapMat2Blob(image_02);
InferenceEngine::BlobMap imgBlobMap;
std::pair<std::string, InferenceEngine::Blob::Ptr> pair01(input_image_01, imgBlob_01);
imgBlobMap.insert(pair01);
std::pair<std::string, InferenceEngine::Blob::Ptr> pair02(input_image_02, imgBlob_02);
imgBlobMap.insert(pair02);
inferRequest.SetInput(imgBlobMap);
inferRequest.StartAsync();
inferRequest.Wait(InferenceEngine::IInferRequest::WaitMode::RESULT_READY);
InferenceEngine::Blob::Ptr output = inferRequest.GetBlob(output_name);
std::vector<unsigned> class_results;
ClassificationResult cls(output, {"x", "y"}, 2, 3);
class_results = cls.getResults();
Unfortunately, I received the following error message from the command
inferRequest.SetInput(imgBlobMap);
[NOT_FOUND] Failed to find input or output with name: 'path/to/image_02.png'
C:\j\workspace\private-ci\ie\build-windows-vs2019#2\b\repos\openvino\inference-engine\src\plugin_api\cpp_interfaces/impl/ie_infer_request_internal.hpp:303
C:\Program Files (x86)\Intel\openvino_2021.3.394\inference_engine\include\details/ie_exception_conversion.hpp:66
How can I create a batch of more than image, do a inference and get the information for classification class and confidence? Is the confidence and class located in the received variable of GetBlob()? Should I need the call of ClassificationResult cls(output, {"x", "y"}, 2, 3);?
I'd recommend you to review Using Shape Inference article from OpenVINO online documentation to be aware of the limitations of using batches. It also refers to Open Model Zoo smart_classroom_demo, where dynamic batching is used in processing multiple previously detected faces. Basically, when you have batch enabled in the model, the memory buffer of your input blob will be allocated to have a room for all batch of images, and your responsibility is to fill data in input blob for each image in batch from your data. You may take a look at function CnnDLSDKBase::InferBatch, of smart_classroom_demo, which is located at file smart_classroom_demo/cpp/src/cnn.cpp, line 51. As you can see, in the loop over num_imgs an auxiliary function matU8ToBlob fills the input blob with data for current_batch_size of images, then set batch size for infer request and run inference.
for (size_t batch_i = 0; batch_i < num_imgs; batch_i += batch_size) {
const size_t current_batch_size = std::min(batch_size, num_imgs - batch_i);
for (size_t b = 0; b < current_batch_size; b++) {
matU8ToBlob<uint8_t>(frames[batch_i + b], input, b);
}
if (config_.max_batch_size != 1)
infer_request_.SetBatch(current_batch_size);
infer_request_.Infer();
there is a similar sample using the batch inputs as input into model within the OpenVINO. You can refer to below link.
https://github.com/openvinotoolkit/openvino/blob/ae2913d3b5970ce0d3112cc880d03be1708f13eb/inference-engine/samples/hello_nv12_input_classification/main.cpp#L236

Tensorflow lite model output always gives same output no matter the input

My goal is to run a Keras model I have made in my ESP32 microcontroller. I have the libraries all working correctly.
I have created a Keras model using google Collab that looks to be working fine when I give it random test data within google Collab. The model has two input features and 4 different outputs.(a multiple-output regression model)
However, when I export and load the model into my c++ application in the ESP32 it does not matter what the inputs are, it always predicts the same output.
I have based myself in this code in order to load and run the model in c++ : https://github.com/tensorflow/tensorflow/blob/master/tensorflow/lite/micro/examples/magic_wand/main_functions.cc
And this is my version of the code
namespace {
tflite::ErrorReporter* error_reporter = nullptr;
const tflite::Model* model = nullptr;
tflite::MicroInterpreter* interpreter = nullptr;
TfLiteTensor* input = nullptr;
TfLiteTensor* output = nullptr;
int inference_count = 0;
// Create an area of memory to use for input, output, and intermediate arrays.
// Finding the minimum value for your model may require some trial and error.
constexpr int kTensorArenaSize = 2 * 2048;
uint8_t tensor_arena[kTensorArenaSize];
} // namespace
static void setup(){
static tflite::MicroErrorReporter micro_error_reporter;
error_reporter = &micro_error_reporter;
model = tflite::GetModel(venti_model);
if (model->version() != TFLITE_SCHEMA_VERSION) {
error_reporter->Report(
"Model provided is schema version %d not equal "
"to supported version %d.",
model->version(), TFLITE_SCHEMA_VERSION);
return;
}
// This pulls in all the operation implementations we need.
// NOLINTNEXTLINE(runtime-global-variables)
static tflite::ops::micro::AllOpsResolver resolver;
// Build an interpreter to run the model with.
static tflite::MicroInterpreter static_interpreter(
model, resolver, tensor_arena, kTensorArenaSize, error_reporter);
interpreter = &static_interpreter;
// Allocate memory from the tensor_arena for the model's tensors.
TfLiteStatus allocate_status = interpreter->AllocateTensors();
if (allocate_status != kTfLiteOk) {
error_reporter->Report("AllocateTensors() failed");
return;
}
// Obtain pointers to the model's input and output tensors.
input = interpreter->input(0);
ESP_LOGI("TENSOR SETUP", "input size = %d", input->dims->size);
ESP_LOGI("TENSOR SETUP", "input size in bytes = %d", input->bytes);
ESP_LOGI("TENSOR SETUP", "Is input float32? = %s", (input->type == kTfLiteFloat32) ? "true" : "false");
ESP_LOGI("TENSOR SETUP", "Input data dimentions = %d",input->dims->data[1]);
output = interpreter->output(0);
ESP_LOGI("TENSOR SETUP", "output size = %d", output->dims->size);
ESP_LOGI("TENSOR SETUP", "output size in bytes = %d", output->bytes);
ESP_LOGI("TENSOR SETUP", "Is input float32? = %s", (output->type == kTfLiteFloat32) ? "true" : "false");
ESP_LOGI("TENSOR SETUP", "Output data dimentions = %d",output->dims->data[1]);
}
static bool setupDone = true;
static void the_ai_algorithm_task(){
/* First time task is init setup the ai model */
if(setupDone == false){
setup();
setupDone = true;
}
/* Load the input data i.e deltaT1 and deltaT2 */
//int i = 0;
input->data.f[0] = 2.0; /* Different values dont change the output */
input->data.f[1] = 3.2;
// Run inference, and report any error
TfLiteStatus invoke_status = interpreter->Invoke();
if (invoke_status != kTfLiteOk) {
error_reporter->Report("Invoke failed");
// return;
}
/* Retrieve outputs Fan , AC , Vent 1 , Vent 2 */
double fan = output->data.f[0];
double ac = output->data.f[1];
double vent1 = output->data.f[2];
double vent2 = output->data.f[3];
ESP_LOGI("TENSOR SETUP", "fan = %lf", fan);
ESP_LOGI("TENSOR SETUP", "ac = %lf", ac);
ESP_LOGI("TENSOR SETUP", "vent1 = %lf", vent1);
ESP_LOGI("TENSOR SETUP", "vent2 = %lf", vent2);
}
The model seems to load ok as the dimensions and sizes are correct. But the output is always the same 4 values
fan = 0.0087
ac = 0.54
vent1 = 0.73
vent2 = 0.32
Any idea on what can be going wrong? Is it something about my model or am I just not using the model correctly in my c++ application?
Could you refer to the "Test the model" section here - https://colab.research.google.com/github/tensorflow/tensorflow/blob/master/tensorflow/lite/micro/examples/hello_world/train/train_hello_world_model.ipynb#scrollTo=f86dWOyZKmN9 and verify if the TFLite model is producing the correct results?
You can find the issue by testing the 1) TFModel (which you have done already) 2) TFLite model and then 3) TFLite Micro Model (C Source File)
You also need to verify that the inputs passed to the model are of the same type and distribution. eg: If your TFModel was trained on Images in the range 0-255, then you need to pass this to the TFLite and TFLite Micro Model. Instead, if you trained the model using preprocessed data (0-255 get normalized to 0-1 during training), then you need to do the same and preprocess the data for the TFLite and TFLite Micro model.
I have found the issue and the answer.
It was not the C++ code, it was the model. Originally, I made my model with 3 hidden layers of 64, 20, and 8 (I am new to ML so I was only playing with random values) and it was giving me the issue.
To solve it I just changed the hidden layers to 32, 16, and 8 and the C++ code was outputting right values.

How can I get layer's top label in c++?

1)Is it possible get each the layer's top labels (e.g: ip1,ip2,conv1,conv2) in c++?
If my layer is
layer {
name: "relu1_1"
type: "Input"
top: "pool1"
input_param {
shape: {
dim:1
dim: 1
dim: 28
dim: 28
}
}
}
I want to get the top label which in my case is "pool1"
I searched the examples provided, but I couldn't find anything. currently I'm able to get only the layers names and layer type by the following commands,
cout << "Layer name:" << "'" << net_->layer_names()[layer_index]<<endl;
cout << "Layer type: " << net_->layers()[layer_index]->type()<<endl;
2) Where can I find the tutorials or the examples which explains most used API's for using caffe framework using c++?
Thankyou in advance.
Look at Net class in doxygen:
const vector< vector< Blob< Dtype > * > > all_ tops = net_->top_vecs(); // get "top" of all layers
Blob<Dtype>* ptop = all_tops[layer_index][0]; // pointer to top blob of layer
If you want the layer's name, you can
const string layer_name = net_->layer_names()[layer_index];
You can access all sorts of names/data using net_ interface, just read the doc!

How create a DICOM image from byte (DCMTK)

I want to use the DCMTK 3.6.1 library in an existing project that can create DICOM image. I want to use this library because I want to make the compression of the DICOM images. In a new solution (Visual Studio 2013/C++) Following the example in the DCMTK official documentation, I have this code, that works properly.
using namespace std;
int main()
{
DJEncoderRegistration::registerCodecs();
DcmFileFormat fileformat;
/**** MONO FILE ******/
if (fileformat.loadFile("Files/test.dcm").good())
{
DcmDataset *dataset = fileformat.getDataset();
DcmItem *metaInfo = fileformat.getMetaInfo();
DJ_RPLossless params; // codec parameters, we use the defaults
// this causes the lossless JPEG version of the dataset
//to be created EXS_JPEGProcess14SV1
dataset->chooseRepresentation(EXS_JPEGProcess14SV1, &params);
// check if everything went well
if (dataset->canWriteXfer(EXS_JPEGProcess14SV1))
{
// force the meta-header UIDs to be re-generated when storing the file
// since the UIDs in the data set may have changed
delete metaInfo->remove(DCM_MediaStorageSOPClassUID);
delete metaInfo->remove(DCM_MediaStorageSOPInstanceUID);
metaInfo->putAndInsertString(DCM_ImplementationVersionName, "New Implementation Version Name");
//delete metaInfo->remove(DCM_ImplementationVersionName);
//dataset->remove(DCM_ImplementationVersionName);
// store in lossless JPEG format
fileformat.saveFile("Files/carrellata_esami_compresso.dcm", EXS_JPEGProcess14SV1);
}
}
DJEncoderRegistration::cleanup();
return 0;
}
Now I want to use the same code in an existing C++ application where
if (infoDicom.arrayImgDicom.GetSize() != 0) //Things of existing previous code
{
//I have added here the registration
DJEncoderRegistration::registerCodecs(); // register JPEG codecs
DcmFileFormat fileformat;
DcmDataset *dataset = fileformat.getDataset();
DJ_RPLossless params;
dataset->putAndInsertUint16(DCM_Rows, infoDicom.rows);
dataset->putAndInsertUint16(DCM_Columns, infoDicom.columns,);
dataset->putAndInsertUint16(DCM_BitsStored, infoDicom.m_bitstor);
dataset->putAndInsertUint16(DCM_HighBit, infoDicom.highbit);
dataset->putAndInsertUint16(DCM_PixelRepresentation, infoDicom.pixelrapresentation);
dataset->putAndInsertUint16(DCM_RescaleIntercept, infoDicom.rescaleintercept);
dataset->putAndInsertString(DCM_PhotometricInterpretation,"MONOCHROME2");
dataset->putAndInsertString(DCM_PixelSpacing, "0.086\\0.086");
dataset->putAndInsertString(DCM_ImagerPixelSpacing, "0.096\\0.096");
BYTE* pData = new BYTE[sizeBuffer];
LPBYTE pSorg;
for (int nf=0; nf<iNumberFrames; nf++)
{
//this contains all the PixelData and I put it into the dataset
pSorg = (BYTE*)infoDicom.arrayImgDicom.GetAt(nf);
dataset->putAndInsertUint8Array(DCM_PixelData, pSorg, sizeBuffer);
dataset->chooseRepresentation(EXS_JPEGProcess14SV1, &params);
//and I put it in my data set
//but this IF return false so che canWriteXfer fails...
if (dataset->canWriteXfer(EXS_JPEGProcess14SV1))
{
dataset->remove(DCM_MediaStorageSOPClassUID);
dataset->remove(DCM_MediaStorageSOPInstanceUID);
}
//the saveFile fails too, and the error is "Pixel
//rappresentation non found" but I have set the Pixel rep with
//dataset->putAndInsertUint16(DCM_PixelRepresentation, infoDicom.pixelrapresentation);
OFCondition status = fileformat.saveFile("test1.dcm", EXS_JPEGProcess14SV1);
DJEncoderRegistration::cleanup();
if (status.bad())
{
int error = 0; //only for test
}
thefile.Write(pSorg, sizeBuffer); //previous code
}
Actually I made test with image that have on one frame, so the for cycle is done only one time. I don't understand why if I choose dataset->chooseRepresentation(EXS_LittleEndianImplicit, &params); or dataset->chooseRepresentation(EXS_LittleEndianEXplicit, &params); works perfectly but not when I choose dataset->chooseRepresentation(EXS_JPEGProcess14SV1, &params);
If I use the same image in the first application, I can compress the image without problems...
EDIT: I think the main problem to solve is the status = dataset->chooseRepresentation(EXS_JPEGProcess14SV1, &rp_lossless) that return "Tag not found". How can I know wich tag is missed?
EDIT2: As suggest in the DCMTK forum I have added the tag about the Bits Allocated and now works for few images, but non for all. For some images I have again "Tag not found": how can I know wich one of tags is missing? As a rule it's better insert all the tags?
I solve the problem adding the tags DCM_BitsAllocated and DCM_PlanarConfiguration. This are the tags that are missed. I hope that is useful for someone.
At least you should call the function chooseRepresentation, after you have applied the data.
**dataset->putAndInsertUint8Array(DCM_PixelData, pSorg, sizeBuffer);**
dataset->chooseRepresentation(EXS_JPEGProcess14SV1, &params);