Multiple input, single output with tensorrt c++ api - c++

Question

I have a model which is made of 3 inputs and 1 output. I generated the TRT engine from my onnx model as shown below
int main() {
int maxBatchSize = 32;
nvinfer1::IBuilder* builder = nvinfer1::createInferBuilder(gLogger);
const auto explicitBatch = 1U << static_cast<uint32_t>(nvinfer1::NetworkDefinitionCreationFlag::kEXPLICIT_BATCH);
nvinfer1::INetworkDefinition* network = builder->createNetworkV2(explicitBatch);
nvonnxparser::IParser* parser = nvonnxparser::createParser(*network, gLogger);
parser->parseFromFile("model3.onnx", 1);
for (int i = 0; i < parser->getNbErrors(); ++i)
{
std::cout << parser->getError(i)->desc() << std::endl;
}
builder->setMaxBatchSize(maxBatchSize);
nvinfer1::IBuilderConfig* config = builder->createBuilderConfig();
config->setMaxWorkspaceSize(1 << 20);
nvinfer1::ICudaEngine* engine = builder->buildEngineWithConfig(*network, *config);
parser->destroy();
network->destroy();
config->destroy();
builder->destroy();
nvinfer1::IHostMemory* serializedModel = engine->serialize();
std::ofstream engine_file("model.engine");
engine_file.write((const char*)serializedModel->data(), serializedModel->size());
serializedModel->destroy();
return 0;
}
How can I perform inference since I have multiple inputs? In the Nvidia guide, only the scenario for single input single output is given.

Answer 1

You can make use of the files that are shipped with the TensorRT installations.
Check out the C:\TensorRT\samples\common directory. Take a look at the buffers.h header file.
Take a look at the MNIST example in the same directory which uses the buffers.h.
The buffers.h file takes care of multiple inputs or outputs.
It creates a BufferManager to deal with those inputs and outputs.