I'm trying to load a simple four-layer convolutional neural network from an ONNX file in C++ with OpenCV. The ONNX file was created from a TensorFlow model using the tf2onnx library in Python. I saved the model with the following piece of code.
(onnx_model_proto, storage) = tf2onnx.convert.from_keras(model, opset=8)
with open(os.path.join("models", 'upscaleModelData.onnx'), "wb") as f:
f.write(onnx_model_proto.SerializeToString())
When reading via cv::dnn::Net net = cv::dnn::readNetFromONNX("model.onnx"); in C++, I get the following error.
[ INFO:0] global ####\master_winpack-build-win64-vc15\opencv\modules\dnn\src\onnx\onnx_importer.cpp (429) cv::dnn::dnn4_v20210608::ONNXImporter::populateNet DNN/ONNX: loading ONNX v4 model produced by 'tf2onnx':1.10.0. Number of nodes = 13, inputs = 1, outputs = 1
OpenCV(4.5.3) Error: Unspecified error (Can't create layer "model/tf.nn.depth_to_space/DepthToSpace:0" of type "DepthToSpace") in cv::dnn::dnn4_v20210608::LayerData::getLayerInstance, file ####\opencv\modules\dnn\src\dnn.cpp, line 621
[ERROR:0] global ####\opencv\modules\dnn\src\onnx\onnx_importer.cpp (2127) cv::dnn::dnn4_v20210608::ONNXImporter::handleNode DNN/ONNX: ERROR during processing node with 1 inputs and 1 outputs: [DepthToSpace]:(model/tf.nn.depth_to_space/DepthToSpace:0)
[ INFO:0] global ####\opencv\modules\dnn\src\onnx\onnx_importer.cpp (2131) cv::dnn::dnn4_v20210608::ONNXImporter::handleNode Input[0] = 'model/conv2d_3/Relu:0'
[ INFO:0] global ####\opencv\modules\dnn\src\onnx\onnx_importer.cpp (2135) cv::dnn::dnn4_v20210608::ONNXImporter::handleNode Output[0] = 'model/tf.nn.depth_to_space/DepthToSpace:0'
OpenCV(4.5.3) Error: Unspecified error (> Node [DepthToSpace]:(model/tf.nn.depth_to_space/DepthToSpace:0) parse error: OpenCV(4.5.3) ####\opencv\modules\dnn\src\dnn.cpp:621: error: (-2:Unspecified error) Can't create layer "model/tf.nn.depth_to_space/DepthToSpace:0" of type "DepthToSpace" in function 'cv::dnn::dnn4_v20210608::LayerData::getLayerInstance'
> ) in cv::dnn::dnn4_v20210608::ONNXImporter::handleNode, file ####\opencv\modules\dnn\src\onnx\onnx_importer.cpp, line 2146
OpenCV(4.5.3) ####\opencv\modules\dnn\src\onnx\onnx_importer.cpp:2146: error: (-2:Unspecified error) in function 'cv::dnn::dnn4_v20210608::ONNXImporter::handleNode'
> Node [DepthToSpace]:(model/tf.nn.depth_to_space/DepthToSpace:0) parse error: OpenCV(4.5.3) ####\opencv\modules\dnn\src\dnn.cpp:621: error: (-2:Unspecified error) Can't create layer "model/tf.nn.depth_to_space/DepthToSpace:0" of type "DepthToSpace" in function 'cv::dnn::dnn4_v20210608::LayerData::getLayerInstance'
I'm using OpenCV on Windows and Visual C++.
This might be a similar issue to this: EMGU - EDSR : Can't create layer DepthToSpace.
I tried reading the same network from TensorFlow PB and H5 file, but I get a similar result.
For reading a PB file, created with cv::dnn::readNetFromTensorflow("model.pb") I get:
OpenCV(4.5.3) Error: Unspecified error (FAILED: fs.is_open(). Can't open "model.pb") in cv::dnn::ReadProtoFromBinaryFile, file ####\opencv\modules\dnn\src\caffe\caffe_io.cpp, line 1133
OpenCV(4.5.3) ####\opencv\modules\dnn\src\caffe\caffe_io.cpp:1133: error: (-2:Unspecified error) FAILED: fs.is_open(). Can't open "model.pb" in function 'cv::dnn::ReadProtoFromBinaryFile'
For reading a H5 file created with cv::dnn::readNetFromTensorflow("upscaleModelData.h5"); I get:
OpenCV(4.5.3) Error: Unspecified error (Cannot determine an origin framework of files: model.h5) in cv::dnn::dnn4_v20210608::readNet, file ####\opencv\modules\dnn\src\dnn.cpp, line 5461
OpenCV(4.5.3) ####\opencv\modules\dnn\src\dnn.cpp:5461: error: (-2:Unspecified error) Cannot determine an origin framework of files: model.h5 in function 'cv::dnn::dnn4_v20210608::readNet'
Does this mean I should make modifications to the model's layers so that it can be read by OpenCV? Is this a compatibility issue? Any feedback, workarounds or alternative approaches (e.g. TensorFlow C++ API solutions) are welcome.
Edit 1: Implementing a layer-type workaround
I reimplemented the depth_to_space layer manually in Python, using the following piece of code, based on these links: onnx-tensorflow, depth_to_space, keras-subpixel-conv.
x_shape = tf.shape(x)
n, h, w, c = x_shape[0], x_shape[1], x_shape[2], x_shape[3]
y = tf.reshape(x, (n, h, w, bs, bs, c // (bs ** 2)))
y = tf.transpose(y, (0, 1, 3, 2, 4, 5))
outputs = tf.reshape(y, (n, h * bs, w * bs, c // (bs ** 2)))
Now, when I run the same C++ code, I get the following error.
[ INFO:0] global ####\opencv\modules\dnn\src\onnx\onnx_importer.cpp (429) cv::dnn::dnn4_v20210608::ONNXImporter::populateNet DNN/ONNX: loading ONNX v4 model produced by 'tf2onnx':1.10.0. Number of nodes = 38, inputs = 14, outputs = 1
OpenCV(4.5.3) Error: Assertion failed (indexMat.total() == 1) in cv::dnn::dnn4_v20210608::ONNXImporter::handleNode, file ####\opencv\modules\dnn\src\onnx\onnx_importer.cpp, line 1842
[ERROR:0] global ####\opencv\modules\dnn\src\onnx\onnx_importer.cpp (2127) cv::dnn::dnn4_v20210608::ONNXImporter::handleNode DNN/ONNX: ERROR during processing node with 2 inputs and 1 outputs: [Gather]:(model/tf.compat.v1.shape/Shape:0)
[ INFO:0] global ####\opencv\modules\dnn\src\onnx\onnx_importer.cpp (2131) cv::dnn::dnn4_v20210608::ONNXImporter::handleNode Input[0] = 'Shape__72:0'
[ INFO:0] global ####\opencv\modules\dnn\src\onnx\onnx_importer.cpp (2131) cv::dnn::dnn4_v20210608::ONNXImporter::handleNode Input[1] = 'Const__76'
[ INFO:0] global ####\opencv\modules\dnn\src\onnx\onnx_importer.cpp (2135) cv::dnn::dnn4_v20210608::ONNXImporter::handleNode Output[0] = 'model/tf.compat.v1.shape/Shape:0'
OpenCV(4.5.3) Error: Unspecified error (> Node [Gather]:(model/tf.compat.v1.shape/Shape:0) parse error: OpenCV(4.5.3) ####\opencv\modules\dnn\src\onnx\onnx_importer.cpp:1842: error: (-215:Assertion failed) indexMat.total() == 1 in function 'cv::dnn::dnn4_v20210608::ONNXImporter::handleNode'
> ) in cv::dnn::dnn4_v20210608::ONNXImporter::handleNode, file ####\opencv\modules\dnn\src\onnx\onnx_importer.cpp, line 2146
OpenCV(4.5.3) ####\opencv\modules\dnn\src\onnx\onnx_importer.cpp:2146: error: (-2:Unspecified error) in function 'cv::dnn::dnn4_v20210608::ONNXImporter::handleNode'
> Node [Gather]:(model/tf.compat.v1.shape/Shape:0) parse error: OpenCV(4.5.3) ####\opencv\modules\dnn\src\onnx\onnx_importer.cpp:1842: error: (-215:Assertion failed) indexMat.total() == 1 in function 'cv::dnn::dnn4_v20210608::ONNXImporter::handleNode'
Related
Hello I have problem with applying tensorflow model in OpenCV. The code below properly load model, but when calling forward method Assertion error is thrown. Do you have any ideas where is the problem? Or how to debug/find it?
cv::dnn::Net net;
string path;
path = "graph.pb";
net = cv::dnn::readNetFromTensorflow(path);
if (net.empty())
{
std::cerr << "Can't load network by using the given files." << std::endl;
return ;
}
Mat image = imread(imagePath)
Mat inputBlob = cv::dnn::blobFromImage(image, 1.0, Size(512, 512), Scalar(0,0,0), true, false);
int N = inputBlob.size[0], C = inputBlob.size[1], H = inputBlob.size[2], W = inputBlob.size[3]; // [1, 3, 512, 512]
net.setInput(inputBlob); //set the network input
Mat output = net.forward(); // <- throws error
Error:
Debug Assertion Failed!
Program: C:\Workspace\ImageAnalysisPlus\x64\Debug\opencv_world3410d.dll File: C:\Program Files (x86)\Microsoft Visual Studio\2017\Professional\VC\Tools\MSVC\14.16.27023\include\vector Line: 1789
Expression: back() called on empty vector
For information on how your program can cause an assertion failure, see the Visual C++ documentation on asserts.
OpenCV version: 4.4.0
Tensorflow model: https://drive.google.com/file/d/1aE0smAw-CyPLch6UY8blK3RreT5RrZfN/view?usp=sharing
Platform: Windows 10, Visual Studio 2017
I tried to load the model in python and OpenCV and it works.
Thank you in advance for any advice.
I have to admit that I didn't try the posted minimal code. It was part of a little bigger code. The minimal code works also on my machine. Knowing that I found out that I was calling the forward method on uninitialized network which caused the error. I initialized local net variable in init method in a class instead of a class attribute.
I'm using tensorflow C++ API for inference. I was able to build a standalone tensorflow library, loading the optimized graph, and feeding an image.
I'm able to print the tensor using the following code:
// run the image through the model.
std::vector<Tensor> outputs;
Status run_status = session->Run({{input_layer, resized_tensor}}, {output_layer}, {}, &outputs);
if (!run_status.ok()) {
LOG(ERROR) << "Running model failed: " << run_status;
return -1;
}
std::cout << outputs[0].DebugString() << "\n";
and then I get the following output:
Tensor<type: float shape: [1,480,720,3] values: [[[58.4225044 79.0690613 94.4875641]]]...>
but after adding this line:
cv::Mat rotMatrix(outputs[0].dim_size(1), outputs[0].dim_size(2), CV_32FC1, outputs[0].flat<float>().data());
or this line:
float *p = outputs[0].flat<float>().data();
I get the following error:
2019-06-26 14:19:29.040705: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
[libprotobuf ERROR external/com_google_protobuf/src/google/protobuf/descriptor_database.cc:118] File already exists in database: google/protobuf/any.proto
[libprotobuf FATAL external/com_google_protobuf/src/google/protobuf/descriptor.cc:1367] CHECK failed: GeneratedDatabase()->Add(encoded_file_descriptor, size):
terminate called after throwing an instance of 'google::protobuf::FatalException'
what(): CHECK failed: GeneratedDatabase()->Add(encoded_file_descriptor, size):
Aborted (core dumped)
I tried it using bazel, qmake, and cmake and built protocol buffer again from source but without any help..
I am looking to do a simple read from an hdf5 file using C++. I will split this up into 4 parts. 1st what the file looks like. 2nd my code which attempts to read the file. 3rd the error message. 4th my conclusions.
1.The File - The dataset can be found in the file as shown:
$ h5ls -r myfile.h5
/ Group
/mydata Dataset {1200}
Note - The dataset is an array of 1200 strings. Note the CTYPE is H5T_C_S1 which is what I will use to read it in.
HDF5 "myfile.h5" {
GROUP "/" {
DATASET "mydata" {
DATATYPE H5T_STRING {
STRSIZE H5T_VARIABLE;
STRPAD H5T_STR_NULLTERM;
CSET H5T_CSET_UTF8;
CTYPE H5T_C_S1;
}
DATASPACE SIMPLE { ( 1200 ) / ( 1200 ) }
DATA {
(0): "pxsntpfcnkeesswwpwopksu", "exsytafcbkecsswwpwopnng",
(2): "ebswtlfcbnecsswwpwopnnm", "pxywtpfcnneesswwpwopksu",
(4): "exsgfnfwbktesswwpwoenag", "exyytafcbnecsswwpwopkng",
2.The Code - My code attempts to read the dataset like so:
#include "H5Cpp.h"
#ifndef H5_NO_NAMESPACE
using namespace H5;
#endif
const H5std_string FILE_NAME("myfile.h5");
const H5std_string DATASET_NAME("mydata");
// open file
H5File file(FILE_NAME, H5F_ACC_RDONLY);
// get dataset
DataSet dataset = file.openDataSet(DATASET_NAME);
// get src dataspace
DataSpace src = dataset.getSpace();
// get dimensions
int NUM_DIMS = src.getSimpleExtentNdims();
std::vector<hsize_t> dims(NUM_DIMS);
src.getSimpleExtentDims(&dims[0]);
hsize_t height = dims[0];
hsize_t width = 23;
// define src hyperslab
std::vector<hsize_t> count(NUM_DIMS, 1);
std::vector<hsize_t> offset(NUM_DIMS, 0);
src.selectHyperslab(H5S_SELECT_SET, &count[0], &offset[0]);
// define dst hyperslab
DataSpace dst(NUM_DIMS, dims);
dst.selectHyperslab(H5S_SELECT_SET, &count[0], &offset[0]);
// read data into memory, array of cstrings
std::vector<char*> data_out(height);
dataset.read(&data_out[0], H5T_C_S1, dst, src);
// print first line
std::cout << data_out[0] << std::endl;
3.The Error - However, it fails from what appears to be a type mismatch between the src and dst hyperslabs, even though I designed the src and dst to have the same dimensions. The error message is as follows:
HDF5-DIAG: Error detected in HDF5 (1.10.3) thread 0:
#000: H5Dio.c line 199 in H5Dread(): can't read data
major: Dataset
minor: Read failed
#001: H5Dio.c line 467 in H5D__read(): unable to set up type info
major: Dataset
minor: Unable to initialize object
#002: H5Dio.c line 993 in H5D__typeinfo_init(): unable to convert between src and dest datatype
major: Dataset
minor: Feature is unsupported
#003: H5T.c line 4546 in H5T_path_find(): can't find datatype conversion path
major: Datatype
minor: Can't get value
#004: H5T.c line 4762 in H5T__path_find_real(): no appropriate function for conversion path
major: Datatype
minor: Unable to initialize object
HDF5-DIAG: Error detected in HDF5 (1.10.3) thread 0:
#000: H5T.c line 1756 in H5Tclose(): immutable datatype
major: Invalid arguments to routine
minor: Bad value
DataType::~DataType - H5Tclose failed
4.My Conclusions - I have attempted many variations, including removing dst and src as parameters for dataset.read(), changing H5T_C_S1 to PredType::C_S1 and PredType::NATIVE_CHAR, however the same error persists.
How do I simply read the dataset into memory? Is the datatype truly mismatched or is there something else I am not defining? Am I still using the wrong datatype in the read function? Am I defining my hyperslabs improperly such that there actually is a type mismatch?
Maybe you want to try out HDFql and abstract yourself from HDF5 low-level details. In C++ using HDFql, you could read your variable-length char dataset mydata (contained in file myfile.h5) like this:
HDFql::execute("SELECT FROM myfile.h5 mydata"); // select (i.e. read) dataset "mydata" from file "myfile.h5" and populate default cursor with it
while(HDFql::cursorNext() == HDFql::Success) // display content of default cursor
{
std::cout << HDFql::cursorGetChar() << std::endl;
}
I am attempting a build of one of our GUI projects. This build works on every machine in the office save the two new machines. Here is the terminal output of the error:
viewPriorityChanged( 11, 0 )
viewPriorityChanged( 12, 0 )
viewPriorityChanged( 13, 0 )
MainWindow.cpp debug print #5
terminate called after throwing an instance of 'std::bad_alloc'
what(): std::bad_alloc
./runlp_common: line 320: 11123 Aborted (core dumped) $3
As you can see I have traced it down in the code with print statements. Here is the method that is erring in the code:
void MainWindow::updateDecoration()
{
Geometry::DisplayTypes closest = Geometry::closestDisplayType();
QString closestStr = Geometry::displayTypeString(closest).toLower();
QString iconPath = QString(QLatin1String(":/images/device_skin_%1.png")).
arg(closestStr);
Messages::conout() << "MainWindow.cpp debug print #5" << std::endl;
m_device = QPixmap(iconPath); <-- not making it past here
Messages::conout() << "MainWindow.cpp debug print #6" << std::endl;
QBitmap clBitmap = m_device.createHeuristicMask();
setFixedSize(m_device.size());
setMask(clBitmap);
update();
}
It is the strangest thing (in my opinion) that this line would be causing just a few machines trouble. Any ideas what could be going on here?
EDIT:
I wanted to give a more detailed description of the terminal output:
MainWindow.cpp debug print #1
MainWindow.cpp debug print #2
ToolsViewConfigTip_Series: Preloading tip images
ToolsViewConfigTip_Capacity: Preloading tip images.
ln: cannot remove '/etc/localtime': Permission denied
Using QGLBridge
Using QGLBridge
QThread::start: Thread creation error: Resource temporarily unavailable
libpng error: Read Error
libpng error: Read Error
libpng error: Read Error
libpng error: Read Error
libpng error: Read Error
libpng error: Read Error
libpng error: Read Error
libpng error: Read Error
QImage::scaled: Image is a null image
viewPriorityChanged( 0, 0 )
viewPriorityChanged( 1, 0 )
viewPriorityChanged( 2, 0 )
viewPriorityChanged( 3, 0 )
viewPriorityChanged( 4, 0 )
viewPriorityChanged( 5, 0 )
viewPriorityChanged( 6, 0 )
viewPriorityChanged( 7, 0 )
viewPriorityChanged( 10, 0 )
viewPriorityChanged( 11, 0 )
viewPriorityChanged( 12, 0 )
viewPriorityChanged( 13, 0 )
MainWindow.cpp debug print #4
MainWindow.cpp debug print #5
terminate called after throwing an instance of 'std::bad_alloc'
what(): std::bad_alloc
./runlp_common: line 320: 22676 Aborted (core dumped) $3
----------- KillAllApps -----------
IsobusMuxServer::clientDisconnected() Client 1 is now disconnected and destroyed
IsobusMuxServer::clientDisconnected() Client 0 is now disconnected and destroyed
-----------------------------------
I took the suggestion about using valgrind in the comments. Prior to this point in the program there actually are no alllocs, frees and no bytes are allocated. I tried running the other various tools in the valgrind package without any luck. The only other error I get is when running helgrind:
ProprietarySocketServer::startServer() bind: Address already in use
ProprietarySocketServer::startServer() Could not bind on port 41263
I'm trying to use the cvLogPolar method, but I always get the following error:
OpenCV Error: Bad flag (parameter or structure field) (Unrecognized or unsupported array type) in cvGetMat, file /tmp/opencv-pcqRHK/opencv-2.4.10.1/modules/core/src/array.cpp, line 2482
libc++abi.dylib: terminating with uncaught exception of type cv::Exception: /tmp/opencv-pcqRHK/opencv-2.4.10.1/modules/core/src/array.cpp:2482: error: (-206) Unrecognized or unsupported array type in function cvGetMat
My code:
cv::Mat logpolar_frame(size, CV_8UC3), bgr_frame;
for(;;) {
capture >> bgr_frame;
if( bgr_frame.empty() ) {
break;
}
cv::imshow( "Example2_10", bgr_frame );
cvLogPolar(&bgr_frame, &logpolar_frame, cv::Point2f(bgr_frame.cols / 2, bgr_frame.rows / 2), 40);
}
I already tried to copy the bgr_frame inside the loop, but this results in the same error.
you need a bloody IplImage* for the arcane c-api legacy functions.
IplImage ipl_in = bgr_frame;
IplImage ipl_out = logpolar_frame;
cvLogPolar(&ipl_in, &ipl_out, ...)
(for 3.0, avoid all of it, and use cv::logPolar(), unfortunately not available in opencv2.4)