Missing file "tensorflow/core/framework/types.pb.h" when building TensorFlow r1.14 C++ files - c++

Question

I have trained a keras machine learning model in python tensorflow 1.14 and my goal is to use that trained model to aid in performing fluid simulation calculations in C++. For this reason, I would like to use the tensorflow C++ API to run the previously trained model. I have been following the tutorial found here: http://www.bitbionic.com/2017/08/18/run-your-keras-models-in-c-tensorflow/ (however using r1.14 instead of r1.1 - but note I have tried with r1.1 as well). I am using Python 2.7, GCC 9.2.1, Bazel 0.24.1 - this is what is tested by Google here: https://www.tensorflow.org/install/source with the exception of GCC (4.8 by google).
The steps I have taken are as follows:
Clone tensorflow and checkout r1.14
Configure with default settings:
WARNING: --batch mode is deprecated. Please instead explicitly shut down your Bazel server using the command "bazel shutdown".
You have bazel 0.24.1 installed.
Please specify the location of python. [Default is /usr/bin/python]:
Found possible Python library paths:
/usr/local/lib/python2.7/dist-packages
/usr/lib/python2.7/dist-packages
Please input the desired Python library path to use. Default is [/usr/local/lib/python2.7/dist-packages]
Do you wish to build TensorFlow with XLA JIT support? [Y/n]:
XLA JIT support will be enabled for TensorFlow.
Do you wish to build TensorFlow with OpenCL SYCL support? [y/N]:
No OpenCL SYCL support will be enabled for TensorFlow.
Do you wish to build TensorFlow with ROCm support? [y/N]:
No ROCm support will be enabled for TensorFlow.
Do you wish to build TensorFlow with CUDA support? [y/N]:
No CUDA support will be enabled for TensorFlow.
Do you wish to download a fresh release of clang? (Experimental) [y/N]:
Clang will not be downloaded.
Do you wish to build TensorFlow with MPI support? [y/N]:
No MPI support will be enabled for TensorFlow.
Please specify optimization flags to use during compilation when bazel option "--config=opt" is specified [Default is -march=native -Wno-sign-compare]:
Would you like to interactively configure ./WORKSPACE for Android builds? [y/N]:
Not configuring the WORKSPACE for Android builds.
Preconfigured Bazel build configs. You can use any of the below by adding "--config=<>" to your build command. See .bazelrc for more details.
--config=mkl # Build with MKL support.
--config=monolithic # Config for mostly static monolithic build.
--config=gdr # Build with GDR support.
--config=verbs # Build with libverbs support.
--config=ngraph # Build with Intel nGraph support.
--config=numa # Build with NUMA support.
--config=dynamic_kernels # (Experimental) Build kernels into separate shared objects.
Preconfigured Bazel build configs to DISABLE default on features:
--config=noaws # Disable AWS S3 filesystem support.
--config=nogcp # Disable GCP support.
--config=nohdfs # Disable HDFS support.
--config=noignite # Disable Apache Ignite support.
--config=nokafka # Disable Apache Kafka support.
--config=nonccl # Disable NVIDIA NCCL support.
Configuration finished
I was recieving a build error (shown below), so I patched this release to fix an error I was receiving with what is described here: https://github.com/clearlinux/distribution/issues/1151 (not included in tutorial)
ERROR: /home/mark/.cache/bazel/_bazel_mark/0de26f992c566e8f868a9b2c0c3da3e9/external/grpc/BUILD:507:1: C++ compilation of rule '#grpc//:gpr_base' failed (Exit 1): gcc failed: error executing command
(cd /home/mark/.cache/bazel/_bazel_mark/0de26f992c566e8f868a9b2c0c3da3e9/execroot/org_tensorflow && \
exec env - \
PATH=/home/mark/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/usr/local/cuda/bin:/snap/bin \
PWD=/proc/self/cwd \
PYTHON_BIN_PATH=/usr/bin/python \
PYTHON_LIB_PATH=/usr/local/lib/python2.7/dist-packages \
TF_CONFIGURE_IOS=0 \
TF_DOWNLOAD_CLANG=0 \
TF_NEED_CUDA=0 \
TF_NEED_OPENCL_SYCL=0 \
TF_NEED_ROCM=0 \
/usr/bin/gcc -U_FORTIFY_SOURCE -fstack-protector -Wall -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -g0 -O2 '-D_FORTIFY_SOURCE=1' -DNDEBUG -ffunction-sections -fdata-sections '-std=c++0x' -MD -MF bazel-out/k8-opt/bin/external/grpc/_objs/gpr_base/log_linux.pic.d '-frandom-seed=bazel-out/k8-opt/bin/external/grpc/_objs/gpr_base/log_linux.pic.o' -fPIC '-DGRPC_ARES=0' -iquote external/grpc -iquote bazel-out/k8-opt/genfiles/external/grpc -iquote bazel-out/k8-opt/bin/external/grpc -isystem external/grpc/include -isystem bazel-out/k8-opt/genfiles/external/grpc/include -isystem bazel-out/k8-opt/bin/external/grpc/include -mavx '-mfpmath=both' -msse4.2 -fno-canonical-system-headers -Wno-builtin-macro-redefined '-D__DATE__="redacted"' '-D__TIMESTAMP__="redacted"' '-D__TIME__="redacted"' -c external/grpc/src/core/lib/gpr/log_linux.cc -o bazel-out/k8-opt/bin/external/grpc/_objs/gpr_base/log_linux.pic.o)
Execution platform: #bazel_tools//platforms:host_platform
external/grpc/src/core/lib/gpr/log_linux.cc:43:13: error: ambiguating new declaration of 'long int gettid()'
43 | static long gettid(void) { return syscall(__NR_gettid); }
| ^~~~~~
In file included from /usr/include/unistd.h:1170,
from external/grpc/src/core/lib/gpr/log_linux.cc:41:
/usr/include/x86_64-linux-gnu/bits/unistd_ext.h:34:16: note: old declaration '__pid_t gettid()'
34 | extern __pid_t gettid (void) __THROW;
| ^~~~~~
external/grpc/src/core/lib/gpr/log_linux.cc:43:13: warning: 'long int gettid()' defined but not used [-Wunused-function]
43 | static long gettid(void) { return syscall(__NR_gettid); }
| ^~~~~~
Target //tensorflow:libtensorflow_cc.so failed to build
INFO: Elapsed time: 513.481s, Critical Path: 35.74s
INFO: 1907 processes: 1907 local.
FAILED: Build did NOT complete successfully
The patch application: downloaded https://github.com/clearlinux-pkgs/tensorflow/blob/master/Add-grpc-fix-for-gettid.patch and then from tensorflow root
>> git am Add-grpc-fix-for-gettid.patch
Bazel build process then completes with no error and creation of .so file. I would show output here but it's very long - if needed I can paste it somewhere.
bazel build --jobs=4 --verbose_failures -c opt --copt=-mavx --copt=-mfpmath=both --copt=-msse4.2 //tensorflow:libtensorflow_cc.so
Made a project with the main.cpp file in the tutorial (found here: https://github.com/tensorflow/tensorflow/blob/master/tensorflow/examples/label_image/main.cc) and a CMakeLists file:
cmake_minimum_required(VERSION 3.16)
project(TensorflowTesting)
set(CMAKE_CXX_STANDARD 14)
#link to tensorflow headers
include_directories(SYSTEM /home/mark/Research/tensorflow_test)
include_directories(SYSTEM /home/mark/Research/tensorflow_test/bazel-tensorflow_test/external/eigen_archive)
include_directories(SYSTEM /home/mark/Research/tensorflow_test/bazel-tensorflow_test/external/protobuf/src)
include_directories(SYSTEM /home/mark/Research/tensorflow_test/bazel-tensorflow_test/bazel-genfiles)
#link to shared object
link_directories(/home/mark/Research/tensorflow_test/bazel-bin/tensorflow) #have to link to the .a and .so files
add_executable(TensorflowTesting main.cpp)
target_link_libraries(TensorflowTesting tensorflow_cc)
When I try to build this, it gives me the error:
In file included from /home/mark/Research/tensorflow_test/tensorflow/core/framework/tensor.h:22,
from /home/mark/Research/tensorflow_test/tensorflow/cc/framework/ops.h:21,
from /home/mark/Research/tensorflow_test/tensorflow/cc/ops/const_op.h:19,
from /home/mark/CLionProjects/TensorflowTesting/main.cpp:38:
/home/mark/Research/tensorflow_test/tensorflow/core/framework/allocator.h:24:10: fatal error: absl/strings/string_view.h: No such file or directory
24 | #include "absl/strings/string_view.h"
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
make[3]: *** [CMakeFiles/TensorflowTesting.dir/build.make:63: CMakeFiles/TensorflowTesting.dir/main.cpp.o] Error 1
make[2]: *** [CMakeFiles/Makefile2:76: CMakeFiles/TensorflowTesting.dir/all] Error 2
make[1]: *** [CMakeFiles/Makefile2:83: CMakeFiles/TensorflowTesting.dir/rule] Error 2
make: *** [Makefile:118: TensorflowTesting] Error 2
And I was able to get around this error by the suggestion here: https://github.com/tensorflow/tensorflow/issues/22007 (added in root of tf) - However this seems wrong that I need to manually include this library.
git clone https://github.com/abseil/abseil-cpp.git
ln -s abseil-cpp/absl ./absl
Now when I try to build this, I get an error:
In file included from /home/mark/Research/tensorflow_test/tensorflow/core/framework/tensor.h:23,
from /home/mark/Research/tensorflow_test/tensorflow/cc/framework/ops.h:21,
from /home/mark/Research/tensorflow_test/tensorflow/cc/ops/const_op.h:19,
from /home/mark/CLionProjects/TensorflowTesting/main.cpp:38:
/home/mark/Research/tensorflow_test/tensorflow/core/framework/tensor_shape.h:22:10: fatal error: tensorflow/core/framework/types.pb.h: No such file or directory
22 | #include "tensorflow/core/framework/types.pb.h"
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
make[3]: *** [CMakeFiles/TensorflowTesting.dir/build.make:63: CMakeFiles/TensorflowTesting.dir/main.cpp.o] Error 1
make[2]: *** [CMakeFiles/Makefile2:76: CMakeFiles/TensorflowTesting.dir/all] Error 2
make[1]: *** [CMakeFiles/Makefile2:83: CMakeFiles/TensorflowTesting.dir/rule] Error 2
make: *** [Makefile:118: TensorflowTesting] Error 2
I have tried many suggestions to fix this. Including also trying to run tensorflow/contrib/makefile/build_all_linux.sh, but when I do, I recieve the error:
remote: Total 73632 (delta 33), reused 36 (delta 12), pack-reused 73529
Receiving objects: 100% (73632/73632), 60.29 MiB | 12.97 MiB/s, done.
Resolving deltas: 100% (50705/50705), done.
Note: checking out '5902e759108d14ee8e6b0b07653dac2f4e70ac73'.
You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.
If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:
git checkout -b <new-branch-name>
HEAD is now at 5902e7591 Add missing declaration in protobuf_headers
Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/benchmark'
Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/googletest'
Cloning into '/home/mark/Research/tensorflow_test/tensorflow/contrib/makefile/downloads/protobuf/third_party/benchmark'...
Cloning into '/home/mark/Research/tensorflow_test/tensorflow/contrib/makefile/downloads/protobuf/third_party/googletest'...
Submodule path 'third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8'
Submodule path 'third_party/googletest': checked out 'c3bb0ee2a63279a803aaad956b9b26d74bf9e6e2'
cat: third_party/eigen3/gebp_neon.patch: No such file or directory
download_dependencies.sh completed successfully.
g++ -M -std=c++11 -DNSYNC_USE_CPP11_TIMEPOINT -DNSYNC_ATOMIC_CPP11 -I../../platform/c++11.futex -I../../platform/c++11 -I../../platform/gcc -I../../platform/posix -pthread -I../../public -I../../internal ../../internal/*.c ../../testing/*.c ../../platform/linux/src/nsync_semaphore_futex.c ../../platform/c++11/src/per_thread_waiter.cc ../../platform/c++11/src/yield.cc ../../platform/c++11/src/time_rep_timespec.cc ../../platform/c++11/src/nsync_panic.cc \
../../platform/c++11/src/start_thread.cc > dependfile
g++ -DNSYNC_USE_CPP11_TIMEPOINT -DNSYNC_ATOMIC_CPP11 -I../../platform/c++11.futex -I../../platform/c++11 -I../../platform/gcc -I../../platform/posix -pthread -I../../public -I../../internal -O -std=c++11 -Werror -Wall -Wextra -pedantic -c ../../internal/common.c
g++ -DNSYNC_USE_CPP11_TIMEPOINT -DNSYNC_ATOMIC_CPP11 -I../../platform/c++11.futex -I../../platform/c++11 -I../../platform/gcc -I../../platform/posix -pthread -I../../public -I../../internal -O -std=c++11 -Werror -Wall -Wextra -pedantic -c ../../internal/counter.c
../../internal/counter.c: In function ‘nsync::nsync_counter_s_* nsync::nsync_counter_new(uint32_t)’:
../../internal/counter.c:39:28: error: ‘void* memset(void*, int, size_t)’ clearing an object of type ‘struct nsync::nsync_counter_s_’ with no trivial copy-assignment; use value-initialization instead [-Werror=class-memaccess]
39 | memset (c, 0, sizeof (*c));
| ^
../../internal/counter.c:29:8: note: ‘struct nsync::nsync_counter_s_’ declared here
29 | struct nsync_counter_s_ {
| ^~~~~~~~~~~~~~~~
cc1plus: all warnings being treated as errors
make: *** [../../platform/posix/make.common:72: counter.o] Error 1
I'm not sure where to go from here and I was hoping to get a suggestion. I would be happy to follow another tutorial if one is suggested here as this one may be out of date. I appreciate any help!

Answer 1

After frustration with the above, I took a different route and ended up using https://github.com/FloopCZ/tensorflow_cc to solve the problem. I had to change versions of tensorflow to v1.15.0. Following the readme and by manually disabling CUDA support in CMake, I was able to compile and install TensorFlow to be used by CMake without bazel. I'm pretty sure I was close with my previous methodology but it was beginning to get messy and the current solution seems more practical for me.