I want to use the HDF5 C++ bindings in a project build with CMake. So I do the usual:
find_package (HDF5 REQUIRED COMPONENTS CXX)
target_link_libraries(foo PUBLIC ${HDF5_LIBRARIES})
target_include_directories(foo PUBLIC ${HDF5_INCLUDE_DIRS})
This used to work till our cluster (HPC) was upgraded.
Now I get errors during linking:
function MPI::Win::Set_name(char const*): error: undefined reference to 'MPI_Win_set_name'
function MPI::Win::Set_attr(int, void const*): error: undefined reference to 'MPI_Win_set_attr'
Although the versions of HDF5 did not change, the new one seems to require linking against MPI which CMake does not tell me/does automatically.
Am I missing anything? Is the CMake FindHDF5 module flawed or am I required to link against MPI manually when HDF5_IS_PARALLEL was set? How is it possible, that I now need to link MY application against mpi?
Some checks I did:
ldd on both hdf5 libraries shows libmpi
there is no -lmpi on either system for my app
HDF5 1.10.1 is used on both, both build against OpenMPI 2.1.2 with GCC 6.4.0
mpicxx -show shows different output: The new one includes -lmpi_cxx, the old not.
h5c++ -show seems to be the same (some other paths of course)
TL&DR: When HDF_IS_PARALLEL is true, one needs to link against MPI even when not using it.
The HDF5 compiler wrapper calls the MPI compiler wrapper which would add that automatically but the CMake module does not follow this path. Read further for how I found that which might help for similar issues.
I found the solution by isolating the compiler commands invoked to the bare minimum. Using grep I found references to MPI_* already in the object file of the cpp source file. This eliminated the possibility of a relation to the libraries linked, so only differences in includes were possible. I compared the new and old HDF5 include directories with diff -qr and found them to be the same.
Being sure it was the headers I examined the preprocessed files (g++ -E). I compared the new vs old one with vimdiff after some extra steps (replaced header include paths that changed from old to new system to keep the clutter to a minimum) Searching for mpi I found the only difference to be the inclusion of mpi_cxx. This is done by mpi.h which was also easily visible from the preprocessed output.
Checking the MPI installation on both systems confirmed, that new one was build with mpi_cxx support (C++ bindings for MPI added in MPI2 and removed in MPI3, but still optionally available as it seems) and the old one without.
As the C headers only have declarations no references land in the sources but the C++ bindings have definitions. This caused the references to land in the object file and later failed to resolve during linking.
Searching around all I found was "Parallel HDF5 IO requires MPI" but nothing related to CMake. https://www.hdfgroup.org/HDF5/release/cmakebuild.html is also pretty sparse about this and does not mention HDF5_IS_PARALLEL (they do mention that they don't provide that find module).
Given that, I ended up adding an interface target, set the includes and libraries from hdf5, check for HDF_IS_PARALLEL and add the mpi includes and libraries to that target.
Related
I ran the following command:
nvcc -arch=sm_70 foo.cu -o predatorPrey -I $BOOST_ROOT -L $BOOST_LIBRARY_PATH -lboost_timer
And got the following compilation error:
boost/include/boost/core/noncopyable.hpp(42): error: defaulted default constructor cannot be constexpr because the corresponding implicitly declared default constructor would not be constexpr
A google search led me here.
All hope seemed lost until this guy used a workaround.
Though, as a junior programmer, I don't understand what he means by
Built boost from source with g++11 open solved the problem
Does that mean rebuilding boost from scratch? How is it different from building boost by default?
So what are actually the workarounds to use both and CUDA in the same project?
For host code usage:
The only general workaround with a high probability of success when building a 3rd party library with the CUDA toolchain is to arrange your project in such a way that the 3rd party code is in a file that ends in .cpp and is processed by the host compiler (e.g. g++ on linux, cl.exe on windows).
Your CUDA code (e.g. kernels, etc.) will need to be in files with filenames ending in .cu (for default processing behavior).
If you need to use this 3rd party code/library functionality in your functions that are in the .cu file(s), you will need to build wrapper functions in your .cpp files to provide the necessary behavior as callable functions, then call these wrapper functions as needed from your .cu file(s).
Link all this together at the project level.
It may be that other approaches can be taken if the specific issue is analyzed. For example, sometimes updating to the latest version of the 3rd party library and/or CUDA version, may resolve the issue.
For usage in device code:
There is no general compatibility approach. If you expect some behavior to be usable in device code, and you run into a compile error like this, you will need to address the issue specifically.
General suggestions may still apply, such as update to the latest version of the 3rd party library you are using, and/or latest CUDA version.
I am building a C++ project on my Ubuntu 64bit system using a provided Makefile, and this project also provides an API library for developers.
The compilation was successful, no errors at all, but when I try to include in my files the API libraries provided in the "api" folder, then g++ complains about undefined references.
It is not a problem about dependencies (I already built the project succesfully), in fact the missing references are about classes and functions provided by the project, but they are in some specific (sub-)folders (I don't know which ones!), I guess in some .so files, and g++ is not finding them, probably because it does not know they are in those specific subfolders.
It is not the first time this happens when trying to use APIs from any project, then I think I am missing something or I am doing something wrong in general when trying to use the libraries provided in a project.
In particular, the main problem is that I don't know how to tell the compiler where some classes or data structures are declared, and moreover I don't know how to locate them in order to know where they are.
Usually, a workaround I use to avoid this problem is to run make install (as root or using sudo) so that libraries and APIs are installed in standard folders (like /usr/include or /usr/lib) and if I do this thend I can include the API libraries using #include <library>, but in this last case it didn't work either, because perhaps some required files containing the not found classes/structures are not properly installed in the right folders in the system.
Another workaround I used sometimes is to try to put all the project files in the same folder instead of using the project folder structure, but clearly this is not good! :-)
But I noticed that several other people managed to use the APIs, then apparently they know some way of finding the files containing the "undefined" references and including them in the compilation.
Then my general question is: given a "classic" C++ project based on "Makefile" files and with usual folder names like src, lib, build, bin, etc., if I want to write C++ files using the libraries provided by the project, but the compiler complains about undefined references, how can I find the files (.so or .o or .cpp) containing such references? Is there any tool to find them? And how can I tell the compiler where they are? Should I use some command-line option for g++ or should I use the #include macro in some smart way?
PS I also tried to use the pkc-config linux tool to get right options to use for compilation and they were available, but the compiler still complains about the undefined references.
Some more degails about the project i tried:
For interested people a link to the project is below:
https://github.com/dreal/dreal3
And instructions on how to build it:
http://dreal.github.io/download/
Look into the -rpath linker option - specifically with the "$ORIGIN" argument. That lets you find libraries relative to your executable location so you don't have to install them to the standard locations but just need to put them somewhere known, relative to the executable. That should help you with one piece of the puzzle.
Note: -Wl, can be used to pass arguments to the linker via g++.
As for pointing the compiler/linker at a library so it can resolve undefined references by using that library, use the -l (that's lowercase L) option to specify the library name and -L to specify directories to search for libraries.
As for looking into a library (.so) file to see what symbols are in there, you have a few tools at your disposal: objdump, nm, readelf and objcopy.
We have a (C++) program that's set up as a series of shared libraries with a nested hierarchy. That is, libB.so uses functions from and thus links against libA.so, libC.so uses functions from and links against both libB.so and libA.so, etc.
For our current CMake+Ninja build system I've noticed that parallel building does not seem to happen across libraries. That is, while Ninja will normally use 12 cores to build, if I've touched a single source file from libA but multiple in libC, ninja will use only a single processor to build just the libA source file until libA.so is linked, at which point it will use 12 processors to compile the libC source files. - And if there's an error in compilation in the libA source, it won't even try to compile the libC files into object files, even if I pass -k to ninja.
Certainly, the linking of libC.so needs to be delayed until libA.so is linked, but the compilation of the source files into object files for the libC sources shouldn't need to be delayed for the linking of libA.
Is there something I'm missing about the optimal way to set up our CMake files to express the dependencies between the libraries? Or is this an insurmountable limitation of how Ninja works?
This was asked recently on the CMake mailing list. The response from one of the devs confirms that the behaviour you report is intentional:
This is unfortunately necessary to get correct builds for CMake projects
in general because we support cases where add_custom_command is used
in library "foo" to generate a header file that is included during
compilation in library "bar" that links to "foo", but we have no good
way to express this dependency besides the ordering dependency of bar
on foo.
While it may be possible for CMake to be improved to relaxed that constraint in some circumstances, this doesn't appear to have been done yet (as of CMake 3.6). There's an open ticket for this in the Kitware issue tracker already.
Update: This appears to have been fixed in CMake 3.9.0, although that change led to a regression which was then subsequently fixed in 3.12.0, or possibly 3.11.2.
In C++, does one need to copy any needed header files into the directory of the main C++ file?
Ex. I have OpenCV installed globally, and the Python bindings are working well. However, if I write:
#include "opencv2/highgui/highgui.hpp"
I receive a "not found" error. Do I need to copy these from their global install location to the project dir? I'm certain there must be a well-established set of practices for this, so I don't want to poke around in the dark on my own.
It depends on which operating system and build tool chain you are using, but as an example using linux, gcc and cmake, this article shows how to build with opencv.
http://docs.opencv.org/trunk/doc/tutorials/introduction/linux_gcc_cmake/linux_gcc_cmake.html
As you can see with the find_package directive, cmake is searching for the opencv include files.
Obviously, you can specify include path directly with g++ -I, but having cmake find it for you has the advantage that it will have a better chance of being found if you compile on a different system. It will also give you an error if it can't find the files.
Lastly, you should ensure that you have the "dev" files, as opposed to just the library. The dev files will have headers to include. While, the library will only have shared objects (*.so) and archives (*.a) that can be used for static or runtime linking.
If it is installed globally, you need to inform the compiler to look globally, i.e.,
#include <opencv2/highgui/highgui.hpp>
I am trying to make my autotools project in C++ link against library, that originates as C library (libsomelib.so), but also has bindings to c++ (libsomelib++.so). I ma trying to use PKG_CHECK_MODULES to check if this package is installed, and use autotools to link against it. However both libs come in one package (c++ version requires configure flag), and have only one .pc file, in which independently of configuration settings there is only line
Libs: -L${libdir} -lsomelib
without any mentioning of ++ version. There is also no separate ++.pc file that i noticed at other programs. Therefore automatic linking against ++ version is impossible. I thought about manually adding -lsomelib++ to linking flags, but that's rather ugly (and it will not work if somebody compiled it without --with-cxx flag). I could also test for it's existence by AC_SEARCH_LIBS, but since it's C++ library it's not so straightforward.
Is missing ++.pc file mistake of package distributor or is it some deeper idea, and i don't know how to use it?
If somebody is really qurious i will say that package in question is ossp-uuid.
Yes, the missing ++.pc usually hints towards an omission on behalf of the packager.
BTW: If simple (DCE) UUIDs are sufficient, you could consider e2fsprogs/util-linux's libuuid (in case you run this OS).