CMake check hash (MD5 / SHA256) for downloaded file - c++

I am using CMake add_custom_command In a Util.cmake script to download a couple of files that will later be used in the build process. These files however may change and I would like to add a way to check the hash value of the local file against a provided value (within CMake) to decide if the file needs to be re-downloaded.
Currently, once the file has been downloaded, CMake will not consider re-downloading it, because the file already exists locally.
In the future, I want to provide a MD5 / SHA256 checksum of that file and make sure the local file is the corect one.
Here is what I am trying to do (this is just an concept example):
add_custom_command( OUTPUT ./file.dat
COMMAND wget ${FILE_PATH}
)
if (opt_HASH)
add_custom_command(OUTPUT ${local_HASH}
COMMAND local_HASH=$(sha256sum ./file.dat)
DEPENDS ./file.dat
)
if (NOT ${opt_HASH} STREQUAL ${local_HASH})
# throw ERROR
endif()
endif()
As you can see I only want to detect a mismatch right now and don't even want to auto-download the changed file. The opt_HASH is obviously provided through CMake, but what is important is that this call needs to depend on the file already being downloaded and I seem to be able to do that with a more simpler call to FILE().
PS: If it's somehow easier, I could also use MD5 over SHA256.

cmakes FILE command supports hashing:
https://cmake.org/cmake/help/v3.8/command/file.html
file(SHA256 ./file.dat CHECKSUM_VARIABLE)
should put the hash into the CHECKSUM_VARIABLE

The usage of add_custom_target could be one solution. By default it will be executed always. The following should work on linux:
add_custom_target(UpdateExternalFiles
COMMAND "sha256sum -c file.dat.checksum ./file.dat || wget ${FILE_PATH}"
COMMAND "sha256sum ./file.dat >> file.dat.checksum"
)
First line verifies the checksum and loads the file on differences. The second line updates the checksum.
Note: This snipped assume that file.dat.checksum will be created with the second command.

Related

Conan-Package: Getting compiled files from other Conan-Packages

I'm using Conan as a dependency manager for C++ and I want to create a package which requires a compiled file from another, already created, Conan-package.
I'm currently trying to create a package for the OpenStreetMap-Library OSM-binary (https://github.com/scrosby/OSM-binary.git).
The Makefile for this project (which can be found at ./OSM-binary/src/Makefile) requires a file called protoc from the protobuf-project (https://github.com/google/protobuf). This protoc-file can be found after compiling the protobuf-project in ./protobuf/src.
Without this file compiling the OSM-sources will fail with an error: make: ../protoc: Command not found
The Problem
As conan's documentation suggests to copy my needed files to folders in my package, e.g header-files to ./include, libs to ./lib, etc.
According to this, after building the protobuf-project via make, I'm copying the mentioned file via
def package(self):
self.copy("*.so", dst="lib", keep_path=False)
self.copy("protoc", dst="scripts", src="./protobuf/src")
to an folder called "scripts".
But at this point the black magic starts.
My first question is, how can I access any of these packed files (e. g. the *.so files or any other files (here the protoc-file) which are present in an package) from another package?
For me, even after reading conan's documentation, it's not clear how conan stores it's packed files and how to access these or any other files packed in the previous step.
Now back at the OSM-Project my approach would be setting the correct path manually in the Makefile via the tools.replace command.
First I declared the protobuf-packaged as an requirement
requires = "protobuf/2.5.0#test/testing"
and replaced the corresponding lines (in version 1.3.3, line 7) in the osm-Makefile with the correct path to the protoc-file.
tools.replace_in_file("OSM-binary/src/Makefile",
"PROTOC ?= protoc",
"PROTOC ?= <path-to-file>/protoc")
Now this leads us to my actual question: How do I get the path to the protoc-file which can be found in the protobuf-package in a folder called scripts, or is there any other way to do it?
Thanks,
Chris
There are different ways to access files from your dependencies:
If you want to directly run some file from your dependencies, you could use the self.run(...., run_environment=True), that will automatically set the PATH, LD_LIBRARY_PATH, etc so the binaries are found in the place where the package is installed. Find more information here
You can directly import the files you want from your dependencies, doing a copy (which is done before the build() method) of such files into the build folder, so they can be directly used there. The path you can use in your script is the current one, or self.build_folder. The imported files will be automatically removed after build, so they are not accidentally repackaged. Check imports docs
You can obtain information from your dependencies from the self.deps_cpp_info attribute. Check the reference here. That means you can get the paths to your protobuf dependency with something like
def build(self):
# Get the directory where protobuf package is installed
protoc_root = self.deps_cpp_info["protobuf"].rootpath
# Note this is a list
protoc_bin_paths = self.deps_cpp_info["protobuf"].bin_paths

How can I replace a given -Ldir1 by -Ldir2, in a linking command line generated by cmake

I have a CMakeLists.txt.
I use it for generating a makefile with ccmake.
Then upon make, my sources are compiled ok.
At link time, the command line produced is essentially
/opt/rh/devtoolset-6/root/usr/bin/c++ myprog.cc.o -o myprog -Ldir3 -L/opt/rh/devtoolset-3/root/usr/lib/gcc/x86_64-redhat-linux/4.9.2 ... -Wl,-rpath,dir4:dir5:/opt/rh/devtoolset-3/root/usr/lib/gcc/x86_64-redhat-linux/4.9.2 ...
The two spots specifying the search path
/opt/rh/devtoolset-3/root/usr/lib/gcc/x86_64-redhat-linux/4.9.2
should actually point to
/opt/rh/devtoolset-6/root/usr/lib/gcc/x86_64-redhat-linux/6.2.1
How can I fix this?
I have devtoolset-3 in my system, but I do not know where this search path is set, or how to change it.
I actually expected that to take place automatically after executing
scl-devtoolset-6
(in my .bashrc), the same way the correct version /opt/rh/devtoolset-6/root/usr/bin/c++ is detected without me doing anything else.
Of course, I get many linking time errors due to version mismatches.
The only place where I see the search path set is in line
link_directories(${LIBDIR_MY})
in CMakeLists.txt, and LIBDIR_MY points to dir3, which is correctly added in the linking command line.
But I found no place where .../devtoolset-3/... is added.
Possible origins of -L:
link_directories in CMakeLists.txt: checked.
target_link_libraries: Where? What is the expected file name pattern to look for?
link_libraries: Where? What is the expected file name pattern to look for?
CMAKE_LIBRARY_PATH: Checked. It is not set.
A find_package command: See below
Somewhere else?
This How do I add a library path in cmake? does not add to my question.
Update 1:
There was in fact a find_package(mylib) (actually, with a different name) in CMakeLists.txt.
Going to the dir of mylib and issuing find . -name “*” -type f -exec grep /opt/rh/devtoolset-3 {} \; there were two files that matched:
build/CMakeCache.txt:
two occurrences of devtoolset-3
PETSC_LIBRARIES:STRING=...devtoolset-3...
FIND_PACKAGE_MESSAGE_DETAILS_PETSc:INTERNAL=[...devtoolset-3...][YES][v()]
It appears to me that this stems from file CMake/cmake-modules/FindPETSc.cmake (possibly invoked by line find_package (PETSc REQUIRED) in CMakeLists.txt), which has a line
set (PETSC_LIBRARIES ${PETSC_LIBRARIES_ALL} CACHE STRING "PETSc libraries" FORCE)
and many prior lines
set (PETSC_LIBRARIES_<various terms> ...)
Notes:
I do not know where in that file devtoolset-3 is first detected and set.
build/include/summit/mylibConfig.cmake.
I still could not track down what made devtoolset-3 appear here.
I found the culprit.
As hinted at in Update 1 of the OP, the sequence is the following:
Line
find_package (PETSc REQUIRED) in file (1)
CMakeLists.txt forced processing file (2)
CMake/cmake-modules/FindPETSc.cmake.
Line*1
petsc_get_variable (PETSC_EXTERNAL_LIB_BASIC petsc_libs_external) in file (2)
CMake/cmake-modules/FindPETSc.cmake sets a local cmake variable
petsc_libs_external from the value of the variable
PETSC_EXTERNAL_LIB_BASIC read from file (3)
~/petsc-3.8.2/lib/petsc/conf/petscvariables.
PETSC_EXTERNAL_LIB_BASIC is not set explicitly in file (3)
~/petsc-3.8.2/lib/petsc/conf/petscvariables. Line
include ~/petsc-3.8.2/arch-linux2-c-debug/lib/petsc/conf/petscvariables forces reading file (4)
~/petsc-3.8.2/arch-linux2-c-debug/lib/petsc/conf/petscvariables.
Line
PETSC_EXTERNAL_LIB_BASIC = ... -Wl,-rpath,/opt/rh/devtoolset-3/root/usr/lib/gcc/x86_64-redhat-linux/4.9.2 -L/opt/rh/devtoolset-3/root/usr/lib/gcc/x86_64-redhat-linux/4.9.2 -Wl,-rpath,/opt/rh/devtoolset-3/root/usr/lib64 -L/opt/rh/devtoolset-3/root/usr/lib64 -Wl,-rpath,/opt/rh/devtoolset-3/root/usr/lib -L/opt/rh/devtoolset-3/root/usr/lib ... in file (4)
/home/sserebrinsky/petsc-3.8.2/arch-linux2-c-debug/lib/petsc/conf/petscvariables brings the "undesired" directories into the executed command line.
*1 (petsc_get_variable is a macro defined in FindPETSc.cmake)

Add source to an existing automake program

I would like to edit an existing software to add a new source file (Source.cpp).
But, I can't manage the compilation process (it seems to be automake and it looks very complicated).
The software (iperf 2: https://sourceforge.net/projects/iperf2/files/?source=navbar) is compiled using a classical ./configure make then make install.
If I just add the file to the corresponding source and include directory, I got this error message:
Settings.cpp:(.text+0x969) : undefined reference to ...
It looks like the makefile isn't able to produce the output file associated with my new source file (Source.cpp). So, I probably need to indicate it manually somewhere.
I searched a bit in the project files and it seemed that the file to edit was: "Makefile.am".
I added my source to the variable iperf_SOURCES in that file but it didn't workded.
Could you help me to find the file where I need to indicate my new source file (it seems a pretty standard compilation scheme but I never used automake softwares and this one seems very complicated).
Thank you in advance
This project is built with the autotools, as you already figured out.
The makefiles are built by automake. It takes its input in files that usually have a am file name extension.
The iperf program is built by the makefile generated from src/Makefile.am. This is indicated by:
bin_PROGRAMS = iperf
All (actually this is a simplification, but which holds in this case) source files of a to be built binary are in the corresponding name_SOURCES variable, thus in this case iperf_SOURCES. Just add your source file to the end of that list, like so (keeping their formatting):
iperf_SOURCES = \
Client.cpp \
# lines omitted
tcp_window_size.c \
my_new_file.c
Now, to reflect this change in any future generated src/Makefile you need to run automake. This will modify src/Makefile.in, which is a template that is used by config.sub at the end of configure to generate the actual makefile.
Running automake can happen in various ways:
If you already have makefiles that were generated after an configure these should take care of rebuilding themselves. This seems to fail sometimes though!
You could run automake (in the top level directory) by hand. I've never done this, as there is the better solution to...
Run autoreconf --install (possibly add --force to the arguments) in the top level directory. This will regenerate the entire build system, calling all needed programs such as autoheader, autoconf and of course automake. This is my favorite solution.
The later two options require calling configure again, IMO ideally doing an out of source built:
# in top level dir
mkdir build
cd build
../configure # arguments
make # should now also compile and link your new source file

Difference between cmake variables and cmake arguments

I have been given some source code, along with a CMakeLists.txt file, and told to run cmake by:
cmake ../src -DOPEN_NI_ROOT=/home/karnivaurus/OpenNI
I have also noticed that there is a file called FindOpenNI.cmake, which I believe is used when find_package(OpenNI) is called by cmake.
Therefore, I am guessing that OPEN_NI_ROOT is some kind of variable that is used by cmake for the remainder of setup.
However, I have tried inserting the line set(OPEN_NI_ROOT "/home/karnivaurus/OpenNI") into my CMakeLists.txt file, in an attempt to avoid the need to add it as an argument at the command line. But this does not seem to do the same thing.
Can somebody please explain how these two variable types are different?
The file FindOpenNI.cmake is open source and can be found at:
https://github.com/victorprad/InfiniTAM/blob/master/InfiniTAM/cmake/FindOpenNI.cmake
The issue is this line in FindOpenNI.cmake (link):
set(OPEN_NI_ROOT "/usr/local" CACHE FILEPATH "Root directory of OpenNI2")
This will set OPEN_NI_ROOT unless it's already in the cache. A simple call to:
set(OPEN_NI_ROOT "/home/karnivaurus/OpenNI")
does not set the variable in the cache, so it will be overridden when the line in FindOpenNI.cmake is hit. Using the command line form of setting the variable will set it in the cache, which is why it works just fine.
The easiest way to avoid having to set the command line option is to set the cache explicitly in your own CMakeLists.txt:
set(OPEN_NI_ROOT "/home/karnivaurus/OpenNI" CACHE FILEPATH "Root directory of OpenNI2")
If you're working from a dirty build directory, it's likely this cache variable already exists, so this line would have no effect. In that case, either work from a clean build directory, or set the FORCE option:
set(OPEN_NI_ROOT "/home/karnivaurus/OpenNI" CACHE FILEPATH "Root directory of OpenNI2" FORCE)
This will write over the cached value. Note that this would prevent you from setting the option in the command line in the future, which is why this method isn't preferred. You can find some more information about the mechanics of this here.

Shortcut to intel compiler directory to use in makefile

I compiled my program with intel C++ compiler for windows (from Intel Composer 2011), and got an error message that libmmdd.lib cannot be found. I googled this problem, and some people said that I have to reinstall my compiler, and I did; however, that didn't resolve the problem, so I started looking in the intel compiler directory, and found that this file (and other required libraries as well) are located at
%CompilerDirectory%\compiler\lib\ia32
It doesn't make sense to write in the make file the whole absolute path of the libraries, so I started searching, and I could only find that %mklroot% points to the math kernel directory. And even with a -L%mklroot%/../compiler/lib/ia32 approach for linking I couldn't link to the libraries correctly, so eventually I did a lame move to solve the problem, which is, I copied every file the linker asks for to the source directory, and so was the problem temporarily solved.
Since this way of solving the problem isn't the best one, I wonder if there's a way to link to those libraries without having to copy the files. It's strange because the compiler should find its own libraries alone, but... I don't know...!
Any ideas? is there something like, %compilerroot%, that points to the compiler directory and that I could put in my makefile (or actually my qmake, since I'm using Qt).
Thanks for any efforts :-)
Instead of using %mklroot% try $$(mklroot) or $(mklroot).
You can find the explanation here:
Variables can be used to store the contents of environment variables.
These can be evaluated at the time that qmake is run, or included in
the generated Makefile for evaluation when the project is built.
To obtain the contents of an environment value when qmakeis run, use
the $$(...) operator:
DESTDIR = $$(PWD)
message(The project will be installed in $$DESTDIR)
In the above assignment, the value of the PWD environment variable is
read when the project file is processed.
To obtain the contents of an environment value at the time when the
generated Makefile is processed, use the $(...) operator:
DESTDIR = $$(PWD)
message(The project will be installed in $$DESTDIR)
DESTDIR = $(PWD)
message(The project will be installed in the value of PWD)
message(when the Makefile is processed.)
In the above assignment, the value of PWD is read immediately when the
project file is processed, but $(PWD) is assigned to DESTDIR in the
generated Makefile. This makes the build process more flexible as long
as the environment variable is set correctly when the Makefile is
processed.
EDIT:
It is strange that neither $$(mklroot) nor $(mklroot) gave you the result you would expect. I did a simple test to verify what I wrote above:
Opened a Command Prompt
Created a new environment variable 'mklroot' with a test value: set mklroot=C:\intel_libs
Verified the result of the previos step: echo %mklroot%. I got C:\intel_libs
Placed your 3 qmake functions at the end of my .pro file:
warning($(%MKLROOT%))
warning($(MKLROOT))
warning($$(MKLROOT))
Ran qmake: qmake. The result:
Project WARNING:
Project WARNING: c:\intel_libs
Project WARNING: c:\intel_libs
As you can see the 2nd and the 3rd warning() displayed the string I set to the environment variable.