Conan-Package: Getting compiled files from other Conan-Packages - c++

I'm using Conan as a dependency manager for C++ and I want to create a package which requires a compiled file from another, already created, Conan-package.
I'm currently trying to create a package for the OpenStreetMap-Library OSM-binary (https://github.com/scrosby/OSM-binary.git).
The Makefile for this project (which can be found at ./OSM-binary/src/Makefile) requires a file called protoc from the protobuf-project (https://github.com/google/protobuf). This protoc-file can be found after compiling the protobuf-project in ./protobuf/src.
Without this file compiling the OSM-sources will fail with an error: make: ../protoc: Command not found
The Problem
As conan's documentation suggests to copy my needed files to folders in my package, e.g header-files to ./include, libs to ./lib, etc.
According to this, after building the protobuf-project via make, I'm copying the mentioned file via
def package(self):
self.copy("*.so", dst="lib", keep_path=False)
self.copy("protoc", dst="scripts", src="./protobuf/src")
to an folder called "scripts".
But at this point the black magic starts.
My first question is, how can I access any of these packed files (e. g. the *.so files or any other files (here the protoc-file) which are present in an package) from another package?
For me, even after reading conan's documentation, it's not clear how conan stores it's packed files and how to access these or any other files packed in the previous step.
Now back at the OSM-Project my approach would be setting the correct path manually in the Makefile via the tools.replace command.
First I declared the protobuf-packaged as an requirement
requires = "protobuf/2.5.0#test/testing"
and replaced the corresponding lines (in version 1.3.3, line 7) in the osm-Makefile with the correct path to the protoc-file.
tools.replace_in_file("OSM-binary/src/Makefile",
"PROTOC ?= protoc",
"PROTOC ?= <path-to-file>/protoc")
Now this leads us to my actual question: How do I get the path to the protoc-file which can be found in the protobuf-package in a folder called scripts, or is there any other way to do it?
Thanks,
Chris

There are different ways to access files from your dependencies:
If you want to directly run some file from your dependencies, you could use the self.run(...., run_environment=True), that will automatically set the PATH, LD_LIBRARY_PATH, etc so the binaries are found in the place where the package is installed. Find more information here
You can directly import the files you want from your dependencies, doing a copy (which is done before the build() method) of such files into the build folder, so they can be directly used there. The path you can use in your script is the current one, or self.build_folder. The imported files will be automatically removed after build, so they are not accidentally repackaged. Check imports docs
You can obtain information from your dependencies from the self.deps_cpp_info attribute. Check the reference here. That means you can get the paths to your protobuf dependency with something like
def build(self):
# Get the directory where protobuf package is installed
protoc_root = self.deps_cpp_info["protobuf"].rootpath
# Note this is a list
protoc_bin_paths = self.deps_cpp_info["protobuf"].bin_paths

Related

Conan package manager - how to remove folders during conan install?

I have a local conanfile.py to consume a package, the package is already located on the local cache (~/.conan/).
In the conanfile.py there is the imports() function in which I copy some files from the package into my build folder.
I have two files with the same name in different directories and I copy them to the same directory and rename one of them.
After I do that, I am left with an empty directory I want to remove, but can't find a way to do so from conanfile.py, every attempt seems to remove the folder before the files gets run. My imports looks as follows:
class SomeConanPkg(ConanFile):
name = "SomeName"
description = "SomeDesc"
requires = (
"SomePkg/1.0.0.0#SomeRepo/stable")
def imports(self):
# copy of 1st file
self.copy("somefile.dll", src=os.path.join("src"), dst=os.path.join(build_dest))
# copy of 2nd file to nested directory
self.copy("somefile.dll", src=os.path.join("src", "folder"), dst=os.path.join(build_dst, "folder"))
# move and rename the file to parent directory
shutil.copy2(os.path.join(build_dst, "folder", "somefile.dll"), os.path.join(build_dst, "renamed_file.dll"))
# now build_dst/folder is an empty directory
I have tried to use conan tools.rmmdir() or just calling shutil.rmmtree() but all of them seems to run before the files gets copied.
I also tried to add a package() or deploy() member functions and execute the remove inside but these methods don't seem to run at all (verified with a debug print).
Any ideas?
I ended us solving it in the package creation side.
Renamed the files as I wanted and then just consumed them
Try conan remove <package name> . If you do not know the exact package name you can also use conan search to see the list of packages before you use conan remove.

"unsupported/Eigen/CXX11/Tensor: No such file or directory" while working with TensorFlow

I'm trying to use tensorflow as a external library in my C++ application (mainly following this tutorial). What I done so far:
I have cloned the tensorflow reporitory (let's say, that the repo root dir is $TENSORFLOW)
Run /.configure (which all settings default, so no CUDA, no OpenCL etc.).
Build shared library with bazel build -c /opt //tensorflow:libtensorflow_cc.so (build completed successfully)
Now I'm trying to #include "tensorflow/core/public/session.h". But after including it (and adding $TENSORFLOW and $TENSORFLOW/bazel-genfiles to include path), I'm receiving error:
$TENSORFLOW/tensorflow/third_party/eigen3/unsupported/Eigen/CXX11/Tensor:1:42:
fatal error: unsupported/Eigen/CXX11/Tensor: No such file or directory
There is a github issue created for similar problem, but it's marked as closed without any solution provided. Also I tried with master branch as well as v.1.4.0 release.
Do you happen to know, what could cause this kind of problem and how to deal with it?
I (and many others) agonized over the same problem. It probably can be solved using bazel but I don't know that tool well enough and now I solve this using make. The source of confusion is that a file named Tensor is included and it itself includes a file named Tensor, which has caused some people to wrongly conclude Tensor is including itself.
If you built and installed the python .whl file there will be a tensorflow directory in dist-packages and an include directory below that, e.g. on my system:
/usr/local/lib/python2.7/dist-packages/tensorflow/include
From the include directory
find . -type f -name 'Tensor' -print
./third_party/eigen3/unsupported/Eigen/CXX11/Tensor
./external/eigen_archive/unsupported/Eigen/CXX11/Tensor
The first one has
#include "unsupported/Eigen/CXX11/Tensor"
and the file that should satisfy this is the second one.
So to compile session.cc that includes session.h, the following will work
INC_TENS1=/usr/local/lib/python2.7/dist-packages/tensorflow/include/
INC_TENS2=${INC_TENS1}external/eigen_archive/
gcc -c -std=c++11 -I $INC_TENS1 -I $INC_TENS2 session.cc
I've seen claims that you must build apps from the tensorflow tree and you must use bazel. However, I believe all the header files you need are in dist-packages/tensorflow/include and at least for starters you can construct makefile or cmake projects.
Slightly off-topic, but I had the same error with a C++ project using opencv-4.5.5 and compiled with Visual Studio (no problem with opencv-4.3.0, and no problem with MinGW).
To make it work, I had to add to my root CMakeLists.txt:
add_definitions(-DOPENCV_DISABLE_EIGEN_TENSOR_SUPPORT)
If that can help someone...
the problem was actually in the relative path of the header file taken in the Tensor file.
installed path for Tensor is /usr/include/eigen3/unsupported/Eigen/CXX11/Tensor
but mentioned in the Tensor file is "unsupported/Eigen/CXX11/Tensor"
So there should be an entry upto /usr/include/eigen3/ in the project path to run this correctly so that it can be used.

Add source to an existing automake program

I would like to edit an existing software to add a new source file (Source.cpp).
But, I can't manage the compilation process (it seems to be automake and it looks very complicated).
The software (iperf 2: https://sourceforge.net/projects/iperf2/files/?source=navbar) is compiled using a classical ./configure make then make install.
If I just add the file to the corresponding source and include directory, I got this error message:
Settings.cpp:(.text+0x969) : undefined reference to ...
It looks like the makefile isn't able to produce the output file associated with my new source file (Source.cpp). So, I probably need to indicate it manually somewhere.
I searched a bit in the project files and it seemed that the file to edit was: "Makefile.am".
I added my source to the variable iperf_SOURCES in that file but it didn't workded.
Could you help me to find the file where I need to indicate my new source file (it seems a pretty standard compilation scheme but I never used automake softwares and this one seems very complicated).
Thank you in advance
This project is built with the autotools, as you already figured out.
The makefiles are built by automake. It takes its input in files that usually have a am file name extension.
The iperf program is built by the makefile generated from src/Makefile.am. This is indicated by:
bin_PROGRAMS = iperf
All (actually this is a simplification, but which holds in this case) source files of a to be built binary are in the corresponding name_SOURCES variable, thus in this case iperf_SOURCES. Just add your source file to the end of that list, like so (keeping their formatting):
iperf_SOURCES = \
Client.cpp \
# lines omitted
tcp_window_size.c \
my_new_file.c
Now, to reflect this change in any future generated src/Makefile you need to run automake. This will modify src/Makefile.in, which is a template that is used by config.sub at the end of configure to generate the actual makefile.
Running automake can happen in various ways:
If you already have makefiles that were generated after an configure these should take care of rebuilding themselves. This seems to fail sometimes though!
You could run automake (in the top level directory) by hand. I've never done this, as there is the better solution to...
Run autoreconf --install (possibly add --force to the arguments) in the top level directory. This will regenerate the entire build system, calling all needed programs such as autoheader, autoconf and of course automake. This is my favorite solution.
The later two options require calling configure again, IMO ideally doing an out of source built:
# in top level dir
mkdir build
cd build
../configure # arguments
make # should now also compile and link your new source file

Installing SML/NJ library

I need to install QCheck/SML unit test library for ML.
I could git clone the code, and create the .cm file, but I'm not sure how to copy the generated file into where. The document simply says (http://contrapunctus.net/league/haques/qcheck/qcheck_2.html):
2.1 SML/NJ
For Standard ML of New Jersey, the CM library specification ‘qcheck.cm’ should be all you need. The default target of make -f
Makefile.nj will ask CM to build and stabilize this library. This
creates a file ‘.cm/x86-unix/qcheck.cm’ (alter the arch/os tag as
needed) which may be copied into the standard CM library path and
added to the ‘pathconfig’.
I used brew install smlnj for the ML installation in Mac, so I have SMLNJ_HOME at /usr/local/Cellar/smlnj/100.78/SMLNJ_HOME.
What is the CM path library in this? In general, how to install a library into SML/NJ?
Edit
From Matt's answer, this is how I made it work.
Setup
Copy the whole qcheck directory into /usr/local/Cellar/smlnj/110.78/SMLNJ_HOME/lib.
Make ~/.smlnj-pathconfig file.
Add qcheck.cm /usr/local/Cellar/smlnj/110.78/SMLNJ_HOME/lib/qcheck in the file.
Usage (in REPL)
CM.make "$/qcheck.cm";
open QCheck;
Things to consider.
I couldn't use the stabilized libraries (qcheck/.cm/x86-unix/qcheck.cm). So, I had to copy the whole directory.
For user's library, I think the install location can be anywhere, as the ~/.smlnj-pathconfig can point to the directory.
For importing a structure in the same directory, use "FILENAME"; is needed instead of CM.make.
The CM library path is located in SMLNJ_HOME/lib. You can place the .cm file here. The instructions say to modify the pathconfig file, however, I would suggest creating a .smlnj-pathconfig file in your home directory instead. You are going to want to then paste the following line into that file:
qcheck.cm <path to directory containing qcheck.cm file>
You can then reference this in one of your .cm files using the anchor name: $/qcheck.cm. I've not used stabilized libraries before, and the generated .cm file is giving me a bunch of errors. If you instead use the qcheck.cm file from the root directory of the qcheck repo, it seems to work for me. Perhaps someone else can comment on why I am getting these errors.

Compiling C++Builder project on command line

Is there a way to compile a C++Builder project (a specific build configuration) from the command line?
Something like:
CommandToBuild ProjectNameToBuild BuildConfiguration ...
There are different ways for automating your builds in C++Builder (as of my experience, I'm speaking about old C++Builder versions like 5 and 6).
You can manually call compilers - bcc32.exe (also dcc32.exe, brcc32.exe and tasm32.exe if you have to compile Delphi units, resource files or assembly language lines of code in your sources) and linker - ilink32.exe.
In this case, you will need to manually provide the necessary input files, paths, and keys as arguments for each stage of compilation and linking.
All data necessary for compilation and linking is stored in project files and, hopefully there are special utilities, included in the C++Builder installation, which can automate this dirty work, provide necessary parameters to compilers and linker and run them. Their names are bpr2mak.exe and make.exe.
First you have to run bpr2mak.exe, passing your project *.bpr or *.bpk file as a parameter and then you will get a special *.mak file as output, which you can use to feed on make.exe, which finally will build your project.
Look at this simple cmd script:
#bpr2mak.exe YourProject.bpr
#ren YourProject.mak makefile
#make.exe
You can provide the real name of "YourProject.mak" as a parameter to make.exe, but the most straightforward way is to rename the *.mak file to "makefile", and then make.exe will find it.
To have different build options, you can do the following:
The first way: you can open your project in the IDE, edit options and save it with a different project name in the same folder (usually there are two project files for debug and release compile options). Then you can provide your building script with different *.bpr files. This way, it looks simple, because it doesn't involves scripting, but the user will have to manually maintain coherency of all project files if something changes (forms or units added and so on).
The second way is to make a script which edits the project file or make file. You will have to parse files, find compiler and linker related lines and put in the necessary keys. You can do it even in a cmd script, but surely a specialised scripting language like Python is preferable.
Use:
msbuild project.cbproj /p:config=[build configuration]
More specifics can be found in Building a Project Using an MSBuild Command.
A little detail not mentioned.
Suppose you have external dependencies and that the .dll file does not initially exist in your folder
You will need to include the external dependencies in the ILINK32.CFG file.
This file is usually in the folder
C:\Program Files (x86)\Borland\CBuilder6\Bin\ilink32.cfg
(consider your installation location)
In this file, place the note for your dependencies.
Example: A dependency for TeeChart, would look like this (consider the last parameter):
-L"C:\Program Files (x86)\Borland\CBuilder6\lib";"C:\Program Files (x86)\Borland\CBuilder6\lib\obj";"C:\Program Files (x86)\Borland\CBuilder6\lib\release";"C:\Program Files (x86)\Steema Software\TeeChart 805 for Builder 6\Builder6\Include\";"C:\Program Files (x86)\Steema Software\TeeChart 805 for Builder 6\Builder6\Lib\"
You will also need to include the -f command to compile.
In cmd, do:
//first generate the file.mak
1 - bpr2mak.exe MyProject.bpr
//then compile the .mak
2 - make.exe -f MyProject.mak
You can also generate a temporary mak file with another name, as the answer above says, directly with bpr2mak
bpr2mak.exe MyProject.bpr -oMyTempMak.mak