I am a beginner in the ways of the "bitbake" and i wonder what happens in the following situations, when building a project with several thousand packages:
You bitbake the full image (all packages) and it finishes successfully.
You make a change to a package - some source code (let's call it package "X")
You bitbake the full image again.
In step 3 is "X" rebuilt? Is it necessary to increment the PV and PR for "X" to be rebuilt? What happens with a "Y" package that depends on "X"? If X is rebuilt is also "Y" rebuilt?
I know that if you modifiy a .bb file the depending packages are not built because a timestamp is checked. Is it the same mechanism with source code changes? (It's a QT project btw, so in the end bitbake runs qmake->make to make the compilation)
I am using bitbake version 1.13.2.
Thanks
Let me try to answer this. for example you have package X, Y and Z. lets say X depends on Y and Y depends on package Z.
If you are doing "bitbake default-image-name" and you are building as scratch (which means none of the package required in the default-image-name is built before). So bitbake now make a dependency tree (you can see dependency tree by "bitbake -g PACKAGE_NAME".). The first package Z will built and then Y and then finally package X.
Now lets say you have made some changes in the X source code and you do "bitbake X" without increasing the PR number in the X recipe file (x.bb), the bitbake don't compile the changes again. I mean to say bitbake just checks the Package Version and Revision (PV and PR). Here we have same Package version (PV) and same Package Revision (PR) so bitbake don't compile the X package again.
To compile the Package X after some modification, You need to apply the changes as a Patch. For that make a patch of changes (eg change.diff or change.patch), add the entry in the X.bb file (see other recipe file for example). after that increase the PR number in the X.bb.
Now "bitbake X" will built the X package again.
Here when we have Increased the PR of X, only the X package get built. Here bitbake will check the dependent package Y and Z which are already built and having the same PR number, it will just use the already built Y and Z package.
If we have rebuilt the Y package (you can clean the package bitbake -c clean package_name if don't want to increase the PR number), the X package will not built again by default even it depends on the Y.
Hope this help.
Bitbake depends of another systems to compile, for example make. make has the capability to detect if a package needs to be recompile. That is the way bitbake deals with recompiling a package.
Related
I am trying to "duplicate" a target in my CMake file without maintaining 2 targets and all it's dependencies.
For example I have a main target MyBigLibrary
add_library(MyBigLibrary STATIC "")
target_compile_definitions(MyBigLibrary PRIVATE definitions..)
target_include_directories(MyBigLibrary PUBLIC public_directories..)
target_include_directories(MyBigLibrary PRIVATE private_directories..)
target_link_libraries(MyBigLibrary INTERFACE libraries..)
...
...
target_sources(MyBigLibrary ..source files..)
What I am trying to achieve is to have a identical copy of MyBigLibrary target (e.g. MyBigLibraryModified) which then I can feed to external script via add_custom_command.
I know there is way to have 2 targets, but then you have maintain 2 targets and all of it's dependencies.
Is there a way to have a phony target, e.g MyBigLibraryModified which is built only MyBigLibrary, and inherits INTERFACE flags as a dependencies?
When I wanted to create two different libraries containing the same compiled objects but with different features, I used the OBJECT library type containing all the source files, then I created two different libraries that listed the OBJECT target in their target_link_libraries.
The OBJECT library in CMake is still not fully correct, even after years of waiting for it to be enhanced, but it works mostly OK in most situations these days.
See the documentation for more info.
Comments under question:
... Are you trying strip symbols from release version of your target? –
Marek R
#MarekR that is correct. – pureofpure
So you are doing that wrong. You do not need (and you should not have) separate target for stripped version of your library/application.
For stripped version build proves is just a bit different.
One way to do it is just do:
$ cmake --install . --prefix PathWithResultsWithSybols .....
$ cmake --install . --prefix PathWithStripedResults --strip ....
See cmake doc
[Ubuntu]
I have compiled CGAL locally:
/path/to/cgal/
/lib/
libCGAL_Core.so libCGAL_Core.so.13.0.2
libCGAL_ImageIO.so.13 libCGAL.so libCGAL.so.13.0.2
libCGAL_Core.so.13 libCGAL_ImageIO.so
libCGAL_ImageIO.so.13.0.2 libCGAL.so.13
/include/
/CGAL/
version.h compiler_config.h
And I have managed to satisfy all of the graph-tool requirements except cgal (at least all of the requirements checked up to cgal):
./configure --with-boost=/path/to/boost --with-cgal=/path/to/cgal
And I get all successes up and until I get the following error message:
checking for __gmpz_init in -lgmp... yes
checking for __gmpz_init in -lgmp... (cached) yes
checking whether CGAL is available in /path/to/cgal... no
configure: error: CGAL library not found.
// the harshest part is that it seems to be searching in the correct
// directory.
I have tried specifying different points in the cgal build directory. The cgal compilation command I used was (from build directory):
cmake path/to/cgal_src_dir -DCMAKE_BUILD_TYPE=Release;
Next, I tried adding includes:
./configure --with-boost=$boost --with-cgal=path/to/cgal CPPFLAGS="-I path/to/cgal/include -I $HOME/.local/include" LDFLAGS="-L path/to/cgal/lib -L $HOME/.local/lib -Wl,-rpath=$HOME/.local/lib"
I will admit that I don't understand the -Wl,-rpath= part, I copied that from the graph-tool installation guide. The .local/lib folder contains the files for the other components, such as gmp, expat, sparsehash, etc.
This is not exact answer but as asked by OP will help in finishing installation, so please don't vote blindly.
To create debian package of libcgal open your CMakeList.txt and at the end of file add:
#--------------------------------------------------------------------
# Create debian files
#--------------------------------------------------------------------
if (UNIX AND NOT APPLE)
SET(CPACK_GENERATOR "DEB")
SET(CPACK_PACKAGE_NAME "libcgal-all")
SET(CPACK_PACKAGE_VERSION "${CGAL_VERSION}")
SET(CPACK_PACKAGE_DESCRIPTION_SUMMARY "C++ library for computational geometry (development files)\n CGAL (Computational Geometry Algorithms Library) makes the most important of the solutions and methods developed in computational geometry available to users in industry and academia in a C++ library. The goal is to provide easy access to useful, reliable geometric algorithms.\n .\n This package contains the header files and static libraries for libCGAL.so, libCGAL_Core.so, and libCGAL_ImageIO.so. The header files and static libraries for libCGAL_Qt4.so can be found in the package libcgal-qt4-dev.")
SET(CPACK_PACKAGE_CONTACT "bordeo")
SET(CPACK_DEBIAN_PACKAGE_DEPENDS "libboost-dev, libboost-thread-dev, libboost-system-dev, libboost-program-options-dev, libgmp10-dev, libmpfr-dev, zlib1g-dev")
SET(CPACK_DEBIAN_PACKAGE_REPLACES "libcgal10, libcgal-dev")
INCLUDE(CPack)
endif()
In case you don't have any dependency remove whole line of SET(CPACK_DEBIAN_PACKAGE_DEPENDS "libcln6, libcln-dev, libreadline6, libreadline6-dev, flex, bison"), and change others as it seems fit.
Now go to the terminal and issue following commands in cgal directory
mkdir build
cd build
cmake-gui ..
# set CMAKE_INSTALL_PREFIX to `~/.local
cmake ..
make -j4
cpack ..
you will find your debian built. Extract or install the debian to ~/.local.
Once this is done go to graph tool directory and start the build like
./configure --prefix="/wherever" --with-boost=/path/to/boost --with-cgal=~/.local
make -j4
make install
Hope this will solve your problem.
I wrote a C++ package "P". It has an R interface package "RP", built using Rcpp. P used Make directly for compilation, but was switched to CMake for portability. CMake is used to find headers (let's call their collection HF) and static libraries (SL in the rest of the present post) for system wide libraries.
I want to update RP to be able to depend on the CMake evolution of P. In the C++ sources RP/src/*.cpp, HF elements are included, and of course SL are statically linked.
What is the best way to call CMake in RP/src/Makevars to retrieve the locations of HF and SL ? The point here is not to replace the build system of Rcpp, but to leverage the search capabilities of CMake.
At the moment, P (CMake version) and RP build on my machine, using absolute path references in RP/src/Makevars such as:
INC_NLOPT = /usr/local/Cellar/nlopt/2.4.2_2/include
LIB_NLOPT = /usr/local/Cellar/nlopt/2.4.2_2/lib/libnlopt.a
Since we use RP internaly at the moment, we can expect CMake, HF and SL to be installed on every machine we will deploy to.
The solution was to create a CMakeFileLists.txt file in RP/src. In this file, a Libvar file containing the required library paths is written using CMake file command. Libvar is then included in Makevars using include.
A configure file at the root of the package is executed to ensure that Libvar is generated before every call to make by R.
I am trying to compile a custom Op that includes tensorflow/core/kernels/reduction_gpu_kernels.cu.h. So the top of my GPU kernel file looks as follows:
...
#include "tensorflow/core/kernels/reduction_gpu_kernels.cu.h"
...
I want to compile it in my own repository using g++ and nvcc, so adding the Op to my TensorFlow source installation and compiling it with Bazel is not an option. Trying to compile using the instructions given here does not work, as the file is not included by the pip installation package.
I have considered the following solutions:
'Hard-coding' the include to a clone of the TensorFlow repo that contains the original source,
Copying the files from a TensorFlow repo to the TensorFlow installation in my PYTHON_PATH,
Building a pip package that would include the required files in the pip installation,
but these aren't very appealing workarounds. Is there any other way of doing this elegantly?
In the most recent version, there are all files available. In the mean-time I solved it using a CMake + Python
You are basically, writing for all necessary values to configure cmake, like
f.write("set(TensorFlow_ABI %i)\n" % tf.__cxx11_abi_flag__)
f.write("set(TensorFlow_INCLUDE_DIRS \"%s\")\n" % tf.sysconfig.get_include())
f.write("set(TensorFlow_LIBRARIES \"%s/libtensorflow_framework.so\")\n" % tf.sysconfig.get_lib())
All what is left is just setting the include_directories corre
https://github.com/cgtuebingen/tf_custom_op
When using other kernels, I would just copy the methods into a my operation (never had this case).
I have a c++ cmake project. In this project I build (among other) one example, where I need to use another project, call it Foo. This Foo project does not offer a cmake build system. Instead, it has a pre-made Makefile.custom.in. In order to build an executable that uses Foo's features, one needs to copy this makefile in his project, and modify it (typically setting the SOURCES variable and a few compiler flags). Basically, this Makefile ends up having the sources for your executable and also all the source files for the Foo project. You will not end up using Foo as a library.
Now, this is a design I don't like, but for the sake of the question, let's say we stick with it.
To create my example inside my cmake build I added a custom target:
CONFIGURE_FILE( ${CMAKE_CURRENT_SOURCE_DIR}/Makefile.custom.in Makefile.custom)
ADD_CUSTOM_TARGET(my_target COMMAND $(MAKE) -f Makefile.custom
WORKING_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR})
This works. I can specify some variables to cmake, which get resolved in the call to CONFIGURE_FILE, and I end up with a working Makefile.custom. Then, invoking make my_target from the build directory, I can build the executable. I can even add it to the all target (to save me the effort of typing make my_target) with
SET_TARGET_PROPERTIES(my_target PROPERTIES EXCLUDE_FROM_ALL FALSE)
Sweet. However, cmake appears to assign a single job to the custom target, slowing down my compilation time (the Foo source folder contains a couple dozens cpp files). On top of that, the make clean target does not forward to the custom makefile. I end up having to add another target:
ADD_CUSTOM_TARGET(really-clean COMMAND "$(MAKE)" clean
COMMAND "$(MAKE)" -f Makefile.custom clean
WORKING_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR})
which, unlike my_target with all, I can't include in the clean target (can I?).
Now, I know that a cleaner solution would be to have the Foo project be built as an external project, and then link to it. However, I've been 'recommended' to use their Makefile.custom.in makefile, modifying the few lines I need (adding my sources, specifying compiler flags, and few other minor modifications). So, regardless of how neat and clean this design pattern is, my questions are:
is there a way to tell cmake that make should use more than 1 job when making the target my_target?
is there a cleaner way to include a pre-existing makefile in a cmake project? Note that I don't want (can't?) use Foo as a library (and link against it). I want (need?) to compile it together with my executable using a makefile not generated by cmake (well, cmake can help a bit, through CONFIGURE_FILE, by resolving some variables, but that's it).
Note: I am aware of ExternalProject (as suggested also in this answer), but I think it's not exactly what I need here (since it would build Foo and then use it as a library). Also, both my project and Foo are written exclusively in C++ (not sure this matter at all).
I hope the question makes sense (regardless of how ugly/annoying/unsatisfactory the resulting design would be).
Edit: I am using cmake version 3.5.2
First, since you define your own target, you can assign more cores to the build process for the target my_target, directly inside your CMakeLists.txt.
You can include the Cmake module ProcessCount to determine the number of cores in your machine and then use this for a parallel build.
include(ProcessorCount)
ProcessorCount(N)
if(NOT N EQUAL 0)
# given that cores != 0 you could modify
# math(EXPR N "${N}+1") # modify (increment/decrement) N at your will, in this case, just incrementing N by one
set(JOBS_IN_PARALLEL -j${N})
endif(NOT N EQUAL 0)
and when you define your custom target have something like the following:
ADD_CUSTOM_TARGET(my_target
COMMAND ${CMAKE_MAKE_PROGRAM} ${JOBS_IN_PARALLEL} -f Makefile.custom
WORKING_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR})
by the way, I don't think there's the need for you to include also CMAKE_BUILD_TOOL among the COMMANDs in your target.
I believe that instead of modifying the lines as above, you could call
make -j8 my_target
and it might start 8 jobs (just an example) without modifying the CMakeLists.txt, but I cannot guarantee this works having defined the COMMAND the way you have, just try if that's enough.
For the second point, I cannot think right now of a "cleaner" way.