Simple and efficient distribution of C++/Boost source code (amalgamation) - c++

My job mostly consists of engineering analysis, but I find myself distributing code more and more frequently among my colleagues. A big pain is that not every user is proficient in the intricacies of compiling source code, and I cannot distribute executables.
I've been working with C++ using Boost, and the problem is that I cannot request every sysadmin of every network to install the libraries. Instead, I want to distribute a single source file (or as few as possible) so that the user can g++ source.c -o program.
So, the question is: can you pack the Boost libraries with your code, and end up with a single file? I am talking about the Boost libraries which are "headers only" or "templates only".
As an inspiration, please look at the distribution of SQlite or the Lemon Parser Generator; the author amalgamates the stuff into a single source file which is trivial to compile.
Thank you.
Edit:
A related question in SO is for Windows environment. I work in Linux.

There is a utility that comes with boost called bcp, that can scan your source and extract any boost header files that are used from the boost source. I've setup a script that does this extraction into our source tree, so that we can package the source that we need along with our code. It will also copy the boost source files for a couple of boost libraries that we use that are no header only, which are then compiled directly into our applications.
This is done once, and then anybody who uses the code doesn't even need to know that it depends on boost. Here is what we use. It will also build bjam and bcp, if they haven't been build already.
#!/bin/sh
BOOST_SRC=.../boost_1_43_0
DEST_DIR=../src/boost
TOOLSET=
if ( test `uname` = "Darwin") then
TOOLSET="--toolset=darwin"
fi
# make bcp if necessary
if ( ! test -x $BOOST_SRC/dist/bin/bcp ) then
if ( test -x $BOOST_SRC/tools/jam/*/bin.*/bjam ) then
BJAM=$BOOST_SRC/tools/jam/*/bin.*/bjam
else
echo "### Building bjam"
pushd $BOOST_SRC/tools/jam
./build_dist.sh
popd
if ( test -x $BOOST_SRC/tools/jam/*/bin.*/bjam ) then
BJAM=$BOOST_SRC/tools/jam/*/bin.*/bjam
fi
fi
echo "BJAM: $BJAM"
pushd $BOOST_SRC/tools/bcp
echo "### Building bcp"
echo "$BJAM $TOOLSET"
$BJAM $TOOLSET
if [ $? == "0" ]; then
exit 1;
fi
popd
fi
if ( ! test -x $BOOST_SRC/dist/bin/bcp) then
echo "### Couldn't find bpc"
exit 1;
fi
mkdir -p $DEST_DIR
echo "### Copying boost source"
MAKEFILEAM=$DEST_DIR/libs/Makefile.am
rm $MAKEFILEAM
# Signals
# copy source libraries
mkdir -p $DEST_DIR/libs/signals/src
cp $BOOST_SRC/libs/signals/src/* $DEST_DIR/libs/signals/src/.
echo -n "boost_sources += " >> $MAKEFILEAM
for f in `ls $DEST_DIR/libs/signals/src | fgrep .cpp`; do
echo -n "boost/libs/signals/src/$f " >> $MAKEFILEAM
done
echo >> $MAKEFILEAM
echo "### Extracting boost includes"
$BOOST_SRC/dist/bin/bcp --scan --boost=$BOOST_SRC ../src/*/*.[Ch] ../src/boost/libs/*/src/*.cpp ../src/smart_assert/smart_assert/priv/fwd/*.hpp $DEST_DIR
if [ $? != "0" ]; then
echo "### bcp failed"
rm -rf $DEST_DIR
exit 1;
fi

Have you considered just writing a build script for a build system like SCons?
You could write a python script to download boost, unpack it compile the needed files (you can even run bjam if needed) and compile your own code.
The only dependency your colleagues will need is Python and SCons.

Run the preprocessor on your code and save the output. If you started with one main.cpp with a bunch of includes in it, you will end up with one file where all of the includes have been sucked in. If you have multiple cpp files, you will have to concatinate them together and then run the preprocessor on the concatinated file, this should work as long as you don't have any duplicate global symbol names.
For a more portable method, do what sqlite does and write your own script to just combine and concatinate together the files you created+boost, and not get the system includes. See mksqlite3c.tcl in the sqlite code
http://www2.sqlite.org/src/finfo?name=tool/mksqlite3c.tcl

Why not just check in all the necessary files to SVN, and send you co-workers the URL of the repository? Then they can check out the code whenever they want to, do an 'svn up' any time they want to update to the latest version, etc.

If you're on a Debian-derived variety of Linux, well problems like this just shouldn't come up: let the packaging system and policy manual do the work. Just make it clear that the libboost-dev or whatever package is a build-dependency of your code and needs to be installed beforehand, and then /usr/include/boost should be right there where your code expects to find it. If you're using a more recent version of boost than the distro ships, it's probably worth figuring out how to package it yourself and work within the existing packaging/dependencies framework rather than reinventing another one.
I'm not familiar enough with .rpm based distros to comment on how things work there. But knowing I can easily setup exactly the build environment I need is, for me, one of the biggest advantages of Debian based development over Windows.

Related

How to save the build time of a C++ DLL

My project is a C++ DLL (cmake project), that expose methods from a c interface to another C# project,
I want to add a methode getBuildTime() in the c interface that return the buildtime (the exact date and time) so from the C# project we can know when the DLL was built.
Is there a way to do this stuff ?
I don't know if this useful or no but I'm using git as a version control system
Build time in C/C++ code can be obtained by built-in macros __DATE__ and __TIME__, e.g. do something like string build_time = __DATE__ " " __TIME__;.
Also file which contains these macros usage needs to be touched before each build to change file's modify time. touch is a unix command, which can be avaialble in Windows too through Cygwin or other means. This is needed to force it to recompile to use new date. Do file touching by adding touch build_time.cpp command to Pre-Build Events, they should be located in your project config somewhere. Alternatively you may add touching to your make file. Altertnatively some compile environments and possibly MSVC too can configure which files to rebuild always which is tweaked inside project's settings.
Also if you use git version control system then you might want to use last git commit hash and time together or instead of build time above. Git commit hash and time is sometimes better than to use build time like suggested above, because git hash and time remain same on each re-compile before next commit, this ensures that your DLL build time changes only when code changes/committed, this may help to have reproducible DLL compiles. To support git hash and time do next things:
If you have unix echo and bash and git commands in your system e.g. by installing Cygwin or new native Windows SubSystem for Linux then you may do next things in command script before_build.cmd that you run before build, also this shell script can be run on Unix systems almost without modifications:
linux_echo -n "" > cfg.h
linux_echo -n "#define GIT_COMMIT """ >> cfg.h
linux_bash -c "echo -n $(git rev-parse --short=8 HEAD)" >> cfg.h
linux_echo """" >> cfg.h
linux_echo -n "#define GIT_COMMIT_TIME """ >> cfg.h
linux_bash -c "echo -n $(git show -s --format=%%ci $(git rev-parse --short=8 HEAD) | tr -d '\n')" >> cfg.h
linux_echo """" >> cfg.h
linux_echo -n "#define COMPILE_TIME """ >> cfg.h
linux_bash -c 'echo -n $(date +"%%Y-%%m-%%d %%H:%%M:%%S %%z")' >> cfg.h
linux_echo """" >> cfg.h
and you will get cfg.h like:
#define GIT_COMMIT "fe0a7891"
#define GIT_COMMIT_TIME "2020-05-13 17:42:55 +0100"
#define COMPILE_TIME "2020-05-13 18:20:28 +0100"
these config file and macros you can use in your code.
The C++ preprocessor predefines several macros (backward compatible with C) including __DATE__ and __TIME__. These expand to string literals containing the date and time of the build (in particular, the date and time that the preprocessor runs on that particular file).
You may need to force the compilation unit containing them to be rebuilt on every change. With make that can be done with .PHONY, with cmake it looks like add_custom_target is the ticket.
Add a target with no output so it will always be built.
Besides using __DATE__ and __TIME__ to capture when the compiler ran, the linker will store a timestamp in the PE header of the DLL it creates. There's no configuration necessary to make this happen, but then you have to write code to read the PE header and convert the timestamp to your favorite date and time format.
The preprocessor solution is way simpler, but the following approach may be convenient, if you need to provide other properties of the build system as well.
Create a cmake script file that generates a source file. Use this script to generate the source file providing the info:
generate_timestamp.cmake
#parameter: OUTPUT_FILE
string(TIMESTAMP _TIMESTAMP "%Y-%m-%dT%H:%M:%S")
file(WRITE ${OUTPUT_FILE} "#include \"buildtime.h\"
const char* getBuildTime() {
return \"${_TIMESTAMP}\";
}")
CMakeLists.txt
set(_BUILD_TIME_FILE buildtime.cpp)
add_library(mylib ${_BUILD_TIME_FILE} ...)
set_source_files_properties(${_BUILD_TIME_FILE} PROPERTIES GENERATED 1)
get_filename_component(_BUILD_TIME_FILE_ABSOLUTE ${_BUILD_TIME_FILE} ABSOLUTE)
# generate source using cmake script file
add_custom_target(buildtime_info COMMAND "${CMAKE_BUILD_TOOL}" -D "OUTPUT_FILE=${_BUILD_TIME_FILE_ABSOLUTE}" -P generate_timestamp.cmake)
# generate file before building mylib
add_dependencies(mylib buildtime_info)

How to deploy Qt applications for Linux

I followed all the steps successfully as mention in the Qt documentation:
Qt for Linux/X11 - Building from Source
Qt for Linux/X11 -
Deployment
But I still couldn't make static Qt application, the executable generated by the above documented steps still needs Qt shared objects on other system.
Any ideas?
You need to deploy the application, for this purpose I use the utility
cqtdeployer
This utility itself collects all the necessary dependencies of your application and you do not have to spend your time on it, or you can automate this process.
You can install from github releases (Windows)
or
from snapstore (Linux)
sudo snap install cqtdeployer
You can use as follows:
Windows:
%cqtdeployer% -bin myApp -qmake path/to/Qt/5.x.x/build/bin/qmake.exe -qmlDir path/to/my/qml/files/dir
Linux:
cqtdeployer -bin myApp -qmake path/to/Qt/5.x.x/build/bin/qmake -qmlDir path/to/my/qml/files/dir
path/to/Qt/5.x.x/build/bin/qmake - This is the way qmake is used to build your program.
path/to/my/qml/files/dir - this is the path directly to your qml file (which you wrote)
And Run application with sh script (Linux) or exe (Windows)
If you'll use the version from snap then make sure that you have all the permissions.
If you need use windows version just install application from installer
Updated
If you want create a simple installer for you application just add qif option for command of cqtdeployer.
Example :
cqtdeployer -bin myApp -qmake path/to/Qt/5.x.x/build/bin/qmake -qmlDir path/to/my/qml/files/dir qif
Details on all the intricacies of cqtdeployer can be found on the official wiki project.
The best way to deploy your application is not necessarily to statically link it for the following reasons:
LGPL licencing means that your application must now be made public and may not sold (I think) - i.e. since its statically linked and the qt libs are within your executable your executable is now part of the open source.
Its a massive pain in the arse... I have gone around this loop and know the pain well.
Installing qt-everywhere is also not so great, I just don't see how you can garantee that the libraries will be the same version as the ones that your program needs.
So what I started to do was create my own script to deploy qt for me. The basic "jist" of this is that you use ldd to find out which qt libraries you need and copy them into a sub folder (./lib) within the same folder as your executable to make an install bundle.
Note: on Windows there is a deployqt application which does somthing similar (can't remember exactly what it is called).
Below I have copied a version of my deploy script. Note that it is quite old now, but I don't see why it should not work (its not written particularly well), but if not it will give you a start place. Also look out for the plugin's. In this script I have added code to copy the audio plugin since I was using that. If you are using other plugins then you will need to copy those (they are usually in sub dir's of the qt libs like .../audio)... I had a todo to try to figure out what plugins are needed from the .pro file but I never got around to that (I would have to pass in the .pro file to this script as well)...
To run, just run this script and pass in the directory that your executable lives in.
#!/bin/bash
# Rememeber start dir
START_DIR=$PWD
# Determine which dir to deploy in and cd to that dir
if [ -d "$1" ]; then
DEPLOY_DIR=$1
else
DEPLOY_DIR=$PWD
fi
echo "Deploy dir: $DEPLOY_DIR"
cd $DEPLOY_DIR
# Run ldd on all files in the directory and create a list of required qt libs
flag=false
for entry in `ldd $DEPLOY_DIR/* | grep -i qt`; do
if $flag; then
# Only add to the array if it is not already in it
if ! [[ $libsArray =~ $entry ]]; then
echo "adding $entry"
libsArray="$libsArray $entry"
fi
flag=false
fi
# If we see a "=>" then the next line will be a library
if [ $entry == "=>" ]; then
flag=true
fi
done
echo
echo
# Create the required folder structure. Note here we are need the qt audio plugin so we are going to manually copy that as well.
mkdir -p lib
mkdir -p lib/audio
# Now copy these files to the deploy directory
for entry in $libsArray; do
echo "cp -v -f $entry $DEPLOY_DIR/lib"
cp -v -f $entry $DEPLOY_DIR/lib
done
# Now get the audio lib - this is a plugin that we are using so we need these libs as well.
# Add other plugins here as well.
# TODO: maybe we can read this in from the *.pro file.
cp -v -f `qmake -query QT_INSTALL_BINS`/../plugins/audio/* $DEPLOY_DIR/lib/audio
# Go back to start dir
cd $START_DIR
Once you have all the files you need you should be able to copy the whole lot to another PC and run it. Note: you may have to set the export LD_LIBRARY_PATH=<path-to-libs> so that the libs can be found... or install the libs into somewhere like /usr/lib/your-appplication/.
But installing libs is another question/subject!

How to check if the user has the library in makefile

For example, if I developed a c++ project and put it on GitHub as a open source project. In my project I used log4cpp as a 3rd-party library to do log stuff. Interested users would download my project and run "make" and suddenly found their machines just don't recognize log4cpp.
I need you guys to verify my original thoughts:
Unlike Java which you may simply include the 3rd-party jar file in your package, I know that C++ is kind of different: you have to let the client compile the library by themselves. It is obvious that, if I do not have the source code from the 3rd-party, I simply cannot help the client to install the libraries. Instead, I need to check if the user has already installed those required library and if not, just warn them not proceed.
If these statements were true, what tools should I consider to use to check? Also, an add-on question: if I want to produce a non-open source product, what should I do? (Since the client need my source code to compile.) If not(Especially the compile by themselves part), please give me your explanation.
Thanks!
Example makefile:
ifeq "$(shell echo '\#include <readline/readline.h>\nint main(){return 0;}' | $(CC) -x c -Wall -O -o /dev/null > /dev/null 2> /dev/null - && echo $$? )" "0"
HAS_FILE = yes
else
HAS_FILE = no
endif
all:
echo has_file=$(HAS_FILE)
I will not say that this is a recommended method, I just use it for simple checks. Autoconf may be a better option.

Change build options for Mac vs. Linux in R package

I'm creating an R package that uses a third-party (closed-source) API for importing .edf files into R (from SR Research Eyelink eye trackers). Someone who has already gotten this to work in Linux has shared his code, and I was able to get it to work on Mac. It was a matter of changing the src/Makevars files to point to the API as it's installed on the mac:
PKG_LIBS=-framework edfapi -F/Library/Frameworks/
To make it work in linux, Makevars needs to have:
PKG_LIBS=-L/usr/local/lib -ledfapi -lm
I know that for windows-specific options, I need to create a Makevars.win file, but how do I have the build options change for Mac versus Linux? I would like to do something like:
if [[ `uname` -eq Darwin ]] ; then
PKG_LIBS=-framework edfapi -F/Library/Frameworks/
fi
if [[ `uname` -eq Linux ]] ;then
PKG_LIBS=-L/usr/local/lib -ledfapi -lm
fi
but putting this into Makevars doesn't work. From researching this it seems that I need a combination of setting options in configure and Makevars, but I haven't quite figured it out. I am comfortable with R programming and know just enough C++ to make some basic functions, but I still don't understand all the nuances involved in the building process. If someone could explain the main purpose of configure/configure.ac versus Makevars/Makevars.in that would be helpful as well.
Ideally I would like to bundle the API along with the R package and have the different versions in a platform-specific folder. The API consists of just 3 header files and a binary (and it rarely changes). I realize this would prevent me from putting the package on CRAN but that is fine. I've managed to successfully build the package with the API files in a different folder, but at runtime it still looks for it in the standard spot (/Library/Frameworks). I realize this is a more loaded question and I can create a separate post as well.
This post helped me figure it out: stackoverflow.com/a/32590600/1457051
configure (in the package root directory) looks like this:
#!/bin/bash
#make the Makevars file
if [ ! -e "./src/Makevars" ]; then
touch ./src/Makevars
fi
#if mac
if [[ `uname` -eq Darwin ]] ; then
echo "PKG_LIBS=-framework edfapi -F/Library/Frameworks/" > ./src/Makevars
#if linux
elif [[ `uname` -eq Linux ]] ;then
echo "PKG_LIBS=-L/usr/local/lib -ledfapi -lm" > ./src/Makevars
fi
Makevars is created and the appropriate options are added based on the platform. There may be a more direct solution, but this works for my purposes.

Can I add Boost source+header files to my own (Open Source) project? [closed]

Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
This question does not appear to be about programming within the scope defined in the help center.
Closed 7 years ago.
Improve this question
Is it allowed by the Boost License to just add the source code of the stuff I need to my project (accompanied by the license of course?). I couldn't find any "descriptive" confirmation. I would have seperate Include/boost and Source/boost directories for easy access.
PS: Seeing as boost::filesystem is going into C++0x TR2, and lambda, regex and whatnot are already in c++0x TR1, I don't see any reason to be juggling with C functions like realpath, getcwd and the like. They don't work well anyways...
UPDATE: I'm adding the required files "one-by-one" folder by folder, but the amount is gigantic. Maybe it's better to include the full boost dist....
Yes you can do that. The license is very liberal. The only condition is that if you redistribute your software in source form you need to include a complete copy of the license.
There is a utility that comes with boost called bcp, that can scan your source and extract any boost header files that are used from the boost source. I've setup a script that does this extraction into our source tree, so that we can package the source that we need along with our code. It will also copy the boost source files for a couple of boost libraries that we use that are no header only, which are then compiled directly into our applications. This makes it easy to rerun the script whenever there is a new version of boost released.
For reference, here is what we use
#!/bin/sh
BOOST_SRC=.../boost_1_43_0
DEST_DIR=../src/boost
TOOLSET=
if ( test uname = "Darwin") then
TOOLSET="--toolset=darwin"
fi
# make bcp if necessary
if ( ! test -x $BOOST_SRC/dist/bin/bcp ) then
if ( test -x $BOOST_SRC/tools/jam/*/bin.*/bjam ) then
BJAM=$BOOST_SRC/tools/jam/*/bin.*/bjam
else
echo "### Building bjam"
pushd $BOOST_SRC/tools/jam
./build_dist.sh
popd
if ( test -x $BOOST_SRC/tools/jam/*/bin.*/bjam ) then
BJAM=$BOOST_SRC/tools/jam/*/bin.*/bjam
fi
fi
echo "BJAM: $BJAM"
pushd $BOOST_SRC/tools/bcp
echo "### Building bcp"
echo "$BJAM $TOOLSET"
$BJAM $TOOLSET
if [ $? == "0" ]; then
exit 1;
fi
popd
fi
if ( ! test -x $BOOST_SRC/dist/bin/bcp) then
echo "### Couldn't find bpc"
exit 1;
fi
mkdir -p $DEST_DIR
echo "### Copying boost source"
MAKEFILEAM=$DEST_DIR/libs/Makefile.am
rm $MAKEFILEAM
# copy source libraries
mkdir -p $DEST_DIR/libs/signals/src
cp $BOOST_SRC/libs/signals/src/* $DEST_DIR/libs/signals/src/.
echo -n "boost_sources += " >> $MAKEFILEAM
for f in `ls $DEST_DIR/libs/signals/src | fgrep .cpp`; do
echo -n "boost/libs/signals/src/$f " >> $MAKEFILEAM
done
echo >> $MAKEFILEAM
echo "### Extracting boost includes"
$BOOST_SRC/dist/bin/bcp --scan --boost=$BOOST_SRC ../src/*/*.[Ch] $DEST_DIR
if [ $? != "0" ]; then
echo "### bcp failed"
rm -rf $DEST_DIR
exit 1;
fi
i would recommend linking to boost externally instead of include the source directly into your projects.
beside the huge spiderweb dependency issue, linking them externally mean you can always refer to latest stable build (assume u checkout from the repository) without explicitly overwrite each old source in your project.