I got sick of looking up the magic symbols in make and decided to try waf.
I'm trying to use calibre to make ebooks and I'd like to create a wscript that takes in a file, runs a program with some arguments that include that file, and produces an output. Waf should only build if the input file is newer than the output.
In make, I'd write a makefile like this:
%.epub: %.recipe
ebook-convert $ .epub --test -vv --debug-pipeline debug
Where % is a magic symbol for the basename of the file and $ a symbol for the output filename (basename.epub).
I could call make soverflow.epub and it would run ebook-convert on soverflow.recipe. If the .recipe hadn't changed since the last build, it wouldn't do anything.
How can I do something similar in waf?
(Why waf? Because it uses a real language that I already know. If this is really easy to use in scons, that's a good answer too.)
I figured out how to make a basic wscript file, but I don't know how to build targets specified on the command-line.
The Waf Book has a section on Task generators. The Name and extension-based file processing section gives an example for lua that I adapted:
from waflib import TaskGen
TaskGen.declare_chain(
rule = 'ebook-convert ${SRC} .epub --test -vv --debug-pipeline debug',
ext_in = '.recipe',
ext_out = '.epub'
)
top = '.'
out = 'build'
def configure(conf):
pass
def build(bld):
bld(source='soverflow.recipe')
It even automatically provides a clean step that removes the epub.
Related
My project folder has the following structure
-Project/
/src
-Main.cpp
-MyReader.cpp
/headers
-MyReader.h
/DataFiles
-File.dat
-File1.dat
My class Object.cpp has a couple of methods which reads from File.dat and File1.dat and parse the information to Map objects. My problem is that I am using Autotools (in which I'm very very newbie) for generating config and installer files and I don't know how to make all the DataFiles files accessible for the program after installation. The program doesn't work properly because of the code fails when trying to read those files through relative paths. Locally, the program runs perfectly after executing in terminal make && ./program.
How can I solve this issue? Thanks in advance for your help!
A platform independent way to do this with Autotools is using the $(datadir) variable to locate the system data directory and work relative to that.
So in your Makefile.am file you can create a name like this:
myprog_infodir = $(datadir)/myprog
# Set a macro for your code to use
myprog_CXXFLAGS = -DDATA_LOCATION=\"$(datadir)/myprog\"
# This will install it from the development directories
myprog_info_DATA = $(top_srcdir)/DataFiles/File.dat $(top_srcdir)/DataFiles/File1.dat
# make sure it gets in the installation package
extra_DIST = $(top_srcdir)/DataFiles/File.dat $(top_srcdir)/DataFiles/File1.dat
Then in your program you should be able to refer to the data like this:
std::ifstream ifs(DATA_LOCATION "/File.dat");
Disclaimer: Untested code
I figured out one method and will give my example here:
In my Makefile.am
AM_CPPFLAGS = -D MATRIXDIR="\"$(pkgdatadir)/matrix\""
nobase_dist_pkgdata_DATA = matrix/AAcode.txt \
matrix/BLOSUM50 matrix/BLOSUM70.50 matrix/BLOSUM100 matrix/BLOSUM50.50 \
matrix/BLOSUM75 matrix/BLOSUM100.50 matrix/BLOSUM55 matrix/BLOSUM75.50 \
... more not shown
I put quite some number of datafiles in the matrix directory, just show a few of them. In my source file, I simply use the macro MATRIXDIR:
scorematrix.cpp:string MatrixScoreMethod::default_path=MATRIXDIR;
This seems to work well for me. You can use other versions of the data automake variable, such as dist_data_DATA instead of pkgdata. It is a good idea to use pkgdata this way your data will not be mixed with other packages. The nobase_ is to tell automake not to strip the matrix directory during install. Those escaped double quotes seems to be needed for string type so that you don't get compiler errors.
I have the following directory structure:
my_dir
|
--> src
| |
| --> foo.cc
| --> BUILD
|
--> WORKSPACE
|
--> bazel-out/ (symlink)
|
| ...
src/BUILD contains the following code:
cc_binary(
name = "foo",
srcs = ["foo.cc"]
)
The file foo.cc creates a file named bar.txt using the regular way with <fstream> utilities.
However, when I invoke Bazel with bazel run //src:foo the file bar.txt is created and placed in bazel-out/darwin-fastbuild/bin/src/foo.runfiles/foo/bar.txt instead of my_dir/src/bar.txt, where the original source is.
I tried adding an outs field to the foo rule, but Bazel complained that outs is not a recognized attribute for cc_binary.
I also thought of creating a filegroup rule, but there is no deps field where I can declare foo as a dependency for those files.
How can I make sure that the files generated by running the cc_binary rule are placed in my_dir/src/bar.txt instead of bazel-out/...?
Bazel doesn't allow you to modify the state of your workspace, by design.
The short answer is that you don't want the results of the past builds to modify the state of your workspace, hence potentially modifying the results of the future builds. It'll violate reproducibility if running Bazel multiple times on the same workspace results in different outputs.
Given your example: imagine calling bazel run //src:foo which inserts
#define true false
#define false true
at the top of the src/foo.cc. What happens if you call bazel run //src:foo again?
The long answer: https://docs.bazel.build/versions/master/rule-challenges.html#assumption-aim-for-correctness-throughput-ease-of-use-latency
Here's more information on the output directory: https://docs.bazel.build/versions/master/output_directories.html#documentation-of-the-current-bazel-output-directory-layout
There could be a workaround to use genrule. Below is an example that I use genrule to copy a file to the .git folder.
genrule(
name = "precommit",
srcs = glob(["git/**"]),
outs = ["precommit.txt"],
# folder contain this BUILD.bazel file is tool which will be symbol linked, we use cd -P to get to the physical path
cmd = "echo 'setup pre-commit.sh' > $(OUTS) && cd -P tools && ./path/to/your-script.sh",
local = 1, # required
)
If you're passing the name of the output file in when running, you can simply use absolute paths. To make this easier, you can use the realpath utility if you're in linux. If you're on a mac, it is included in brew install coreutils. Then running it looks something like:
bazel run my_app_dir:binary_target -- --output_file=`realpath relative/path/to.output
This has been discussed and explained in a Bazel issue. Recommendation is to use a tool external to Bazel:
As I understand the use-case, this is out-of-scope for building and in the scope of, perhaps, workspace configuration. What I'm sure of is that an external tool would be both easier and safer to write for this purpose, than to introduce such a deep design change to Bazel.
The tool would copy the files from the output tree into the source tree, and update a manifest file (also in the source tree) that lists the path-digest pairs. The sources and the manifest file would all be versioned. A genrule or a sh_test would depend on the file-generating genrules, as well as on this manifest file, and compare the file-generating genrules' outputs' digests (in the output tree) to those in the manifest file, and would fail if there's a mismatch. In that case the user would need to run the external tool, thus update the source tree and the manifest, then rerun the build, which is the same workflow as you described, except you'd run this tool instead of bazel regenerate-autogenerated-sources.
Is it possible to glob source code files in a meson build?
Globbing source files is discouraged and is bad practice, and not only on Meson. It causes weird errors, makes it hard to have some developement files aside for development but that you don't want to build or ship, and can cause problems with incremental builds.
Explicit is better than implicit.
2021-03-02 EDIT:
Read also Why can't I specify target files with a wildcard? in the Meson FAQ.
Meson does not support this syntax and the reason for this is simple. This can not be made both reliable and fast.
If after all the warnings, you still want to do it at your own risk, the FAQ tells you how in But I really want to use wildcards!. You just use an external script to do the globbing and return the list of files (that script is called grabber.sh in that example).
c = run_command('grabber.sh')
sources = c.stdout().strip().split('\n')
e = executable('prog', sources)
I found an example in the meson unit tests showing how to glob source, but in the comments it says this is not recommended.
if build_machine.system() == 'windows'
c = run_command('grabber.bat')
grabber = find_program('grabber2.bat')
else
c = run_command('grabber.sh')
grabber = find_program('grabber.sh')
endif
# First test running command explicitly.
if c.returncode() != 0
error('Executing script failed.')
endif
newline = '''
'''
sources = c.stdout().strip().split(newline)
e = executable('prog', sources)
The reason this is not recommended: attempting to add files by glob'ing a directory will NOT make those files automatically appear in the build. You have to manually re-invoke meson for the files to be added to the build. Re-invoking ninja or other back-ends is not sufficient, you must reinvoke meson itself.
meson.build
glob = run_command('python', 'glob')
sources = glob.stdout().strip().split('\n')
glob:
import glob
sources = glob.glob('./src/*.cpp') + glob.glob('./src/**/*.cpp')
for i in sources:
print(i)
No it's not possible. Every sources have to be explicitly stated to build a target.
I am trying to create an automated process with Waf to optimize, minify, etc. the source files of a website based on the HTML5 boilerplate's ANT build script. Part of this includes running all the PNG's in the img directory through two utilities, optipng and advpng.
This is my current best attempt at these tasks:
def build(ctx):
ctx(
rule = '${OPTIPNG} -quiet -o5 -i 1 -out ${TGT} ${SRC}',
source = 'img1.png',
target = 'img1-opt.png'
)
ctx(
rule = '${ADVPNG} -z -1 ${SRC}',
source = SOMETHING,
target = SOMETHING ELSE
)
I first run optipng on img1, where my first problem arises. I would like the output file to have the same name as the input file. However, setting target to the same name results in Waf detecting a deadlock. So, I moved ahead by appending a suffix.
advpng is a bit strange in that it doesn't create a new output file: it modifies the input file in place. So, I now need a way to access the output of optipng, which now resides in the build's output directory.
What is the proper way of accomplishing this task?
The first problem is addressed in part by the waf book section 6.3. That is about copying, so you have to tweak it. Part of section 11 on Providing arbitrary configuration files is somewhat relevant, as well.
You have to write some Python to resolve the target file in the build directory. This script works:
top = '.'
out = 'build'
def configure(ctx):
pass
def build(ctx):
txt = ctx.srcnode.find_node('text.txt')
ctx(
rule = 'cp ${SRC} ${TGT}',
source = txt,
target = txt.get_bld() # Or: ctx.bldnode.make_node('text.txt')
)
ctx(
rule = 'wc ${SRC} > ${TGT}',
source = 'text.txt', # Or: txt.get_bld(),
target = 'count.txt'
)
For the second step, waf seemed to resolve the literal source file name in the build directory, but you could also use the bld node from step 1 to resolve it explicitly.
How can I add a include path to wscript?
I know I can declare which files from which folders I want to include per any cpp file, like:
def build(bld):
bld(features='c cxx cxxprogram',
includes='include',
source='main.cpp',
target='app',
use=['M','mylib'],
lib=['dl'])
but I do not want to set it per every file. I want to add a path to "global includes" so it will be included everytime any file will be compiled.
I've found an answer. You have to simply set the value of 'INCLUDES' to list of paths you want.
Do not forget to run waf configure again :)
def configure(cfg):
cfg.env.append_value('INCLUDES', ['include'])
I spent some time working out a good way to do this using the "use" option in bld.program() methods. Working with the boost libraries as an example, I came up with the following. I hope it helps!
'''
run waf with -v option and look at the command line arguments given
to the compiler for the three cases.
you may need to include the boost tool into waf to test this script.
'''
def options(opt):
opt.load('compiler_cxx boost')
def configure(cfg):
cfg.load('compiler_cxx boost')
cfg.check_boost()
cfg.env.DEFINES_BOOST = ['NDEBUG']
### the following line would be very convenient
### cfg.env.USE_MYCONFIG = ['BOOST']
### but this works too:
def copy_config(cfg, name, new_name):
i = '_'+name
o = '_'+new_name
l = len(i)
d = {}
for key in cfg.env.keys():
if key[-l:] == i:
d[key.replace(i,o)] = cfg.env[key]
cfg.env.update(d)
copy_config(cfg, 'BOOST', 'MYCONFIG')
# now modify the new env/configuration
# this adds the appropriate "boost_" to the beginning
# of the library and the "-mt" to the end if needed
cfg.env.LIB_MYCONFIG = cfg.boost_get_libs('filesystem system')[-1]
def build(bld):
# basic boost (no libraries)
bld.program(target='test-boost2', source='test-boost.cpp',
use='BOOST')
# myconfig: boost with two libraries
bld.program(target='test-boost', source='test-boost.cpp',
use='MYCONFIG')
# warning:
# notice the NDEBUG shows up twice in the compilation
# because MYCONFIG already includes everything in BOOST
bld.program(target='test-boost3', source='test-boost.cpp',
use='BOOST MYCONFIG')
I figured this out and the steps are as follows:
Added following check in the configure function in wscript file. This tells the script to check for the given library file (libmongoclient in this case), and we store the results of this check in MONGOCLIENT.
conf.check_cfg(package='libmongoclient', args=['--cflags', '--libs'], uselib_store='MONGOCLIENT', mandatory=True)
After this step, we need to add a package configuration file (.pc) into /usr/local/lib/pkgconfig path. This is the file where we specify the paths to lib and headers. Pasting the content of this file below.
prefix=/usr/local
libdir=/usr/local/lib
includedir=/usr/local/include/mongo
Name: libmongoclient
Description: Mongodb C++ driver
Version: 0.2
Libs: -L${libdir} -lmongoclient
Cflags: -I${includedir}
Added the dependency into the build function of the sepcific program which depends on the above library (i.e. MongoClient). Below is an example.
mobility = bld( target='bin/mobility', features='cxx cxxprogram', source='src/main.cpp', use='mob-objects MONGOCLIENT', )
After this, run the configure again, and build your code.