I have my project targets (binaries and libraries) specified in json files.
I can create an environment for the specified target without any problem. Now I'm trying to support a specific build directory.
My knowledge about scons is still very basic but it seems that the right way to do that is using a SConscript together with VariantDir. But I already have my targets specified in json and creating a SConscript file for each target would be redundant (plus a cost in maintenance).
So my question is: is it possible to create a SConscript object dynamically, at run time?
Thanks in advance.
VariantDir doesn't work with SConscturct file (may be i wrong, but i don't found any way to do it).
Just create SConscript file with variant dir and doing what you need.
#SConsruct
env = CreateEnvironment()
SConscript('SConscript', variant_dir = 'mybuilddir', exports = 'env', duplicate = 0)
# Do all work in SConscript
Import('env')
env.Program(...)
env.SharedLibrary(...)
...
Also, you can split your process into 2 states.
State 1 - generated SConscript files.
State 2 - run generated SConscript files.
if 'generate' in COMMAND_LINE_TARGETS:
# your code to generated SConscript from json
Exit(0)
sconscriptFiles = getSconscriptFiles() # some code to get your sconscript, by mask for example
if len(sconscriptFiles) < 1:
print "You need to generate files at first: scons generate"
Exit(1)
for file in sconscriptFiles :
SConscript(file, variant_dir = 'build' + file, duplicate = 0)
You can specify the build directory with the VariantDir() function, or as part of the SConscript() call. All of the different options are discussed here. Considering you dont want to use several SConstruct files, you should just use the VariantDir() function as described in more detail here.
Here is a simple example:
env = Environment()
# It may be as simple as setting src_dir='.', but set accordingly
# duplicate=0 tells SCons NOT to copy source files to variantDir, set accordingly
# VariantDir() can be called multiple times so as to change dirs per builder call
VariantDir(variant_dir = 'pathToBuildDir', src_dir = 'pathToSource', duplicate=0)
# Now call the builders here
Its still not clear why you want to mix json with SCons. Unless you have some very compelling reasons to do so, I would suggest keeping it all in SCons, which is Python.
EDIT: I just realized you asked about creating a SConscript object, not a file.
I looked through the SCons programming APIs and didnt find anything that allows you to create a SConscript object. Actually, I dont think the concept of a SConscript object exists, since it just treats calls to the SConscript() function as files that need to be opened and processed, and they are almost treated as an extension to the SConstruct.
So, to summarize: You'll either have to create subsidiary SConscript files, or work with calls to VariantDir(). Depending on your project directory structure, it may not be necessary to create SConscript files. You could just do everything from the root SConstruct. The SConscript files arent necessary, they just help organize the build scripts better.
Related
Context:
When I want to build a projet, there are usually CMakeCache.txt and CMakeVars.txt files that are created at a first stage, e.g. using cmake-gui > configure.
Questions:
What are the differences between these files and what are they used for?
If I have tweaked a lot of options within cmake-gui, which file should I safely backup in an other location (for copying it back later on in order to not spend much time figuring out which options I had checked...) before removing the build/ directory contents (this directory normally includes these two files) if I want to start from a clean build/ directory (which is sometimes needed) ?
I would like to edit an existing software to add a new source file (Source.cpp).
But, I can't manage the compilation process (it seems to be automake and it looks very complicated).
The software (iperf 2: https://sourceforge.net/projects/iperf2/files/?source=navbar) is compiled using a classical ./configure make then make install.
If I just add the file to the corresponding source and include directory, I got this error message:
Settings.cpp:(.text+0x969) : undefined reference to ...
It looks like the makefile isn't able to produce the output file associated with my new source file (Source.cpp). So, I probably need to indicate it manually somewhere.
I searched a bit in the project files and it seemed that the file to edit was: "Makefile.am".
I added my source to the variable iperf_SOURCES in that file but it didn't workded.
Could you help me to find the file where I need to indicate my new source file (it seems a pretty standard compilation scheme but I never used automake softwares and this one seems very complicated).
Thank you in advance
This project is built with the autotools, as you already figured out.
The makefiles are built by automake. It takes its input in files that usually have a am file name extension.
The iperf program is built by the makefile generated from src/Makefile.am. This is indicated by:
bin_PROGRAMS = iperf
All (actually this is a simplification, but which holds in this case) source files of a to be built binary are in the corresponding name_SOURCES variable, thus in this case iperf_SOURCES. Just add your source file to the end of that list, like so (keeping their formatting):
iperf_SOURCES = \
Client.cpp \
# lines omitted
tcp_window_size.c \
my_new_file.c
Now, to reflect this change in any future generated src/Makefile you need to run automake. This will modify src/Makefile.in, which is a template that is used by config.sub at the end of configure to generate the actual makefile.
Running automake can happen in various ways:
If you already have makefiles that were generated after an configure these should take care of rebuilding themselves. This seems to fail sometimes though!
You could run automake (in the top level directory) by hand. I've never done this, as there is the better solution to...
Run autoreconf --install (possibly add --force to the arguments) in the top level directory. This will regenerate the entire build system, calling all needed programs such as autoheader, autoconf and of course automake. This is my favorite solution.
The later two options require calling configure again, IMO ideally doing an out of source built:
# in top level dir
mkdir build
cd build
../configure # arguments
make # should now also compile and link your new source file
my autotools project has a couple of unit-tests.
one of these tests (filereader) needs to read a file (data/test1.bin)
Here's my filesystem layout:
- libfoo/tests/filereader.c
- libfoo/tests/data/test1.bin
and my libfoo/tests/Makefile.am:
AUTOMAKE_OPTIONS = foreign
AM_CPPFLAGS = -I$(top_srcdir)/foo
LDADD = $(top_builddir)/src/libfoo.la
EXTRA_DIST = data/file1.bin
TESTS = filereader
check_PROGRAMS= filereader
filereader_SOURCES = filereader.c
this works great, as long as i do in-tree builds.
However, when running the test-suite out-of-tree (e.g. make distcheck), the filereader test cannot find the input file anymore.
This is obviously because only the source tree contains the input file, but not the build tree.
i wonder what is the canonical way to fix this problem?
compile the directory of the test-file into the unittest (AM_CPPFLAGS+=-DSRCDIR=$(srcdir))
pass the qualified input file as a cmdline argument to the test? (e.g. $(builddir)/filereader $(srcdir)/data/file1.bin)
copy the input file from the source tree to the build tree? (cp $(srcdir)/data/file1.bin $(builddir)/data/file1.bin? how would a proper make-rule look like??)
Canonically, the solution would be to define the path to your file into the unittest, so the first option you laid out. The second one is also possible but it requires using an in-between driver script.
I would suggest avoiding the third one, but if you do want to go down that route, use $(LN_S) rather than cp; this way you reduce the I/O load of the test.
There is a way to do this with autoconf. From the netcdf-c configure.ac:
##
# Some files need to exist in build directories
# that do not correspond to their source directory, or
# the test program makes an assumption about where files
# live. AC_CONFIG_LINKS provides a mechanism to link/copy files
# if an out-of-source build is happening.
##
AC_CONFIG_LINKS([nc_test4/ref_hdf5_compat1.nc:nc_test4/ref_hdf5_compat1.nc])
AC_CONFIG_LINKS([nc_test4/ref_hdf5_compat2.nc:nc_test4/ref_hdf5_compat2.nc])
AC_CONFIG_LINKS([nc_test4/ref_hdf5_compat3.nc:nc_test4/ref_hdf5_compat3.nc])
AC_CONFIG_LINKS([nc_test4/ref_chunked.hdf4:nc_test4/ref_chunked.hdf4])
AC_CONFIG_LINKS([nc_test4/ref_contiguous.hdf4:nc_test4/ref_contiguous.hdf4])
The SCons User Guide tells about the usage of Multiple Construction Environments to build build multiple versions of a single program and gives the following example:
opt = Environment(CCFLAGS = '-O2')
dbg = Environment(CCFLAGS = '-g')
o = opt.Object('foo-opt', 'foo.c')
opt.Program(o)
d = dbg.Object('foo-dbg', 'foo.c')
dbg.Program(d)
Instead of manually assigning different names to the objects compiled with different environments, VariantDir() / variant_dir sounds like a better solution...
But if I place the Program() builder inside the SConscript:
Import('env')
env.Program('foo.c')
How can I export different environments to the same SConscript file?
opt = Environment(CCFLAGS = '-O2')
dbg = Environment(CCFLAGS = '-g')
SConscript('SConscript', 'opt', variant_dir='release') #'opt' --> 'env'???
SConscript('SConscript', 'dbg', variant_dir='debug') #'dbg' --> 'env'???
Unfortunately the discussion in the SCons Wiki does not bring more insight to this topic.
Thanks for your input!
SConscript is a method defined on the environment itself:
for dir, env in (('release', opt), ('debug', dbg)):
env.SConscript('SConscript', 'env', variant_dir=dir)
And then from the SConscript you can:
Import('env')
Alternately, you can pass a dictionary as the exports arg to SConscript. The keys are the name the SConscript will use to import it, and the values are the objects in the SConstruct. So:
SConscript('SConscript', exports={'env': dbg}, variant_dir='debug')
SConscript('SConscript', exports={'env': opt}, variant_dir='release')
then in the SConscript Import('env') will get dbg the first time and opt the second time. This also works for exporting/importing anything else you like, not just env.
See Export() and SConscript() in the man page for more info.
I've recently picked up scons to implement a multi-platform build framework for a medium sized C++ project. The build generates a bunch of unit-tests which should be invoked at the end of it all. How does one achieve that sort of thing?
For example in my top level sconstruct, I have
subdirs=['list', 'of', 'my', 'subprojects']
for subdir in subdirs:
SConscript(dirs=subdir, exports='env', name='sconscript',
variant_dir=subdir+os.sep+'build'+os.sep+mode, duplicate=0)
Each of the subdir has its unit-tests, however, since there are dependencies between the dlls and executables built inside them - i want to hold the running of tests until all the subdirs have been built and installed (I mean, using env.Install).
Where should I write the loop to iterate through the built tests and execute them? I tried putting it just after this loop - but since scons doesn't let you control the order of execution - it gets executed well before I want it to.
Please help a scons newbie. :)
thanks,
SCons, like Make, uses a declarative method to solving the build problem. You don't want to tell SCons how to do its job. You want to document all the dependencies and then let SCons solve how it builds everything.
If something is being executed before something else, you need to create and hook up the dependencies.
If you want to create dmy touch files, you can create a custom builder like:
import time
def action(target, source, env):
os.system('echo here I am running other build')
dmy_fh = open('dmy_file','w')
dmy_fh.write( 'Dummy dependency file created at %4d.%02d.%02d %02dh%02dm%02ds\n'%time.localtime()[0:6])
dmy_fh.close()
bldr = Builder(action=action)
env.Append( BUILDERS = {'SubBuild' : bldr } )
env.SubBuild(srcs,tgts)
It is very important to put the timestamp into the dummy file, because scons uses md5 hashes. If you have an empty file, the md5 will always be the same and it may decide to not do subsequent build steps. If you need to generate different tweaks on a basic command, you can use function factories to modify a template. e.g.
def gen_a_echo_cmd_func(echo_str):
def cmd_func(target,source,env):
cmd = 'echo %s'%echo_str
print cmd
os.system(cmd)
return cmd_fun
bldr = Builder(action = gen_a_echo_cmd_func('hi'))
env.Append(BUILDERS = {'Hi': bldr})
env.Hi(srcs,tgts)
bldr = Builder(action = gen_a_echo_cmd_func('bye'))
env.Append(BUILDERS = {'Bye': bldr})
env.Bye(srcs,tgts)
If you have something that you want to automatically inject into the scons build flow ( e.g. something that compresses all your build log files after everything else has run ), see my question here.
The solution should be as simple as this.
Make the result of the Test builders depend on the result of the Install builder
In pseudo:
test = Test(dlls)
result = Install(dlls)
Depends(test,result)
The best way would be if the Test builder actually worked out the dll dependencies for you, but there may be all kinds of reasons it doesn't do that.
In terms of dependencies, what you want is for all the test actions to depend on all the program-built actions. A way of doing this is to create and export a dummy-target to all the subdirectories' sconscript files, and in the sconscript files, make the dummy-target Depends on the main targets, and have the test targets Depends on the dummy-target.
I'm having a bit of trouble figuring out how to set up the dummy target, but this basically works:
(in top-level SConstruct)
dummy = env.Command('.all_built', 'SConstruct', 'echo Targets built. > $TARGET')
Export('dummy')
(in each sub-directory's SConscript)
Import('dummy')
for target in target_list:
Depends(dummy, targe)
for test in test_list:
Depends(test, dummy)
I'm sure further refinements are possible, but maybe this'll get you started.
EDIT: also worth pointing out this page on the subject.
Just have each SConscript return a value on which you will build dependencies.
SConscript file:
test = debug_environment.Program('myTest', src_files)
Return('test')
SConstruct file:
dep1 = SConscript([...])
dep2 = SConscript([...])
Depends(dep1, dep2)
Now dep1 build will complete after dep2 build has completed.