How to build in different environments with SCons? - build

I just discovered SCons, a great build tool.
I need to build my project in multiple environments, i.e. with different library paths and include paths depending on the machine.
Since SConstruct has all of Python available, I can imagine various ways to accomplish this. One possibility would be to have a single SConstruct script and instantiate multiple Environment objects.
envFoo = Environment()
envFoo.Append(CPPPATH = [...])
envBar = Environment()
envBar.Append(CPPPATH = [...])
Then select one of these Environment objects somehow, possibly with a command line parameter to scons.
A question for experienced scons users: Is this the way to go? What is the most convenient way of doing this?

This would definitely work, and would probably be a good solution for simpler situations. I have a more complex situation where I actually create an EnvironmentFactory using Python classes, something like this:
env = EnvironmentFactory.createEnv(cmdLineArgs)
Another useful SCons tool would be to "automatically" populate the Environment from the command line options using the ParseFlags(), ParseConfig(), and MergeFlags() as described in this chapter.

You could do this
envSelect = ARGUMENTS.get('env', "default")
if envSelect == "default":
env = Environment()
elif envSelect == "env2":
env = Environment(whatever you want to do here)
elif envSelect == "env3":
env = Environment(whatever you want to do here)
else:
env = Environment(whatever you want to do here)
With the ARGUMENTS.get you specify a default when you just run the command scons
or else you you do scons env=env2 hope this answers your question
EDIT : here's a snippet from a project I'm working on where I want to determine the os when scons runs
import sys
import os
import glob
import subprocess
#Find the host Operating System
platform = sys.platform
if platform != "win32":
env = Environment()
else:
#Specify to compile in 32-bit mode for Visual Studio
#This is needed as Qt libraries on Windows for Visual
#Studio are 32-bit only
env = Environment(TARGET_ARCH = 'x86')
#Get Qt directory as scons argument or use default setting
if platform != "win32":
qtDir = ARGUMENTS.get('qt', '/usr/local/Trolltech/Qt-4.8.4/')
else:
qtDir = ARGUMENTS.get('qt', 'C:\\Qt\\4.8.4\\')

Related

How can I implement a command line option in Bazel to switch between which dependency version is used for building?

For some background, the C++ program I am working on has the possibility to interoperate with some other applications that use various Protobuf versions. In the source code for my program, I have the compiled .pb.cc files from these other applications for the Protobuf interface. These .pb.cc files were compiled with a particular version of Protobuf, and I don't have any control over this. I am using Bazel to build, and I want to be able to specify a Bazel build configuration for my program, which will use a particular version of Protobuf which matches that of one of the possible other applications.
Originally, I wanted to put something in the .bazelrc file so that I can specify a particular version of Protobuf depending on the config, for example:
# in .bazelrc:
build:my_config --protobuf_version=3_20_1
build:my_other_config --protobuf_version=3_21_6
Then from the terminal, I could build with the command
bazel build --config=my_config //path/to/target:target
which would build as if I had typed
bazel build --protobuf_version=3_20_1 //path/to/target:target
At this point, I wanted to use the select() function, as detailed in the Bazel docs for Configurable Build Attributes, to use a particular Protobuf version during building. But, the Protobuf dependencies are all specified in the WORKSPACE file, which is more limited than a BUILD file, and this select() function cannot be used there. So then my idea was to pull in every version of the Protobuf library that I would possibly need, and give them different names in the WORKSPACE file, and then in the BUILD files, use a select() function to choose the correct version. But, the Bazel rule for compiling the proto_library is used as such:
proto_library(
name = "foo",
srcs = ["foo.proto"],
strip_import_prefix = "/foo/bar/baz",
)
I don't see of any opportunity to use a select() function here to specify which Protobuf version's proto_library rule should be used. The proto_library rule is also defined in from the WORKSPACE file with:
load("#rules_proto//proto:repositories.bzl", "rules_proto_dependencies", "rules_proto_toolchains")
rules_proto_dependencies()
rules_proto_toolchains()
Now, I would say that I am stuck. I don't see a way to specify on the command line which version of Protobuf should be used with the proto_library rule.
In the end, I would like a way to do the equivalent in the WORKSPACE file of
# in WORKSPACE
if my_config:
# specific protobuf version:
http_archive(
name = "com_google_protobuf",
sha256 = "8b28fdd45bab62d15db232ec404248901842e5340299a57765e48abe8a80d930",
strip_prefix = "protobuf-3.20.1",
urls = ["https://github.com/protocolbuffers/protobuf/archive/v3.20.1.tar.gz"],
)
elif my_other_config:
# same as above, but with different version
else:
# same as above, but with default version
According to some google groups discussion, this doesn't seem to be possible in the WORKSPACE file, so I would need to do it in a BUILD file, but the dependencies are specified in the WORKSPACE.
I figured out a way that works that seems to go against Bazel's philosophy, but most importantly does what I want.
The repository dependencies are loaded in the first of two steps, the first involving the WORKSPACE file, and the second involving the BUILD file. Command line flags for the build cannot be normally be directly passed to the WORKSPACE, but it is possible to get some information to the WORKSPACE by setting an environment variable and creating a repository_rule. In the WORKSPACE, this environment variable can be used, for example, to change the url argument to http_archive which specifies the dependency version.
This repository rule is created in a separate file .bzl file, which is then loaded in the WORKSPACE. As a generalized example of how get environment variable values into the WORKSPACE, the following file my_repository_rule.bzl could be created:
# in file my_repository_rule.bzl
def _my_repository_rule_impl(repository_ctx):
# read the particular environment variable we are interested in
config = repository_ctx.os.environ.get("MY_CONFIG_ENV_VAR", "")
# necessary to create empty BUILD file for this rule
# which will be located somewhere in the Bazel build files
repository_ctx.file("BUILD")
# some logic to do something based on the value of the environment variable passed in:
if config.lower() == "example_config_1":
ADDITIONAL_INFO = "foo"
elif config.lower() == "example_config_2":
ADDITIONAL_INFO = "bar"
else:
ADDITIONAL_INFO = "baz"
# create a temporary file called config.bzl to be loaded into WORKSPACE
# passing in any desired information from this rule implementation
repository_ctx.file("config.bzl", content = """
MY_CONFIG = {}
ADDITIONAL_INFO = {}
""".format(repr(config), repr(ADDITIONAL_INFO ))
)
my_repository_rule = repository_rule(
implementation=_my_repository_rule_impl,
environ = ["MY_CONFIG_ENV_VAR"]
)
This can be used in the WORKSPACE as such:
# in file WORKSPACE
load("//:my_repository_rule.bzl", "my_repository_rule ")
my_repository_rule(name = "local_my_repository_rule ")
load("#local_my_repository_rule //:config.bzl", "MY_CONFIG", "ADDITIONAL_INFO")
print("MY_CONFIG = {}".format(MY_CONFIG))
print("ADDITIONAL_INFO = {}".format(ADDITIONAL_INFO))
When a target is built with bazel build, the WORKSPACE will receive the value of the MY_CONFIG_ENV_VAR from the terminal and store it in the Starlark variable MY_CONFIG, and any other additional information determined in the implementation.
The environment variable can be passed by normal means, such as typing in a bash shell, for example:
MY_CONFIG_ENV_VAR=example_config_1 bazel build //path/to/target:target
It can also be passed as a flag with the --repo_env flag. This flag sends an extra environment variable to be available to the repository rules, meaning the following is equivalent:
bazel build --repo_env=MY_CONFIG_ENV_VAR=example_config_1 //path/to/target:target
This can be made easier to switch between by including the following in the .bazelrc file:
# in file .bazelrc
build:my_config_1 --repo_env=MY_CONFIG_ENV_VAR=example_config_1
build:my_config_2 --repo_env=MY_CONFIG_ENV_VAR=example_config_2
So running bazel build --config=my_config_1 //path/to/target:target will show the debug output from the print statements in WORKSPACE as the following:
MY_CONFIG = example_config_1
ADDITIONAL_INFO = foo
If ADDITIONAL_INFO in the rule implementation (in the file my_repository_rule.bzl) were set to a version number such as "3.20.1", then the WORKSPACE could, for example, use this in an http_archive call to pull the desired version of the dependency.
# in file WORKSPACE
if ADDITIONAL_INFO == "3.20.1":
sha256 = "8b28fdd45bab62d15db232ec404248901842e5340299a57765e48abe8a80d930"
http_archive(
name = "com_google_protobuf",
sha256 = sha256,
strip_prefix = "protobuf-{}".format(ADDITIONAL_INFO),
urls = ["https://github.com/protocolbuffers/protobuf/archive/v{}.tar.gz".format(ADDITIONAL_INFO)],
)
Of course, the value of the sha256 kwarg could also be passed in from the repository rule as a separate string variable, or as part of a dictionary, for example.

Boost Build: Use a feature or a variable

I have a DB integration test that I'm running using Boost Build. The test needs some commandline args (DB username, password). What's the best way to set that via Boost Build in a way that's configurable by the user (via environment variables, bjam commandline, user-config.jam)?
I know I can do this with variables:
import os ;
local DB_PASS = [ os.environ DB_PASS ] ;
run dbtest : test.cpp : --dbpass $(DB_PASS) ;
This can be set via a the commandline (bjam -s DB_PASS=pass) or via an environment variable.
On the other hand, Boost Build tends to do most of its configuration via the feature mechanism. I could probably define a new feature and get the configuration data to the right place that way.
What's the pros and cons of each approach? Which one should I take? If features: how would I do that?
NB: The actual test is within a Jamfile that's used by the Jamroot, so not directly in the root file.
I would just use your suggestion of variables. They provide a great deal of flexibility. I don't see how a "feature" in this case would help things.

Changing FirefoxProfile() preferences more than once using Selenium/Python

So I am trying to download multiple excel links to different file paths depending on the link using Selenium.
I am able to set up the FirefoxProfile to download all links to a certain single path, but I can't change the path on the fly as I try to download different files into different file paths. Does anyone have a fix for this?
self.fp = webdriver.FirefoxProfile()
self.ft.set_preferences("browser.download.folderList", 2)
self.ft.set_preferences("browser.download.showWhenStarting", 2)
self.ft.set_preferences("browser.download.dir", "C:\SOURCE FILES\BACKHAUL")
self.ft.set_preferences("browser.helperApps.neverAsk.saveToDisk", ("application/vnd.ms-excel))
self.driver = webdriver.Firefox(firefox_profile = self.fp)
This code will set the path I want once. But I want to be able to set it multiple times while running one script.
On Linux and Mac you can set the profile preferences to download to a directory that is a symbolic link to another directory. Then, while running the Selenium driver, you can change the destination of the symlink to the directory you want to download a file in, using the os library in Python.
Code outline (without try/except etc):
import os
from selenium import webdriver
DRIVER = "/usr/local/bin/geckodriver"
fp = webdriver.FirefoxProfile()
fp.set_preference("browser.download.dir", "/your/symlink/name")
fp.set_preference("browser.download.folderList",2)
fp.set_preference("browser.helperApps.neverAsk.saveToDisk", "your/mime/type")
driver = webdriver.Firefox(firefox_profile=fp, executable_path=DRIVER)
# delete symlink before changing to prevent a FileExistsError when changing the link
if os.path.exists("/your/symlink/name"):
os.unlink("/your/symlink/name")
# actually change/create the link
os.symlink("/real/target/directory", "/your/symlink/name")
# download the file
driver.get("https://example.com/file")
# set new target directory
os.unlink("/your/symlink/name")
os.symlink("/different/target/directory", "/your/symlink/name")
driver.get("https://example.com/otherfile")
You can define it only while initializing driver. So to do it with a new path you should driver.quit and start it again.

How to stop Scons adding lib infront of a shared library

I've been playing around with Scons on OSX and I'm trying to make a Shared Library (.dll, .so, .dylib).
It's all worked perfectly except for one thing that is really annoying me, it adds 'lib' in front of the library name. For example, I choose the name WL and it becomes libWL.dylib. I cannot work out why Scons does this and it is driving me mad.
The code I am using is:
# -*- coding: utf-8 -*-
import os
SourceList = ['Window.cpp']
env = Environment(ENV = os.environ)
#Libraries we are using
Targets = 'WL'
libraries = ['SDL2']
#Paths to the libraries and include paths
Paths = ['/usr/local/lib', '/usr/local/include']
Export('SourceList env libraries Paths Targets')
SConscript('src/SConscript', variant_dir='bin', duplicate=0)
and
Import('SourceList env libraries Paths Targets')
SharedLibrary(target = Targets,source = SourceList,LIBS = libraries, LIBPATH=Paths)
I'm not super knowledgeable about how shared libraries work so I don't know if I could just change the name after it's compiled. But I would like it to just not add in letters
In each environment, SCons uses variables to specify the prefixes and suffixes of things like libraries and programs. These variables get initialized, based on the detected platform that it's currently running on...but you can simply overwrite this setting after the call of the Environment() constructor:
env = Environment()
env['SHLIBPREFIX'] = ''
For "darwin"-like systems, SCons calls the standard "posix" initialization first...that's where the default "lib" prefix comes from.
Tip: You can treat an Environment very much like a dictionary (hash map), and set its values how you need them to be. For displaying its current contents you can use the Dump() method:
print env.Dump()
in a SConstruct/SConscript which gives you a full listing of the defined variables.
You can find a list of standard variables in the MAN page ( http://scons.org/doc/production/HTML/scons-man.html ) and the UserGuide ( http://scons.org/doc/production/HTML/scons-user.html ).

scons - how to run something /after/ all targets have been built

I've recently picked up scons to implement a multi-platform build framework for a medium sized C++ project. The build generates a bunch of unit-tests which should be invoked at the end of it all. How does one achieve that sort of thing?
For example in my top level sconstruct, I have
subdirs=['list', 'of', 'my', 'subprojects']
for subdir in subdirs:
SConscript(dirs=subdir, exports='env', name='sconscript',
variant_dir=subdir+os.sep+'build'+os.sep+mode, duplicate=0)
Each of the subdir has its unit-tests, however, since there are dependencies between the dlls and executables built inside them - i want to hold the running of tests until all the subdirs have been built and installed (I mean, using env.Install).
Where should I write the loop to iterate through the built tests and execute them? I tried putting it just after this loop - but since scons doesn't let you control the order of execution - it gets executed well before I want it to.
Please help a scons newbie. :)
thanks,
SCons, like Make, uses a declarative method to solving the build problem. You don't want to tell SCons how to do its job. You want to document all the dependencies and then let SCons solve how it builds everything.
If something is being executed before something else, you need to create and hook up the dependencies.
If you want to create dmy touch files, you can create a custom builder like:
import time
def action(target, source, env):
os.system('echo here I am running other build')
dmy_fh = open('dmy_file','w')
dmy_fh.write( 'Dummy dependency file created at %4d.%02d.%02d %02dh%02dm%02ds\n'%time.localtime()[0:6])
dmy_fh.close()
bldr = Builder(action=action)
env.Append( BUILDERS = {'SubBuild' : bldr } )
env.SubBuild(srcs,tgts)
It is very important to put the timestamp into the dummy file, because scons uses md5 hashes. If you have an empty file, the md5 will always be the same and it may decide to not do subsequent build steps. If you need to generate different tweaks on a basic command, you can use function factories to modify a template. e.g.
def gen_a_echo_cmd_func(echo_str):
def cmd_func(target,source,env):
cmd = 'echo %s'%echo_str
print cmd
os.system(cmd)
return cmd_fun
bldr = Builder(action = gen_a_echo_cmd_func('hi'))
env.Append(BUILDERS = {'Hi': bldr})
env.Hi(srcs,tgts)
bldr = Builder(action = gen_a_echo_cmd_func('bye'))
env.Append(BUILDERS = {'Bye': bldr})
env.Bye(srcs,tgts)
If you have something that you want to automatically inject into the scons build flow ( e.g. something that compresses all your build log files after everything else has run ), see my question here.
The solution should be as simple as this.
Make the result of the Test builders depend on the result of the Install builder
In pseudo:
test = Test(dlls)
result = Install(dlls)
Depends(test,result)
The best way would be if the Test builder actually worked out the dll dependencies for you, but there may be all kinds of reasons it doesn't do that.
In terms of dependencies, what you want is for all the test actions to depend on all the program-built actions. A way of doing this is to create and export a dummy-target to all the subdirectories' sconscript files, and in the sconscript files, make the dummy-target Depends on the main targets, and have the test targets Depends on the dummy-target.
I'm having a bit of trouble figuring out how to set up the dummy target, but this basically works:
(in top-level SConstruct)
dummy = env.Command('.all_built', 'SConstruct', 'echo Targets built. > $TARGET')
Export('dummy')
(in each sub-directory's SConscript)
Import('dummy')
for target in target_list:
Depends(dummy, targe)
for test in test_list:
Depends(test, dummy)
I'm sure further refinements are possible, but maybe this'll get you started.
EDIT: also worth pointing out this page on the subject.
Just have each SConscript return a value on which you will build dependencies.
SConscript file:
test = debug_environment.Program('myTest', src_files)
Return('test')
SConstruct file:
dep1 = SConscript([...])
dep2 = SConscript([...])
Depends(dep1, dep2)
Now dep1 build will complete after dep2 build has completed.