I have a Haskell project that aims to create some C++ bindings. I've written the C wrappers and compiled them into a stand-alone statically linked library.
I'd like to write the Haskell bindings to link statically to the C wrappers so that I don't have to distribute the C wrappers separately but I can't seem to get it working and would appreciate some help.
I specify the C library as an extra library but my cabal build step doesn't seem to add it to compile command.
I've created a small project to illustrate this (http://github.com/deech/CPlusPlusBindings).
It contains a small C++ class (https://github.com/deech/CPlusPlusBindings/tree/master/cpp-src), the C wrapper (https://github.com/deech/CPlusPlusBindings/tree/master/c-src), a working C test routine (https://github.com/deech/CPlusPlusBindings/tree/master/c-test) and the Haskell file (https://github.com/deech/CPlusPlusBindings/blob/master/src/BindingTest.chs).
The C library is added in Setup.hs not in the Cabal file because that's how I have it my real project which builds the C library using "make" through Cabal just before the build stepf. I have verified that at the build step the extraLibs part of BuildInfo contains the library name and extraLibDirs contains the right directory.
The output of my cabal build is:
creating dist/setup
./dist/setup/setup build --verbose=2
creating dist/build
creating dist/build/autogen
Building CPlusPlusBinding-0.1.0.0...
Preprocessing library CPlusPlusBinding-0.1.0.0...
Building library...
creating dist/build
/usr/local/bin/ghc --make -fbuilding-cabal-package -O -odir dist/build -hidir dist/build -stubdir dist/build -i -idist/build -isrc -idist/build/autogen -Idist/build/autogen -Idist/build -I/home/deech/Old/Haskell/CPlusPlusBinding/c-src -I/home/deech/Old/Haskell/CPlusPlusBinding/cpp-includes -optP-include -optPdist/build/autogen/cabal_macros.h -package-name CPlusPlusBinding-0.1.0.0 -hide-all-packages -package-db dist/package.conf.inplace -package-id base-4.6.0.1-8aa5d403c45ea59dcd2c39f123e27d57 -XHaskell98 -XForeignFunctionInterface BindingTest
Linking...
/usr/bin/ar -r dist/build/libHSCPlusPlusBinding-0.1.0.0.a dist/build/BindingTest.o
/usr/bin/ar: creating dist/build/libHSCPlusPlusBinding-0.1.0.0.a
/usr/bin/ld -x --hash-size=31 --reduce-memory-overheads -r -o dist/build/HSCPlusPlusBinding-0.1.0.0.o dist/build/BindingTest.o
In-place registering CPlusPlusBinding-0.1.0.0...
/usr/local/bin/ghc-pkg update - --global --user --package-db=dist/package.conf.inplace
Unfortunately neither the compilation nor the linking step uses the C library. There are no other warnings or errors.
To solve this problem I had to:
re-link the Haskell library with the C bindings' object files and
use the ghc-options tag in my Cabal file to make sure they linked in the right order.
All the changes are in the test project (http://github.com/deech/CPlusPlusBindings).
Below the process of creating a new archive that includes both the C and Haskell objects is explained in detail and it is not simple. The complexity occurs because there is no way (as of Cabal 1.16.0.2) to hook into the linker part of the build process.
Setting the flags in the Cabal file is trivial so it is not described here.
Relinking The Haskell Library
Set the build type to custom by adding:
build-type: custom
to the cabal file.
Insert customized build logic by replacing the main method in Setup.hs with:
main = defaultMainWithHooks simpleUserHooks {
buildHook = myBuildHook,
...
}
This tells the build process that instead of going with the default build process defined in simpleUserHooks it should use the myBuildHook function which is defined below. Similarly the clean up process is overridden with the custom function myCleanHook.
Define the build hook. This build hook will run make on the command line to build the C++, and C portions and then use the C object files when creating linking the Haskell bindings.
We start off myBuildHook:
myBuildHook pkg_descr local_bld_info user_hooks bld_flags = do
by first running make with no arguments:
rawSystemExit normal "make" []
Then add the locations of the header files and library directories and the library itself to the PackageDescription record and update the LocalBuildInfo with the new package description:
let new_pkg_descr = (addLib . addLibDirs . addIncludeDirs $ pkg_descr)
new_local_bld_info = local_bld_info {localPkgDescr = new_pkg_descr}
Before the buildHook fired the configureHook stored the order of compilation in the compBuildOrder (component build order) key of the LocalBuildInfo record. We need to isolate the building of the library so we separate the library building and executable building parts of the build process.
The build order is just a list and we know the build component is a library if it's just a plain CLibName type constructor so we isolate those elements from the list and update the LocalBuildInfo record with only them:
let (libs, nonlibs) = partition
(\c -> case c of
CLibName -> True
_ -> False)
(compBuildOrder new_local_bld_info)
lib_lbi = new_local_bld_info {compBuildOrder = libs}
Now we run the default build hook with the updated records:
buildHook simpleUserHooks new_pkg_descr lib_lbi user_hooks bld_flags
Once it's done building an archive has been created but we have to re-create it to include the C objects generated by the make command in step 1. So we grab some settings and a list of the C object file paths:
let verbosity = fromFlag (buildVerbosity bld_flags)
info verbosity "Relinking archive ..."
let pref = buildDir local_bld_info
verbosity = fromFlag (buildVerbosity bld_flags)
cobjs <- getLibDirContents >>= return . map (\f -> combine clibdir f)
. filter (\f -> takeExtension f == ".o")
And then hand it off to withComponentsLBI which acts on each component of the build. In this case since we're only dealing with the library part there is only one component. Cabal provides getHaskellObjects for getting a list of the Haskell object files and createArLibArchive for creating an archive so we can re-run the linker:
withComponentsLBI pkg_descr local_bld_info $ \comp clbi ->
case comp of
(CLib lib) -> do
hobjs <- getHaskellObjects lib local_bld_info pref objExtension True
let staticObjectFiles = hobjs ++ cobjs
(arProg, _) <- requireProgram verbosity arProgram (withPrograms local_bld_info)
let pkgid = packageId pkg_descr
vanillaLibFilePath = pref </> mkLibName pkgid
Ar.createArLibArchive verbosity arProg vanillaLibFilePath staticObjectFiles
_ -> return ()
The default buildHook which was run in Step 4 created a temporary package database file named "package.conf.inplace" which holds the description of the library that was built so that executable can link against it without the library needing to be installed to the default system package file. Unfortunately every buildHook run blanks it out so we need to hold on to a temporary copy:
let distPref = fromFlag (buildDistPref bld_flags)
dbFile = distPref </> "package.conf.inplace"
(tempFilePath, tempFileHandle) <- openTempFile distPref "package.conf"
hClose tempFileHandle
copyFile dbFile tempFilePath
Now we store a path to that copy into the LocalBuildInfo structure along with the executable parts of the build process which were filtered out in Step 3.
let exe_lbi = new_local_bld_info {
withPackageDB = withPackageDB
new_local_bld_info ++
[SpecificPackageDB tempFilePath],
compBuildOrder = nonlibs
}
and store the path again in the extraTmpFiles part of the PackageDescription so it can be removed by the default clean up hook.
exe_pkg_descr = new_pkg_descr {extraTmpFiles = extraTmpFiles new_pkg_descr ++ [tempFilePath]}
Now we finally run the default buildHook again with the updated records (which now know about the new archive) on just the executable components:
buildHook simpleUserHooks exe_pkg_descr exe_lbi user_hooks bld_flags
Related
I have an OCaml project that is currently built using OCamlMake. I am not happy with the current build system since it leaves all build artefacts in the same directory as source files, plus it also requires to manually specify order of dependencies between modules. I would like to switch to a build system that does not suffer from these issues. I decided to give Oasis a try but have run into issues.
The problems arise from the fact that the project is built in a very specific way. It supports several different database backends (PostgreSQL, MySQL, SQLite). Currently, to compile a database backend user must install extra libraries required by that backend and enable it by setting an environment variable. This is how it looks in the Makefile:
ifdef MYSQL_LIBDIR
DB_CODE += mysql_database.ml
DB_AUXLIBS += $(MYSQL_LIBDIR)
DB_LIBS += mysql
endif
Notice that this also adds extra module to the list of compiled modules. The important bit is that there is no dependency (in a sense of module import) between any module reachable from the applications entry point and database backend module. What happens rather is that each database backend module contains top-level code that runs when a module is initiated and registers itself, using side-effects, with the main application.
I cannot get this setup to work with Oasis. I declared each of the database backend modules as a separate library that can be enabled for compilation with a flag:
Library mysql-backend
Path : .
Build $: flag(mysql)
Install : false
BuildTools : ocamlbuild
BuildDepends : main, mysql
Modules : Mysql_backend
However, I cannot figure out a way to tell Oasis to link the optional modules into the executable. I tried to figure out a way of doing this by modifying myocamlbuild.ml file but failed. Can I achieve this with the rule function described here?
If what I describe cannot be achieved with ocamlbuild, is there any ither tool that would do the job and avoid problems of OCamlMake?
Well, I guess that answers it: https://github.com/links-lang/links/pull/77 :)
I saw the question and started working on it before I noticed Drup's answer above. Below is a self-contained ocamlbuild solution that is essentially the same as Drup's.
open Ocamlbuild_plugin
let enable_plugin () =
let plugins = try string_list_of_file "plugin.config" with _ -> [] in
dep ["ocaml"; "link_with_plugin"; "byte"]
(List.map (fun p -> p^".cmo") plugins);
dep ["ocaml"; "link_with_plugin"; "native"]
(List.map (fun p -> p^".cmx") plugins);
()
let () = dispatch begin
function
| Before_rules -> enable_plugin ()
| _ -> ()
end
Using the tag link_with_plugin on an ocamlbuild target will make it depend on any module whose path (without extension) is listed in the file plugin.config. For example if you have plugins pluga.ml, plugb.ml and a file main.ml, then writing pluga plugb in plugin.config and having <main.{cmo,cmx}>: link_with_plugin will link both plugin modules in the main executable.
Unfortunately, this is beyond oasis capabilities. This limitation has nothing to do with ocamlbuild, it just because oasis author tried to keep it simple, and didn't provide optional dependencies as a feature.
As always, an extra level of indirection may solve your problem. What you need, is a configuration script (configure) that will generate _oasis file for you, depending on parameters, provided by a user.
For example, in our project we have a similar setup, i.e., multiple different backends, that might be chosen by a user during the configuration phase, with --{enable,disable}-<feature>. We achieved this by writing our own ./configure script that generate _oasis file, depending on configuration. The configuration script just concatenates the resulting _oasis files from pieces, described in oasis folder.
An alternative solution would be to use m4 or just cpp, and have an _oasis.in file, that is preprocessed.
I'm trying to use ocamlbuild to automatically invoke piqi to generate ocaml modules from protobuf definitions. This is a 2 step process where I have to go from a %.protobuf file to a %.proto.piqi and finally to %_piqi.ml.
My rule for the first step looks as follows:
rule "piqi: .proto -> .piqi"
~dep:"%.proto"
~prod:"%.proto.piqi"
begin fun env build ->
Cmd (S [ A piqi ; A "of-proto"
; A "-I"; P protobuf_include
; P (env "%.proto") ]
)
end;
But this doesn't work because the %.proto.piqi target is actually dependent on all the "*.proto" files in my source directory because the individual .proto files import each other in the source. However, I'm not sure how to express this dependency between them in ocamlbuild. It would be enough if all of the proto files where copied over to _build rather than just the one in ~dep
Calling build on a file from inside a rule action will register it as a dynamic dependency of the target. You can just loop over all *.proto files that you know may be accessed by the current one (either globbing the source directory or, more cleverly if that gives any benefit, parsing the include statements) and build them.
(Another way to think of this solution is to remark that, if you wanted to have some foo.proto file generated by some preprocessing step from a foo.proto.pp, then you would need any compilation needing foo.proto to actually call build on it.)
See the dynamic dependencies section of the new manual draft.
P.S.: I don't know anything about protobuf but from protoc --help it looks like protoc --include-imports --descriptor_set_out FILE may give you the list of a .proto dependencies in some format. Parse that and call build on all of them, and you've got a nice and robust rule.
Until now, I have been using Poly/ML for several small projects where all source code files are all in the same directory. To build these projects, all I had to do was run the following command in the REPL:
> PolyML.make "Main";
But now I have a project whose scale makes it impractical to put all source code files in the same directory. To build these projects in the REPL, I need to run the following commands:
> PolyML.make "foo/Foo";
> PolyML.make "bar/Bar";
> PolyML.make "qux/Qux";
> PolyML.make "Main";
which is not terribly practical as the number of subsystems grows.
Is there any way to automate the process of building projects with nested directory structures in Poly/ML?
P.D.: I have had a look at both SML/NJ's Compilation Manager and MLton's ML Basis system. While unquestionably powerful, these are too complicated for my needs.
Put a file called ml_bind.ML in each of the sub-directories and have those files build the component for that directory.
PolyML.make expects the name of the source file to match the name of the component (structure, signature or functor). So if it is looking for a structure called "Foo" it will expect the source for "Foo" in a file called "Foo", "Foo.ML" or "Foo.sml". If instead it finds a directory called "Foo" it recursively enters the "Foo" directory and uses the "ml_bind.ML" file as the guide to build the "Foo" structure. Typically, "Foo/ml_bind.ML" will look like
structure Foo = FooFunctor(structure A = FooA and B = FooB);
with files "Foo/FooFunctor.ML", "Foo/FooA.ML" and "Foo/FooB.ML" containing the source for "FooFunctor", "FooA" and "FooB" respectively.
You can find examples of this in the code for the Poly/ML compiler which comes as part of the Poly/ML source code distribution.
You could have a build.sml file listing and use-ing all project files:
use "bar/bar.sml";
use "foo/foo.sml";
use "main.sml";
Or, a little bit more concise:
app use [
"foo/foo.sml",
"bar/bar.sml",
"main.sml"
]
Where app is the standard List.app.
Then you can build just this one file:
$ polyc -o main main.sml
$ # or
$ poly
> PolyML.make "build.sml"
I'd like to use Boost.Test for Test Driven Development.
I asked scons to create two executables, the main one, and the test one.
All my main project files are in ./src/, and all my test dedicated files are in ./test/
The problem is:
the main project object files are put in ./build/src/
the test project object files are put in ./build/test/
and in such a configuration my executable Test cannot link since all the main project object files (of the classes on which I perform my tests) are not in the same directory.
Do you have an idea how I could tweak my scons file so as the linking of the executable Test can use the object files in ./src./ ?
Below is my main.scons file:
import os
env=Environment(CPPPATH=['/usr/local/boost/boost_1_52_0/boost/','./src/'],
CPPDEFINES=[],
LIBPATH=['/usr/local/boost/boost_1_52_0/boost/libs/','.'],
LIBS=['boost_regex'],
CXXFLAGS="-std=c++0x")
env['ENV']['TERM'] = os.environ['TERM']
env.Program('Main', Glob('src/*.cpp'))
#
testEnv = env.Clone()
testEnv['CPPPATH'].append('./test/')
testEnv['LIBS'].append('boost_unit_test_framework')
testEnv.Program('Test', Glob('test/*.cpp'))
While the "duplicate object lists" approach is fine for simple projects, you may run into limitations in which your test code doesn't need to link against the entire object space of your main program. For example, to stub out a database layer that's not the focus of a particular unit test.
As an alternative, you can create (static) libraries of common code that you link against your primary executable and your test framework.
common_sources = ['src/foo.cpp', 'src/bar.cpp'] # or use Glob and then filter
env.Library("common", common_sources)
program_sources = ['src/main.cpp']
env.Program("my_program", program_sources, LIBS=['common'])
...
testEnv['LIBPATH'] = ['.'] # or wherever you build the library
testEnv.Program("unit_test", test_sources, LIBS=['common'])
This also avoids the duplicate main() problem that you mention because only the program_sources and test_sources lists should contain the appropriate file with the main routine.
I have continued searching, and found This post on the web which intrigued me, using the scons env.Object. Indeed this object contains the list of all target object files.
And with slight modifications I have the scons file that does what I wanted (though now I have a problem of dupplicated main function but that's another problem):
import os
env=Environment(CPPPATH=['/usr/local/boost/boost_1_52_0/boost/','./src/'],
CPPDEFINES=[],
LIBPATH=['/usr/local/boost/boost_1_52_0/boost/libs/','.'],
LIBS=['boost_regex'],
CXXFLAGS="-std=c++0x")
env['ENV']['TERM'] = os.environ['TERM']
# here I keep track of the main project object files
mainObjectFiles = env.Object( Glob('src/*.cpp'))
env.Program('PostgresCpp', mainObjectFiles)
#
testEnv = env.Clone()
testEnv['CPPPATH'].append('./test/')
testEnv['LIBS'].append('boost_unit_test_framework')
# here I append all needed object files
testObjectFiles = Glob('test/*.cpp')
testedObjectFiles = Glob('src/*.cpp')
allObjectFilesExceptMain = [x for x in mainObjectFiles if x != 'src/main.o']
allObjectFilesExceptMain.append(testObjectFiles)
testEnv.Program('Test',allObjectFiles)
I've recently picked up scons to implement a multi-platform build framework for a medium sized C++ project. The build generates a bunch of unit-tests which should be invoked at the end of it all. How does one achieve that sort of thing?
For example in my top level sconstruct, I have
subdirs=['list', 'of', 'my', 'subprojects']
for subdir in subdirs:
SConscript(dirs=subdir, exports='env', name='sconscript',
variant_dir=subdir+os.sep+'build'+os.sep+mode, duplicate=0)
Each of the subdir has its unit-tests, however, since there are dependencies between the dlls and executables built inside them - i want to hold the running of tests until all the subdirs have been built and installed (I mean, using env.Install).
Where should I write the loop to iterate through the built tests and execute them? I tried putting it just after this loop - but since scons doesn't let you control the order of execution - it gets executed well before I want it to.
Please help a scons newbie. :)
thanks,
SCons, like Make, uses a declarative method to solving the build problem. You don't want to tell SCons how to do its job. You want to document all the dependencies and then let SCons solve how it builds everything.
If something is being executed before something else, you need to create and hook up the dependencies.
If you want to create dmy touch files, you can create a custom builder like:
import time
def action(target, source, env):
os.system('echo here I am running other build')
dmy_fh = open('dmy_file','w')
dmy_fh.write( 'Dummy dependency file created at %4d.%02d.%02d %02dh%02dm%02ds\n'%time.localtime()[0:6])
dmy_fh.close()
bldr = Builder(action=action)
env.Append( BUILDERS = {'SubBuild' : bldr } )
env.SubBuild(srcs,tgts)
It is very important to put the timestamp into the dummy file, because scons uses md5 hashes. If you have an empty file, the md5 will always be the same and it may decide to not do subsequent build steps. If you need to generate different tweaks on a basic command, you can use function factories to modify a template. e.g.
def gen_a_echo_cmd_func(echo_str):
def cmd_func(target,source,env):
cmd = 'echo %s'%echo_str
print cmd
os.system(cmd)
return cmd_fun
bldr = Builder(action = gen_a_echo_cmd_func('hi'))
env.Append(BUILDERS = {'Hi': bldr})
env.Hi(srcs,tgts)
bldr = Builder(action = gen_a_echo_cmd_func('bye'))
env.Append(BUILDERS = {'Bye': bldr})
env.Bye(srcs,tgts)
If you have something that you want to automatically inject into the scons build flow ( e.g. something that compresses all your build log files after everything else has run ), see my question here.
The solution should be as simple as this.
Make the result of the Test builders depend on the result of the Install builder
In pseudo:
test = Test(dlls)
result = Install(dlls)
Depends(test,result)
The best way would be if the Test builder actually worked out the dll dependencies for you, but there may be all kinds of reasons it doesn't do that.
In terms of dependencies, what you want is for all the test actions to depend on all the program-built actions. A way of doing this is to create and export a dummy-target to all the subdirectories' sconscript files, and in the sconscript files, make the dummy-target Depends on the main targets, and have the test targets Depends on the dummy-target.
I'm having a bit of trouble figuring out how to set up the dummy target, but this basically works:
(in top-level SConstruct)
dummy = env.Command('.all_built', 'SConstruct', 'echo Targets built. > $TARGET')
Export('dummy')
(in each sub-directory's SConscript)
Import('dummy')
for target in target_list:
Depends(dummy, targe)
for test in test_list:
Depends(test, dummy)
I'm sure further refinements are possible, but maybe this'll get you started.
EDIT: also worth pointing out this page on the subject.
Just have each SConscript return a value on which you will build dependencies.
SConscript file:
test = debug_environment.Program('myTest', src_files)
Return('test')
SConstruct file:
dep1 = SConscript([...])
dep2 = SConscript([...])
Depends(dep1, dep2)
Now dep1 build will complete after dep2 build has completed.