Optional dependencies with ocamlbuild - ocaml

I have an OCaml project that is currently built using OCamlMake. I am not happy with the current build system since it leaves all build artefacts in the same directory as source files, plus it also requires to manually specify order of dependencies between modules. I would like to switch to a build system that does not suffer from these issues. I decided to give Oasis a try but have run into issues.
The problems arise from the fact that the project is built in a very specific way. It supports several different database backends (PostgreSQL, MySQL, SQLite). Currently, to compile a database backend user must install extra libraries required by that backend and enable it by setting an environment variable. This is how it looks in the Makefile:
ifdef MYSQL_LIBDIR
DB_CODE += mysql_database.ml
DB_AUXLIBS += $(MYSQL_LIBDIR)
DB_LIBS += mysql
endif
Notice that this also adds extra module to the list of compiled modules. The important bit is that there is no dependency (in a sense of module import) between any module reachable from the applications entry point and database backend module. What happens rather is that each database backend module contains top-level code that runs when a module is initiated and registers itself, using side-effects, with the main application.
I cannot get this setup to work with Oasis. I declared each of the database backend modules as a separate library that can be enabled for compilation with a flag:
Library mysql-backend
Path : .
Build $: flag(mysql)
Install : false
BuildTools : ocamlbuild
BuildDepends : main, mysql
Modules : Mysql_backend
However, I cannot figure out a way to tell Oasis to link the optional modules into the executable. I tried to figure out a way of doing this by modifying myocamlbuild.ml file but failed. Can I achieve this with the rule function described here?
If what I describe cannot be achieved with ocamlbuild, is there any ither tool that would do the job and avoid problems of OCamlMake?

Well, I guess that answers it: https://github.com/links-lang/links/pull/77 :)

I saw the question and started working on it before I noticed Drup's answer above. Below is a self-contained ocamlbuild solution that is essentially the same as Drup's.
open Ocamlbuild_plugin
let enable_plugin () =
let plugins = try string_list_of_file "plugin.config" with _ -> [] in
dep ["ocaml"; "link_with_plugin"; "byte"]
(List.map (fun p -> p^".cmo") plugins);
dep ["ocaml"; "link_with_plugin"; "native"]
(List.map (fun p -> p^".cmx") plugins);
()
let () = dispatch begin
function
| Before_rules -> enable_plugin ()
| _ -> ()
end
Using the tag link_with_plugin on an ocamlbuild target will make it depend on any module whose path (without extension) is listed in the file plugin.config. For example if you have plugins pluga.ml, plugb.ml and a file main.ml, then writing pluga plugb in plugin.config and having <main.{cmo,cmx}>: link_with_plugin will link both plugin modules in the main executable.

Unfortunately, this is beyond oasis capabilities. This limitation has nothing to do with ocamlbuild, it just because oasis author tried to keep it simple, and didn't provide optional dependencies as a feature.
As always, an extra level of indirection may solve your problem. What you need, is a configuration script (configure) that will generate _oasis file for you, depending on parameters, provided by a user.
For example, in our project we have a similar setup, i.e., multiple different backends, that might be chosen by a user during the configuration phase, with --{enable,disable}-<feature>. We achieved this by writing our own ./configure script that generate _oasis file, depending on configuration. The configuration script just concatenates the resulting _oasis files from pieces, described in oasis folder.
An alternative solution would be to use m4 or just cpp, and have an _oasis.in file, that is preprocessed.

Related

Is there a way to make Bazel work with transitive repositories?

I work with a massive codebase distributed across many repositories and using even more third-party dependencies. The goal is to make the build hermetic and I contemplate using Bazel to achieve it. On the one hand, Bazel has git_repository rule to refer to the external repos in the WORKSPACE file. On the other hand, WORKSPACE files are not loaded recursively, so to get to indirect dependencies I need to build all inclusive WORKSPACE file somehow. I wonder if somebody already tackled that problem using Bazel or some other existing tools. Is there a way to expand the WORKSPACE as part of the build? May be WORKSPACE can #include other (generated) files?
WORKSPACE files can load and then call macros, which gives similar functionality to #include.
A common pattern is each project having a macro which calls macros (for dependencies on other projects) and creates *_archive rules (for dependencies directly on files to download) so it builds. For example, protobuf has protobuf_deps to implement this pattern. If you create a repository with protobuf (using git_repository, or http_archive, or any of the other repository rules), then you can load that macro and call it, and you'll automatically get all the transitive dependencies.
For example (from Chromium):
load("#bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")
# This com_google_protobuf repository is required for proto_library rule.
# It provides the protocol compiler binary (i.e., protoc).
http_archive(
name = "com_google_protobuf",
strip_prefix = "protobuf-master",
urls = ["https://github.com/protocolbuffers/protobuf/archive/master.zip"],
)
load("#com_google_protobuf//:protobuf_deps.bzl", "protobuf_deps")
protobuf_deps()
I'm showing http_archive because it's easier to work with, but you can easily change it to git_archive if you want.
Another common pattern which makes this all work is the way protobuf_deps checks native.existing_rule before creating each http_archive. That allows you to instantiate a specific version (or from a specific source, etc) of the dependency directly in your WORKSPACE file to override the one protobuf would otherwise bring in.

Loading OCaml modules not in the current directory

I'm writing a large OCaml project. I wrote a file foo.ml, which works perfectly. In a subdirectory of foo.ml's directory, there is a file bar.ml.
bar.ml references code in foo.ml, so its opening line is:
open Foo
This gives me an error at compile time:
Unbound module Foo.
What can I do to fix this without changing the location of foo.ml?
The easy path is to use one of OCaml build system like ocamlbuild or oasis. Another option would be jbuilder but jbuilder is quite opiniated about file organization and does not allow for the kind of subdirectory structure that you are asking for.
The more explicit path comes with a warning: OCaml build process is complicated with many moving parts that can be hard to deal with.
After this customary warning, when looking for modules, OCaml compiler first looks for module in the current compilation environment, then looks for compiled interface ".cmi" files in the directories specified by the "-I" option flags (plus the current directory and the standard library directory).
Thus in order to compile your bar.ml file, you will need to add the parent directory in the list of included directories with the -I .. option.
After all this, you will discover that during the linking phase, all object files (i.e. .cmo or .cmx) need to be listed in a topological order compatible with the dependency graph of your project.
Consequently, let me repeat my advice: use a proper build system.

waf: nested projects and _cache.py: not supported?

I am converting a project from autotools to waf with the hope that it can be easily compiled in windows as well.
I am using a super project with two children folders that are 2 projects.
One of them is a library, the other, a program, like this:
superproject/wscript
superproject/libraryproject/wscript
superproject/programproject/wscript
It seems that waf has terrible support for subprojects. I have a wscript in each of these directories.
I recurse from superproject into the 2 other projects, but the _cache.py file is shared for both projects. This has the following side effects (issues):
When using the boost tool, I had to use it like this to avoid name collisions:
# In library project
cfg.check_boost('boost_program_options', uselib_store='BOOST_LIBRARYPROJECT')
# In program project
cfg.check_boost('boost_program_options', uselib_store='BOOST_PROGRAMPROJECT')
boost-libs and boost-includes command line options are also lost by default, so I have to set them manually, like this:
cfg.env.LIBPATH_BOOST_PROGRAMPROJECT = cfg.options.boost_libs
...
The _cache.py file is overwritten by the programproject/wscript, loosing all the configuration for the flags.
Questions:
Is there any good way to nest projects and avoid at least issue 2?
Is there any reasonable way to avoid both that doesn't require a script and building projects separately?
Configuration file is not written twice.
My mistake was to do this:
cfg.env = ConfigSet()
I wanted a new and clean ConfigSet but doing that in both projects made the first set of flags to be lost.
Since the environment seems to be shared among all project configurations, is it good style to name the variables with custom names? For example, instead of using:
cfg.check_boost('program_options')
Should I use:
cfg.check_boost('program_options', uselib_store='BOOST_MYPROGRAMPROJECT')
Is this good style or it's usually done in another way?
Can be done in a cleaner way deriving ConfigSets?

Using closure library with jsTestDriver

I'm learning about google closure tools by writing a simple JavaScript game. I'm having trouble figuring out how to set up jsTestDriver so that it works well with closure library.
Specifically: I'd like to use the goog.require mechanism to include any additional JavaScript files rather than have to manually add them all to the config file.
Following meyertee's suggestion I made a simple script to automatically write the dependencies to a config file
#!/bin/bash
cp tests/jsTestDriver.conf.proto tests/jsTestDriver.conf
libs/closure-library/closure/bin/build/closurebuilder.py --root="./libs/closure-library" --root="./js" --namespace="lds" | sed "s#^# - \.\./#" >> tests/jsTestDriver.conf
The tests/jsTestDriver.conf.proto file is a simple template:
test:
- "*.js"
load:
- ../libs/knockout-2.1.0.js
# Crucial, the load key needs to be last, and this comment must be followed by a newline.
It is a very fragile script, but hopefully someone (other than me) will find it useful.
You can do it semi-automatically by letting Closure Compile generate a manifest file, which will output all files in the correct order of dependency. You can then transform that file to relative paths and paste them into the JsTestDriver config file. That's how I do it.
You could even write a script that does this transformation automatically.
This is the relevant compiler argument:
--output_manifest manifest.MF
There are some details on the Closure Compiler's Google Code Wiki
Edit:
There are also some Python scripts to help you calculate dependencies. You can use calcdeps.py or closurebuilder.py to generate a manifest file, which even includes files that haven't been 'required' by your code.
Since JsTestDriver does not following the Closure Library convention of declaring dependencies with goog.provide() and goog.require(), your best option may be meyertee's solution.
However, the Closure Library includes its own testing framework. See:
Test Driven Development with the Closure Framework
Asserts API

How to automate module reloading when unit testing with Erlang?

I'm using Emacs and trying to get my unit testing work flow as automated as possible. I have it set up so it is working but I have to manually compile my module under test or the module containing the tests before the Erlang Shell recognizes my changes.
I have two files mymodule.erl and mymodule_tests.erl. What I would like to be able to do is:
Add test case to mymodule_tests
Save mymodule_tests
Switch to the Erlang Shell
Run tests with one line, like eunit:test(mymodule) or mymodule_tests:test()
Have Erlang reload mymodule and mymodule_tests before actually performing the tests
I have tried writing my own test method but it doesn't work.
-module (mytests).
-export([test/0]).
-import(mymodule).
-import(mymodule_tests).
-import(code).
test() ->
code:purge(mymodule),
code:delete(mymodule),
code:load_file(mymodule),
code:purge(mymodule_tests),
code:delete(mymodule_tests),
code:load_file(mymodule_tests),
mymodule_tests:test().
I have also tried by putting -compile(mymodule). into mymodule_tests to see if I could get mymodule to automatically reload when updating mymodule_tests but to no avail.
I have also googled quite a bit but can't find any relevant information. As I'm new to Erlang I'm thinking that I'm either searching for the wrong terms, e.g. erlang reload module, or that you are not supposed to be able reload other modules when compile another module.
Maybe the Erlang make can help you.
make:all([load]).
Reading from the doc:
This function first looks in the
current working directory for a file
named Emakefile (see below) specifying
the set of modules to compile and the
compile options to use. If no such
file is found, the set of modules to
compile defaults to all modules in the
current working directory.
And regarding the "load" option:
Load mode. Loads all recompiled
modules.
There's also a make:files/1,2 which allows you to specify the list of modules to check.
Have you tried using l(mymodule). to reload the module after it's been compiled?