I have a project and there is a ./tests directory at its root containing several hundred MB of data that is used by the tests of several libraries:
./tests
./src/lib1/dune
./src/lib1/tests/dune
./src/lib1/tests/tests.ml
./src/lib2/dune
./src/lib2/tests/dune
./src/lib2/tests/tests.ml
...
I also defined tests that use the data in ./tests for each library like this:
(rule
(alias runtest)
(action (run ./tests/tests.exe)))
I now have to somehow communicate the location of the test data to each of my tests.exe. What is the most elegant way of doing this using dune?
It seems that dune copies my test data into _build which is unnecessary because the data never changes and it doesn't make sense to waste several hundred MB of space that way. From the documentation it seems that %{project_root} would contain the path to my source files but unfortunately, the variable evaluates to . which is useless for the tests which are run after a cd _build/default/src/libX and thus . does not point to the project root anymore. So is there a dune-way to specify the path to the original source directory without ugly hacks?
Right now, I'm using an environment variable containing the full path before I run dune runtest but is there a more integrated way?
I have not tried it myself but it sounds like the data_only_dirs stanza is what you are looking for: https://dune.readthedocs.io/en/stable/dune-files.html#data-only-dirs-since-1-6
Related
In an iOS c++/Qt application, I need to ship a few files and to keep them in their directory structure.
For the Android version, we bundle a zip which we unzip on the target before creating the QApplication.
On iOS, it seems that CMake is not capable of bundling files in a tree:
https://cmake.org/cmake/help/latest/prop_tgt/RESOURCE.html#prop_tgt:RESOURCE
https://cmake.org/cmake/help/latest/prop_sf/MACOSX_PACKAGE_LOCATION.html
I am not sure if this is a limitation of cmake or if this is a global limitation on iOS.
From the docs about iOS bundles:
It uses a relatively flat structure with few extraneous directories in an effort to save disk space and simplify access to the files.
What would be the preferred approach?
Is there a solution to ship the files from CMake directly?
If not, how can I achieve this so that they are available before the QApplication is created?
The xcode command
Thanks to #Cy-4AH, I added the folder in Xcode and could get the command to do this:
CpResource _PATH_TO_DIRECTORY_ _APP_BUNDLE_DIRECTORY_/_RESOURCE_DIR_NAME_
cd /Users/denis/opt/qfield/ios/QField
export PATH="....."
builtin-copy -exclude .DS_Store -exclude CVS -exclude .svn -exclude .git -exclude .hg -strip-debug-symbols -strip-tool /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/strip -resolve-src-symlinks _PATH_TO_DIRECTORY_ _APP_BUNDLE_DIRECTORY_
But how can I create this from cmake? builtin-copyis an xcode command.
Simple system copy command
From an old (2008) discussion, we could use simple cp commands.
This works up to signing, but then I get an error unsealed contents present in the bundle root.
From this answer, it seems related that I cannot simply add folders in the resource directory. From the docs anatomy of framework bundles: Nonlocalized resources reside at the top level of the Resources directory
(Disclaimer: I'm not a CMake user, and there may be a more CMake-ey way to do this)
If you can set up post-build action, the following terminal script can efficiently sync files into your bundle from another location. I use it in my game engine because it only copies updated or new files upon subsequent builds, and preserves directory structure:
mkdir -p PATHTO/ORIGINFOLDERNAME
mkdir -p PATHTOBUILDFOLDER/PROJECTNAME.app/Contents/Resources/DESTINATIONFOLDERNAME
rsync -avu --delete --exclude=".*" PATHTO/ORIGINFOLDERNAME/ PATHTOBUILDFOLDER/PROJECTNAME.app/Contents/Resources/DESTINATIONFOLDERNAME
The mkdir commands are only to ensure that the folders are generated, if they were deleted.
So apparently the CMake method also works for directories.
target_sources(${QT_IOS_TARGET} PRIVATE ${_resource})
set_source_files_properties(${_resource} PROPERTIES MACOSX_PACKAGE_LOCATION Resources)
It will just be added at the root directory of the bundle and not within the Resources.
If the embedded file is not too big, you might consider :
in your source tree, generating a C++ file embedding that file as a constant array. For example, if your file contains just hello, world with a new line, you could have something like
/// file contents.cc
const char file_contents[] = "hello, world\n";
and at the beginning of your program (perhaps in your main function, before your QApplication) call a C++ function which writes such a file (perhaps in /tmp/).
in your build automation (e.g your Makefile or your qmake things), have something which generates the C++ contents.cc file from the genuine source
This is with a POSIX/Linux point of view, adapt my answer to your iOS.
Context:
When I want to build a projet, there are usually CMakeCache.txt and CMakeVars.txt files that are created at a first stage, e.g. using cmake-gui > configure.
Questions:
What are the differences between these files and what are they used for?
If I have tweaked a lot of options within cmake-gui, which file should I safely backup in an other location (for copying it back later on in order to not spend much time figuring out which options I had checked...) before removing the build/ directory contents (this directory normally includes these two files) if I want to start from a clean build/ directory (which is sometimes needed) ?
Having used CMake, I've become used to out-of-source builds, which are encouraged with CMake. How can out-of-source builds be done with Cargo?
Using in-source-builds again feels like a step backwards:
Development tools need to be configured to ignore paths. Sometimes multiple plugins and development tools - especially using VIM or Emacs!
Some tools can't be configured to easily hide build files. While dotfiles are typically hidden, they will still show Cargo.lock and target/, worse still, recursively exposing their contents.
Deleting un-tracked files to remove everything outside of version control, typically to cleanup editor temp files or some test output, can backfire if you forgot to add a new file to version control and don't manually check the file list properly before deleting them.
Dependencies are downloaded into your source code path, sometimes adding *.rs files in the target directory as part of building indirect deps, so operating on all *.rs files may accidentally pickup other files which aren't in a hidden directory, so might not be ignored even after development tools have been configured.
While it's possible to work around all these issues, I'd rather just have an external build path and keep the source directory pristine.
You can specify the directory of the target/ folder either via configuration file (key build.target-dir) or environment variable (CARGO_TARGET_DIR). Here is an example using a configuration file:
Suppose you want to have a directory ~/work/ in which you want to save the Cargo project (~/work/foo/) and next to it the target directory (~/work/my-target/).
$ cd ~/work
$ cargo new --bin foo
$ mkdir .cargo
$ $EDITOR .cargo/config
Then insert the following into the configuration file:
[build]
target-dir = "./my-target"
If you then build in your normal Cargo project directory:
$ cd foo
$ cargo build
You will notice that there is no target/ dir, but everything is in ~/work/my-target/.
However, the Cargo.lock is still saved inside the Cargo project directory, but that kinda makes sense. For executables, you should check the Cargo.lock file into your git! For libraries, you shouldn't. I guess having to ignore one file is better than having to ignore an entire folder.
Lastly, there are a few caveats to changing the target-dir, which are listed in the PR which introduced the feature.
While useful manually setting this up isn't all that convenient, I wanted to be able to build multiple crates within a source tree, having all of them out-of-source, something that ../target-dir configuration option wouldn't achieve.
Helper utility for convenient out-of-source builds
Using the environment variable I've written a small utility to wrap cargo, so it automatically builds out-of-source, supporting crates both at the top-level, on in a subdirectory of the source tree.
Thanks to Lukas for pointing out CARGO_TARGET_DIR and target-dir configuration option.
What I really wanted was a dynamic CARGO_TARGET_DIR that changes relative to where I am.
This bash alias puts all builds in a mirrored directory structure, e.g. instead of putting target into ~/mydir/myproj it puts in into ~/rustbuild/mydir/myproj
alias cargo='CARGO_TARGET_DIR=$(echo $PWD | sed "s|$HOME|$HOME/rustbuild|g") cargo'
You could also make your rustbuild directory hidden.
my autotools project has a couple of unit-tests.
one of these tests (filereader) needs to read a file (data/test1.bin)
Here's my filesystem layout:
- libfoo/tests/filereader.c
- libfoo/tests/data/test1.bin
and my libfoo/tests/Makefile.am:
AUTOMAKE_OPTIONS = foreign
AM_CPPFLAGS = -I$(top_srcdir)/foo
LDADD = $(top_builddir)/src/libfoo.la
EXTRA_DIST = data/file1.bin
TESTS = filereader
check_PROGRAMS= filereader
filereader_SOURCES = filereader.c
this works great, as long as i do in-tree builds.
However, when running the test-suite out-of-tree (e.g. make distcheck), the filereader test cannot find the input file anymore.
This is obviously because only the source tree contains the input file, but not the build tree.
i wonder what is the canonical way to fix this problem?
compile the directory of the test-file into the unittest (AM_CPPFLAGS+=-DSRCDIR=$(srcdir))
pass the qualified input file as a cmdline argument to the test? (e.g. $(builddir)/filereader $(srcdir)/data/file1.bin)
copy the input file from the source tree to the build tree? (cp $(srcdir)/data/file1.bin $(builddir)/data/file1.bin? how would a proper make-rule look like??)
Canonically, the solution would be to define the path to your file into the unittest, so the first option you laid out. The second one is also possible but it requires using an in-between driver script.
I would suggest avoiding the third one, but if you do want to go down that route, use $(LN_S) rather than cp; this way you reduce the I/O load of the test.
There is a way to do this with autoconf. From the netcdf-c configure.ac:
##
# Some files need to exist in build directories
# that do not correspond to their source directory, or
# the test program makes an assumption about where files
# live. AC_CONFIG_LINKS provides a mechanism to link/copy files
# if an out-of-source build is happening.
##
AC_CONFIG_LINKS([nc_test4/ref_hdf5_compat1.nc:nc_test4/ref_hdf5_compat1.nc])
AC_CONFIG_LINKS([nc_test4/ref_hdf5_compat2.nc:nc_test4/ref_hdf5_compat2.nc])
AC_CONFIG_LINKS([nc_test4/ref_hdf5_compat3.nc:nc_test4/ref_hdf5_compat3.nc])
AC_CONFIG_LINKS([nc_test4/ref_chunked.hdf4:nc_test4/ref_chunked.hdf4])
AC_CONFIG_LINKS([nc_test4/ref_contiguous.hdf4:nc_test4/ref_contiguous.hdf4])
So I am running into an issue when I go to build my projects using tfs build controller using the Output location "AsConfigred" it will not detect my unit tests. Let me give a little info on my setup.
TFS 2013 Update 2, Default Process Template
Here is a few screenshots that can hopefully help fill in what I can't in typing. I am copying my build out to a file share on our network so that we can use other utilities use the output. I don't want to use "PerProject" or "SingleFolder" because they mess up the file structure we have configured (These both will run the tests). So i have the files copy to folder names "SingleOutputFolder" which is a child of the DropLocation. I would like to be able to run from the drop folder or run from the bin folder for each of my tests (I don't care which). However it doesn't seem to detect/run ANY of the tests. Any help would be greatly appreciated. Please let me know if you need any additional information.
I have tried using ***test*.dll, Install\SingleFolderOutput**.test.dll, and $(TF_BUILD_DROPLOCATION)\Install\SingleFolderOutput*test*.dll
But I am not sure what variables are available and understand where the scope of its execution is.
Given that you're using Build Output location set to AsConfigured you have to change the default values of the Test sources spec setting to allow build to find the test libraries in the bin folders. Here's an example.
If the full path to the unit test libraries is:
E:\Builds\7\<TFS Team Project>\<Build Definition>\src\<Unit Test Project>\bin\Release\*test*.dll
use
..\src\*UnitTest*\bin\*\*test*.dll;
This question was asked on MSDN forums here.
MSDN Forums Suggested Workaround
The suggested workaround in the accepted answer (as of 8 a.m. on June 20) is to specify the full path to the test projects' binary folders: For example:
C:\Builds\{agentId}\{teamProjectName}\{buildDefinitionName}\src\{solutionName}\{testProjectName}\bin*\Debug\*test*.dll*
which really should have been shown as
{agentWorkingFolder}\src\{relativePathToTestProjectBinariesFolder}\*test*.dll
However this approach is very brittle, for the following reasons:
Any new test projects you add to the solution will not be executed until you add them to the build definition's list of test sources:
It will break under any of the following circumstances:
the build definition is renamed
the working folder in build agent properties is modified
you have multiple build agents, and a different agent than the one you specified in {id} runs the build
Improved Workaround
My workaround mitigates the issues listed in #2 (can't do anything about #1).
In the path specified above, replace the initial part:
{agentWorkingFolder}
with
..
so you have
..\src\{relativePathToTestProjectBinariesFolder}\*test*.dll
This works because the internal working directory is apparently the \binaries\ folder that is a sibling of the \src\ folder. Navigating up to the parent folder (whatever it is named, we don't care) and back in to \src\ before specifying the path to the test projects binaries does the trick.
Note: If you have multiple test projects, you add additional entries, separated with semicolons:
..\src\{relativePathToTestProjectONEBinariesFolder}\*test*.dll;..\src\{relativePathToTestProjectTWOBinariesFolder}\*test*.dll;..\src\{relativePathToTestProjectTHREEBinariesFolder}\*test*.dll;
What I ended up doing was adding a post build event to copy all of the test.dll into the staging location folder in the specific build that is basically equivalent to where it would go on a SingleFolder build and do that on each test project.
if "$(TeamBuildOutDir)" == "" (
echo "Building Interactively not in TFS"
) else (
echo "Building in TFS"
xcopy "$(TargetDir)*.*" "$(TeamBuildBinaries)\" /Y /E /S
)
MSBUILD parameter in the build def that told it to basically drop in the folder that TFS looks for them.
/p:TeamBuildBinaries="$(TF_BUILD_BINARIESDIRECTORY)"
Kept the default Test assembly file specification:
**\*test*.dll
View this link for the information on the variable that I used and what relative path it exists at.
Another solution is to do the reverse.
Leave all of the files in the root so that all of the built in functionality works. There is more than just test execution in there. What about static code analysis, impact analysis..among others. You would have to do something custom for them all.
Instead use a pre-drop powershell script to create your Install arrangement from the root files.
If it is an application then you can use the _ApplicationFolder Nuget package to create an _PublishApplications folder same as you get for web applications.