If I have a bunch of tests for my project, I can run them - after cmakeing and makeing to build, building - with make test.
But what if I only want to run one of my tests? That is, one of the items for which I have a add_test() in the tests CMakeFile.txt ?
tl;dr - Do this:
cd $YOUR_BUILD_DIRECTORY
ctest -R name_of_your_test
Explanation
Here is how you likely got confused:
You were trying to do this with make, since make test runs all the tests for you. This won't work for a single test (although there's a workaround - see #danger89's answer). ctest is the cross-platform way to do it.
You started using ctest, e.g. in your main project directory, and you probably got something like:
*********************************
No test configuration file found!
*********************************
Usage
ctest [options]
which wasn't helpful.
... So you thought "Ok, maybe ctest has a -C switch, like CMake and GNU Make" - and indeed, it has a -C switch! but that didn't do what you expected:
[joeuser:/home/joeuser/src/myproj]$ ctest -C build
Test project /home/joeuser/src/myproj
No tests were found!!!
What you actually need to do:
cd $YOUR_BUILD_DIRECTORY
ctest -R name_of_your_test
(note that -R matches a regular expression.) This should work. Note that you can list the tests to be run rather than actually run them by passing -N to ctest.
Thanks goes to #RTsyvarev for pointing me in the right direction
einpoklum is not fully correct. Actually you can use make test if you like. CMake will generate a Makefile and I found out that it accepts an ARGS variable.
Meaning you are able run specific tests via for example the -R (regex) parameter. Which will look like this when using make test:
make test ARGS="-R '^test_'"
This will run only the testcases files that starts with test_ in their filename.
Note: -R above is just an example, it will accept all the arguments that ctest command will accept as well.
Anyway the make test example above will do exactly the same as running:
ctest -R '^test_'
Related
I would like to pass parameters to our Catch2 tests via ctest when running through bamboo or jenkins so that they product junit test results. So I would like to do something like:
make test ARGS="-r junit -o test_results.xml"
That would forward these on to my test:
unittest -r junit -o test_results.xml
That way when I run make tests it just runs the tests normally which pretty prints results to the console.
I know args can be added in the add_test() command but I'm looking for something more dynamic.
I'm hoping there is a way to do this in modern CMake.
I am looking at the unittest docs and I see that I could put multiple suite blocks in one test file.
However, I would like to have multiple test files and run them all with a single command. I could write a bash script to compile and run each script one after another:
#!/bin/bash
nim c -r test1.nim
nim c -r test2.nim
...
But is there a better way? For example in Python I can automatically discover and run all files of the form test*.py.
Put all your unit tests in a tests directory, running nimble test will run all of them.
I'm using CTest with CMake to run some tests. I use the enable_testing() command which provides me with a default command for make test. All of the tests in my subdirectory are accounted for (by doing an add_test command) and make test works great, except one problem.
There is a certain test, which I've named skip_test, that I do NOT want being run when I do make test. I would like to add a custom target so I can run make skip_test and it will run that test.
I can do this by doing add_custom_target(skip_test ...) and providing CTest with the -R flag and telling it to look for files containing "skip_test" in their name. This also seems to work. My problem now is: how can I get the make test command to ignore skip_test?
If I try commenting out enable_testing and adding my own add_custom_target(test ....), I get "No tests found!!!" now for either make test or make skip_test. I also tried making a Custom CTest file and adding set(CTEST_CUSTOM_TESTS_IGNORE skip_test). This worked so that now make test ignored "skip_test", but now running make skip_test responds with "no tests found!!!".
Any suggestions would be appreciated!
I actually used a different solution. Here is what I did. For the tests that I wanted to exclude, I used the following command when adding them:
"add_test( ..... CONFIGURATIONS ignore_flag)" where ignore_flag is whatever phrase you want. Then, in my CMakeLists.txt, when I define a custom target
add_custom_target( ignore_tests ...)
I give it ctest .... -C ignore_flag
Now, make test WILL skip these tests! make ignore_Tests will run the ignored tests + the un-ignored tests, which I'm okay with.
I'm not sure of a way to do this entirely via CTest, but since you've tagged this question with "googletest", I assume you're using that as your test framework. So, you could perhaps make use of Gtest's ability to disable tests and also to run disabled tests.
By changing the test(s) in question to have a leading DISABLED_ in their name(s), these won't be run by default when you do make test.
You can then add your custom target which will invoke your test executable with the appropriate Gtest flags to run only the disabled tests:
add_custom_target(skip_test
MyTestBinary --gtest_filter=*DISABLED_* --gtest_also_run_disabled_tests VERBATIM)
It's a bit of an abuse of the Gtest functionality - it's really meant to be used to temporarily disable tests while you refactor whatever to get the test passing again. This beats just commenting out the test since it continues to compile it, and it gives a nagging reminder after running the suite that you have disabled tests.
I am maintaining an autoconf package and wanted to integrate automatic testing. I use the Boost Unit Test Framework for my unit tests and was able to sucessfully integrate it into the package.
That is it can be compiled via make check, but is is not run (although I read that make check both compiles and runs the tests). As result, I have to run it manually after building the tests which is cumbersome.
Makefile.am in the test folder looks like this:
check_PROGRAMS = prog_test
prog_test_SOURCES = test_main.cpp ../src/class1.cpp class1_test.cpp class2.cpp ../src/class2_test.cpp ../src/class3.cpp ../src/class4.cpp
prog_test_LDADD = $(BOOST_FILESYSTEM_LIB) $(BOOST_SYSTEM_LIB) $(BOOST_UNIT_TEST_FRAMEWORK_LIB)
Makefile.am in the root folder:
SUBDIRS = src test
dist_doc_DATA = README
ACLOCAL_AMFLAGS = ${ACLOCAL_FLAGS} -I m4
Running test/prog yields the output:
Running 4 test cases...
*** No errors detected
(I don't think you need the contents of my test cases in order to answer my question, so I omitted them for now)
So how can I make automake run my tests every time I run make check?
At least one way of doing this involves setting TESTS variable. Here's what documentation on automake says about it:
If the special variable TESTS is defined, its value is taken to be a list of programs or scripts to run in order to do the testing.
So adding the line
TESTS = $(check_PROGRAMS)
should instruct it to run the tests on make check.
I recently converted my build system to automake/autoconf. In my project I have a few unit tests that need some input data files in the direcory from where they are run. When I run make distcheck and it tries the VPATH build, these tests fail because they are apparently not run from the directory where the input files are. I was wondering if there is some quick fix for this. For example, can I somehow tell the system not to run these tests on make distcheck (but still run them on make check)? Or to cd to the directory where the files are before running the tests?
I had the same problem and used solution similar to William's. My Makefile.am looks something like this:
EXTRA_DIST = testdata/test1.dat
AM_CPPFLAGS = -DDATADIR=\"$(srcdir)/\"
Then, in my unittest, I use the DATADIR define:
string path = DATADIR "/testdata/test1.dat"
This works with make check and make distcheck.
The typical solution is to write the tests so that they look in the source directory for the data files. For example, you can reference $srcdir in the test, or convert test to test.in and refer to #srcdir#.
If your tests are all in the source directory, you can run all the tests in that directory by setting TESTS_ENVIRONMENT in Makefile.am:
TESTS_ENVIRONMENT = cd $(srcdir) &&
This will fail if some of your tests are created by configure and therefore live only in the build directory, in which case you can selectively cd with something like:
TESTS_ENVIRONMENT = { test $${tst} = mytest && cd $(srcdir); true; } &&
Trying to use TESTS_ENVIRONMENT like this is fragile at best, and it would be best to write the tests so that they look in the source directory for the data files.