I'm using CTest with CMake to run some tests. I use the enable_testing() command which provides me with a default command for make test. All of the tests in my subdirectory are accounted for (by doing an add_test command) and make test works great, except one problem.
There is a certain test, which I've named skip_test, that I do NOT want being run when I do make test. I would like to add a custom target so I can run make skip_test and it will run that test.
I can do this by doing add_custom_target(skip_test ...) and providing CTest with the -R flag and telling it to look for files containing "skip_test" in their name. This also seems to work. My problem now is: how can I get the make test command to ignore skip_test?
If I try commenting out enable_testing and adding my own add_custom_target(test ....), I get "No tests found!!!" now for either make test or make skip_test. I also tried making a Custom CTest file and adding set(CTEST_CUSTOM_TESTS_IGNORE skip_test). This worked so that now make test ignored "skip_test", but now running make skip_test responds with "no tests found!!!".
Any suggestions would be appreciated!
I actually used a different solution. Here is what I did. For the tests that I wanted to exclude, I used the following command when adding them:
"add_test( ..... CONFIGURATIONS ignore_flag)" where ignore_flag is whatever phrase you want. Then, in my CMakeLists.txt, when I define a custom target
add_custom_target( ignore_tests ...)
I give it ctest .... -C ignore_flag
Now, make test WILL skip these tests! make ignore_Tests will run the ignored tests + the un-ignored tests, which I'm okay with.
I'm not sure of a way to do this entirely via CTest, but since you've tagged this question with "googletest", I assume you're using that as your test framework. So, you could perhaps make use of Gtest's ability to disable tests and also to run disabled tests.
By changing the test(s) in question to have a leading DISABLED_ in their name(s), these won't be run by default when you do make test.
You can then add your custom target which will invoke your test executable with the appropriate Gtest flags to run only the disabled tests:
add_custom_target(skip_test
MyTestBinary --gtest_filter=*DISABLED_* --gtest_also_run_disabled_tests VERBATIM)
It's a bit of an abuse of the Gtest functionality - it's really meant to be used to temporarily disable tests while you refactor whatever to get the test passing again. This beats just commenting out the test since it continues to compile it, and it gives a nagging reminder after running the suite that you have disabled tests.
Related
Have some auto-generated golang code for protobuf messages and I'm looking to add some additional testing, without locating the file under the same directory path. This is to allow easy removal of the existing generated code to be sure that if a file is dropped from being generated, it's not left included in the codebase by accident.
The current layout of these files are controlled by prototool so I have something like the following:
/pkg/<other1>
/pkg/<other2>
/pkg/<name-generated>/v1/component_api.pb.go
/pkg/<name-generated>/v1/component_api.pb.gw.go
/pkg/<name-generated>/v1/component_api.pb.validate.go
The *.validate.go comes from envoyproxy/protoc-gen-validate, and *.pb.go & *.pb.gw.go are coming from protobuf and grpc libraries. The other1 and other2 are two helper libraries that we have included along with the generated code to make it easier for client side apps. The server side is in a separate repo and imports as needed.
Because it's useful to be able to delete /pkg/<name> before re-running prototool I've placed some tests of component_api (mostly to exercise the validate rules generated automatically) under the path:
/internal/pkg/<name>/v1/component_api_test.go
While this works for go test -v ./..., it appears not to work to well when generating coverage with -coverpkg.
go test -coverpkg=./... -coverprofile=coverage/go/coverage.out -v ./...
go build <pkgname>/internal/pkg/<name>/v1: no non-test Go files in ....
<output from the tests in /internal/pkg/<name>/v1/component_api_test.go>
....
....
coverage: 10.5% of statements in ./...
ok <pkgname>/internal/pkg/<name>/v1 0.014s coverage: 10.5% of statements in ./...
FAIL <pkgname>/pkg/other1 [build failed]
FAIL <pkgname>/pkg/other2 [build failed]
? <pkgname>/pkg/<name>/v1 [no test files]
FAIL
Coverage tests failed
Generated coverage/go/html/main.html
The reason for use of -coverpkg is that without it there doesn't seem to be anything spotting that any of the code under <pkgname>/pkg/<name>/v1 is covered, and we've see issues with what it reports previously not showing the real level of coverage, which are solved by use of -coverpkg:
go test -cover -coverprofile=coverage/go/coverage.out ./...
ok <pkgname>internal/pkg/<name>/v1 0.007s coverage: [no statements]
ok <pkgname>/pkg/other1 0.005s coverage: 100.0% of statements
ok <pkgname>/pkg/other2 0.177s coverage: 100.0% of statements
? <pkgname>/pkg/<name>/v1 [no test files]
Looking at the resulting coverage/go/coverage.out includes no mention of anything under <pkgname>/pkg/<name>/v1 being exercised.
I'm not attached to the current layout beyond being limited on <pkgname>/pkg/<name>/v1 being automatically managed by prototool and it's rules around naming for the generated files. Would like to ensure the other modules we have can remain exported to be used as helper libraries and I would like to be able to add tests for <pkgname>/pkg/<name>/v1 without needing to locate them in the same directory to allow for easy delete + recreate of generated files, while still getting sensible coverage reports.
I've tried fiddling with the packages passed to -coverpkg and replacing ./... on the command-line and haven't been able to come up with something that works. Perhaps I'm just not familiar with the right invocation?
Other than that is there a different layout that will take care of this for me?
To handle this scenario, simply create a doc.go file in the same directory as the dis-located tests with just the package and comment. This will allow the standard arguments to work and golang appears to be reasonably happy with an empty file.
Once in place the following will work as expected.
go test -coverpkg=./... -coverprofile=coverage/go/coverage.out -v ./...
Idea based on suggestion in https://stackoverflow.com/a/47025370/1597808
I am maintaining an autoconf package and wanted to integrate automatic testing. I use the Boost Unit Test Framework for my unit tests and was able to sucessfully integrate it into the package.
That is it can be compiled via make check, but is is not run (although I read that make check both compiles and runs the tests). As result, I have to run it manually after building the tests which is cumbersome.
Makefile.am in the test folder looks like this:
check_PROGRAMS = prog_test
prog_test_SOURCES = test_main.cpp ../src/class1.cpp class1_test.cpp class2.cpp ../src/class2_test.cpp ../src/class3.cpp ../src/class4.cpp
prog_test_LDADD = $(BOOST_FILESYSTEM_LIB) $(BOOST_SYSTEM_LIB) $(BOOST_UNIT_TEST_FRAMEWORK_LIB)
Makefile.am in the root folder:
SUBDIRS = src test
dist_doc_DATA = README
ACLOCAL_AMFLAGS = ${ACLOCAL_FLAGS} -I m4
Running test/prog yields the output:
Running 4 test cases...
*** No errors detected
(I don't think you need the contents of my test cases in order to answer my question, so I omitted them for now)
So how can I make automake run my tests every time I run make check?
At least one way of doing this involves setting TESTS variable. Here's what documentation on automake says about it:
If the special variable TESTS is defined, its value is taken to be a list of programs or scripts to run in order to do the testing.
So adding the line
TESTS = $(check_PROGRAMS)
should instruct it to run the tests on make check.
I'm using Autoconf to build my c++ project. It uses third party code which is also built with the help of Autoconf/Automake. So in my configure.ac I've got the following line:
AC_CONFIG_SUBDIRS([subdirectoryname])
Everything works fine, but I also use the feature to let tests be automatically made if make check is executed - which is done by the third party code as well. Because these tests take a while it's annoying to execute them each time I want to test my own code. So is there any way to avoid that the check option is passed to the subdirectory's Makefile?
Update: Overriding check-recursive does not seem to be an option, as my top-level Makefile.am looks (more or less) like this:
SUBDIRS=library src
So disabling checking on this level would also disable the checking inside my src folder. And that's not what I want to achieve. I just want to disable the checking in the library directory.
Overriding check-recursive in your Makefile.am should work:
check-recursive:
#true
or, if you only wanted to check in a specific directory:
check-recursive:
$(MAKE) -C src check
according to the autoconf manual, it will execute a configure.gnu script in the subdirectory if it finds one. Theoretically that could be a script which adds a --disable-tests or similar option to a call to ./configure
That said, I've yet to get this to work on a project of my own. :-/
I have an error similar to the one in this post. Now, I'm sure I've made some stupid error somewhere, probably related to releasing an object or an observer or what-not, but since I can't seem to find a way to debug the code I thought I could use the NSDebugEnabled, NSZombieEnabled and MallocStackLogging (as shown here).
Can it be done using OCUnit? If so, how? I just can't find an "executable" to set these parameters on...
Thanks!
Aviad.
Unfortunately, Dave's solution didn't work - I kept getting errors and mistakes. I eventually got GHUnit to work on my project, found the problem by debugging, but it had its own problems so I now use both it and OCUnit which is slightly better integrated in terms of showing the results in the results tab.
sigh. When will we get to see a good, complete unit testing framework for Obj-C?
This may have been fixed in recent Xcodes, but I get zombies by doing
Go into schemes (cmd <)
Open Test, then Arguments tab
Uncheck "Use the Run action's arguments and environment variables"
"+" an environment variable "NSZombieEnabled" = "YES"
Well, NSZombieEnabled and friends are environment variables, which means they have to be run on an executable. The default setup for a unit testing bundle is for the tests to be run during the build process, and not during execution.
So the way to fix this is to make it so that your tests don't run during the build phase, but instead run them as part of an executable.
Here's how I do that:
Inside your Unit Test bundle target, remove the "Run Script" build phase. It's that step that executes the tests after compiling them.
From the Project menu, choose "New Custom Executable..." and name it something meaningful, like "otest"
Make the executable path to be the otest binary, which should be located at /Developer/Tools/otest
Set the following environment variables on the otest executable:
DYLD_FRAMEWORK_PATH => {UnitTest.bundle}/Contents/Frameworks
DYLD_LIBRARY_PATH => {UnitTest.bundle}/Contents/Frameworks
Set the following program arguments on the otest executable:
-SenTest All (this will run all of the unit tests)
{UnitTest.bundle}
You can now select your unit test bundle as the active target, and the otest executable as the active executable, and then build and debug. This will let you set breakpoints, set other environment variables (like NSZombieEnabled), and so on.
If you only want to debug a certain suite or specific unit test, you can change the -SenTest All argument to -SenTest MyUnitTestSuite or -SenTest MyUnitTestSuite/myUnitTestMethod.
It took me quite some time but I finally managed to make it work for my project.
To create the "logic" tests I followed Apple guidelines on creating logic tests.
This works fine once you understand that the logic tests are run during build.
To be able to debug those tests it is required to create a custom executable that will call those tests. The article by Sean Miceli on the Grokking Cocoa blog provides all the information to do this. Following it however did not yield immediate success and needed some tweaking.
I will go over the main steps presented in Sean's tutorial providing some "for dummies" outline which took me some time to figure out:
Setup a target that contains the unit tests but DOES NOT run them
Setup the otest executable to run the tests
Setup the otest environment variables so that otest can find your unit tests
Step 1 - Setting up the target
Duplicate your unit tests target located under your project Targets. This will also create a duplicate of your unit tests product (.octest file). In the figure below "UnitTest" is the original target.
Rename both the unit tests target and the unit tests product (.octest file) to the same name. In the figure below "UnitTestsDebug" is the duplicate target.
Delete the RunScript phase of the new target
The name of both can be anything but I would avoid spaces.
Step 2 - Setting up otest
The most important point here is to get the correct otest, i.e. the one for your current iOS and not the default Mac version. This is well described in Sean's tutorial. Here are a few more details which helped me setting things right:
Go Project->New Custom Executable. This will pop open a window prompting you to enter an Executable Name and an Executable Path.
Type anything you wish for the name.
Copy paste the path to your iOS otest executable. In my case this was /Developer/Platforms/iPhoneSimulator.platform/Developer/SDKs/iPhoneSimulator4.2.sdk/Developer/usr/bin/otest
Press enter. This will bring you to the configuration page of your executable.
The only thing to change at this point is to select "Path Type: Relative to current SDK". Do not type in the path, this was done at step 3.
Step 3 - Setting up the otest arguments and environment variables
The otest arguments are straightforward to setup... But this proved to be my biggest problem. I initially had named my logic test target "LogicTests Debug". With this name and "LogicTests Debug.octest" (with quotes) as argument to otest I kept having otest terminating with exit code 1 and NEVER stopping into my code...
The solution: no space in your target name!
The arguments to otest are:
-SenTest Self (or All or a test name - type man otest in terminal to get the list)
{LogicTestsDebug}.octest - Where {LogicTestsDebug} needs to be replaced by your logic test bundle name.
Here is the list of environment variables for copy/pasting:
DYLD_ROOT_PATH: $SDKROOT
DYLD_FRAMEWORK_PATH: "${BUILD_PRODUCTS_DIR}: ${SDK_ROOT}:${DYLD_FRAMEWORK_PATH}"
IPHONE_SIMULATOR_ROOT: $SDKROOT
CFFIXED_USER_HOME: "${HOME}/Library/Application Support/iPhone Simulator/User"
DYLD_LIBRARY_PATH: ${BUILD_PRODUCTS_DIR}:${DYLD_LIBRARY_PATH}
DYLD_NEW_LOCAL_SHARED_REGIONS: YES
DYLD_NO_FIX_PREBINDING: YES
Note that I also tried the DYLD_FORCE_FLAT_NAMESPACE but this simply made otest crash.
Step 4 - Running your otest executable
To run your otest executable and start debugging your tests you need to:
Set your active target to your unit test target (LogicTestsDebug in my case)
Set your active executable to your otest executable
You can build and run your executable and debug your tests with breakpoints.
As a side note if you are having problems running your otest executable it can be related to:
Faulty path. I had lots of problem initially because I was pointing to the mac otest. I kept crashing on launch with termination code 6.
Faulty arguments. Until I removed the space from bundle (.octest) name I kept having otest crash with exit code 1.
Wrong path in environment variables. [Sean tutorial][8] has lots of follow-up questions giving some insight on what other people tried. The set I have now seems to work so I suggest you start with this.
You may get some message in the console which might lead you to think something is wrong with your environment variables. You may notice a message regarding CFPreferences. This message is not preventing the tests from running properly so don't focus on it f you have problems running otest.
Last once everything is working you will be able to stop at breakpoints in your tests.
One last thing...
I've read on many blogs that the main limitation of the integrated XCode SenTestKit is that tests cannot be run while building the application. Well as it turns out this is in fact quite easy to manage. You simply need to add your Logic tests bundle as a dependency to your application project. This will make sure your logic tests bundle is built, i.e. all tests are run, before your application is built.
To do this you can drag and drop your logic test bundle onto your application target.
I'm using GNU autotools for the build system on a particular project. I want to start writing automated tests for verifcation. I would like to just type "make check" to have it automatically run these. My project is in C++, although I am still curious about writing automated tests for other languages as well.
Is this compatible with pretty much every unit testing framework out there (I was thinking of using cppunit)? How do I hook these unit testing frameworks into make check? Can I make sure that I don't require the unit test software to be installed to be able to configure and build the rest of the project?
To make test run when you issue make check, you need to add them to the TESTS variable
Assuming you've already built the executable that runs the unit tests, you just add the name of the executable to the TESTS variable like this:
TESTS=my-test-executable
It should then be automatically run when you make check, and if the executable returns a non-zero value, it will report that as a test failure. If you have multiple unit test executables, just list them all in the TESTS variable:
TESTS=my-first-test my-second-test my-third-test
and they will all get run.
I'm using Check 0.9.10
configure.ac
Makefile.am
src/Makefile.am
src/foo.c
tests/check_foo.c
tests/Makefile.am
./configure.ac
PKG_CHECK_MODULES([CHECK], [check >= 0.9.10])
./tests/Makefile.am for test codes
TESTS = check_foo
check_PROGRAMS = check_foo
check_foo_SOURCES = check_foo.c $(top_builddir)/src/foo.h
check_foo_CFLAGS = #CHECK_CFLAGS#
and write test code, ./tests/check_foo.c
START_TEST (test_foo)
{
ck_assert( foo() == 0 );
ck_assert_int_eq( foo(), 0);
}
END_TEST
/// And there are some tcase_xxx codes to run this test
Using check you can use timeout and raise signal. it is very helpful.
You seem to be asking 2 questions in the first paragraph.
The first is about adding tests to the GNU autotools toolchain - but those tests, if I'm understanding you correctly, are for both validating that the environment necessary to build your application exists (dependent libraries and tools) as well as adapt the build to the environment (platform specific differences).
The second is about unit testing your C++ application and where to invoke those tests, you've proposed doing so from the autotools tool chain, presumably from the configure script. Doing that isn't conventional though - putting a 'test' target in your Makefile is a more conventional way of executing your test suite. The typical steps for building and installing an application with autotools (at least from a user's perspective, not from your, the developer, perspective) is to run the configure script, then run make, then optionally run make test and finally make install.
For the second issue, not wanting cppunit to be a dependency, why not just distribute it with your c++ application? Can you just put it right in what ever archive format you're using (be it tar.gz, tar.bz2 or .zip) along with your source code. I've used cppunit in the past and was happy with it, having used JUnit and other xUnit style frameworks.
Here is a method without dependencies:
#src/Makefile.am
check_PROGRAMS = test1 test2
test1_SOURCES = test/test1.c code_needed_to_test1.h code_needed_to_test1.c
test2_SOURCES = test/test2.c code_needed_to_test2.h code_needed_to_test2.c
TESTS = $(check_PROGRAMS)
The make check will naturally work and show formatted and summarized output:
$ make check
...
PASS: test1
PASS: test2
============================================================================
Testsuite summary for foo 1.0
============================================================================
# TOTAL: 2
# PASS: 2
# SKIP: 0
# XFAIL: 0
# FAIL: 0
# XPASS: 0
# ERROR: 0
============================================================================
When you do a make dist nothing from src/test/* will be
in the tarball. Test code is not in the dist, only source will be.
When you do a make distcheck it will run make check and run your tests.
You can use Automake's TESTS to run programs generated with check_PROGRAMS but this will assume that you are using a log driver and a compiler for the output. It is probably easier to still use check_PROGRAMS but to invoke the test suite using a local rule in the Makefile:
check_PROGRAMS=testsuite
testsuite_SOURCES=...
testsuite_CFLAGS=...
testsuite_LDADD=...
check-local:
./testsuite