Adding XCTest unit tests to existing app doesn't work - c++

I've added tests (XCTests) to an existing C++ command line app in Xcode 5
via Test Navigator > (+),
changed the extension of the test class to .mm,
added the XCTest framework to the project.
All compiles fine. Now running the tests just gives me a 'Test failed' message, nothing in the console and neither green nor red lights in the Navigator (i.e. no tests executed).
Starting with a fresh Xcode 5 project and changing the extension of the test class to .mm just works fine so I'd assume it's not just about lacking support for Objective-C++ in XCTest.
Even with a plain, vanilla test target added to the existing C++ project the tests fail before ever running.
Any more gotchas to watch out for when adding XCTests to existing (Objective-)C++ targets?
Update #1
With Xcode 5.0.2 (on 10.8.5) xctest now crashes in the same scenario with an
*** NSTask: Task create for path '/Users/XXX/Library/Developer/Xcode/DerivedData/RM_Next_Gen-gpihzjouhxvifqcslmywktktizer/Build/Products/Debug/YYY Tests.xctest/Contents/MacOS/YYY Tests' failed: 22, "Invalid argument". Terminating temporary process.
objc[3478]: GC: forcing GC OFF because OBJC_DISABLE_GC is set
*** multi-threaded process forked ***

Turns out it was a command line C++ project which apparently isn't currently supported by XCTest. Testing the C++ code from an Objective-C(++) project works just fine..
Just hoping for better documentation of XCTest at some point in the (near) future.

I couldn't find any documentation on using XCTest to test code that is not Objective C (C / C++). I am wanting to test a command line C project, and I was concerned that your answer suggested that doing this is unsupported by Xcode. I'm a novice to XCTest and TDD, but I thought others might appreciate what I discovered.
I created a New Project > Command Line Tool, based on C. Project is called foo; Xcode will call the target foo by default too.
-Files created: main.c
Add test target by going to Test Navigator > (+) > New Test Target. Call the target fooTests. Xcode will create this target and an example test within it (all within the file called fooTests.m). The example test will be called testExample and it will contain the test function below:
- (void)testExample
{
XCTFail(#"No implementation for \"%s\"", __PRETTY_FUNCTION__);
}
If you run the tests now (either from the Product>Test menu item or ⌘U) then the test will fail (it's supposed to). For the sake of brevity, I won't go through the TDD cycle, but simply list the files and contents that need to be added to test functions that are not within Objective-C classes.
Let's say that we have a function that we want to test, called bar(), and called from main.c. Go to the project file explorer and add a new header file called bar.h. Add the function prototype:
int bar();
If you want to use XCTest to test a function, it can't be in main.c, so add it to a new C Source file called bar.c. Important: when you add the file, you must add it to the target foo and to fooTests. if you don't add it to fooTests then when compiling the tests the linker won't find the function.
int bar() {
return 0;
}
Within fooTests.m, add the include:
#include "bar.h"
You can now refer to any function contained in bar.h within your tests. Now create 2 unit tests, one that will fail, the other that will pass.
Delete the method testExample and replace with the following code:
- (void)testBarWillFail
{
int rc = bar();
XCTAssertTrue(rc == -1, #"Expected rc==-1, rc==%d", rc);
}
- (void)testBarWillPass
{
int rc = bar();
XCTAssertTrue(rc == 0, #"Expected rc==0, rc==%d", rc);
}
Running the tests will produce the following in the debug window:
Test Suite 'All tests' finished at 2014-01-12 13:38:45 +0000.
Executed 2 tests, with 1 failure (0 unexpected) in 0.000
Within the Issue Navigator pane you'll see:
file: /foo/fooTests/fooTests.m: test failure: -[fooTests testBarWillFail] failed: ((rc == -1) is true) failed - Expected rc==-1, rc==0
This is all as expected! Repeat the process with all other C code you wish to test.
If anyone knows a better way of doing this then I'd be delighted to hear it!

Related

Rust tests fail to even run

I'm writing a project to learn how to use Rust and I'm calling my project future-finance-labs. After writing some basic functions and verifying the app can be built I wanted to include some tests, located in aggregates/mod.rs. [The tests are in the same file as the actual code as per the documentation.] I'm unable to get the tests to run despite following the documentation to the best of my ability. I have tried to build the project using PowerShell as well as Bash. [It fails to run on Fedora Linux as well]
Here is my output on Bash:
~/future-finance-labs$ cargo test -- src/formatters/mod.rs
Finished test [unoptimized + debuginfo] target(s) in 5.98s
Running target/debug/deps/future_finance_labs-16ed066e1ea3b9a1
running 0 tests
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out
Using PowerShell I get the same output with some errors like the following:
error: failed to remove C:\Users\jhale\AppData\Local\Packages\CanonicalGroupLimited.UbuntuonWindows_79rhkp1fndgsc\LocalState\rootfs\home\jhale\future-finance-labs\target\debug\build\mime_guess-890328c8763afc22\build_script_build-890328c8763afc22.build_script_build.c22di3i8-cgu.0.rcgu.o: The system cannot find the path specified. (os error 3)
After my initial excitement at the prospect of writing a few tests that passed on the first attempt, I quickly realized all the green was indicative; rather, of a failure to even run the tests. I just want to run the unit tests. Running cargo test alone without a separate and file fails as well. Why can't I run any test in this project with my current setup?
It can't find your test because the rust compiler doesn't know about it. You need to add mod aggregates to main.
mod aggregates;
fn main() {
println!("Hello, world!");
}
After you do that, you'll see that your aggregates/mod.rs doesn't compile for many reasons.
And as Mihir was trying to say, you need to use the name of the test, not the name of the file to run a specific test:
cargo test min_works
cargo test aggregates
See also:
How do I “use” or import a local Rust file?
Rust Book: Controlling How Tests Are Run

Robolectric unit tests using legacy instead of binary resources only when running multi-module unit test run config

I have a multi-module android project. I have a bunch of unit tests in each module and I have always been able to run them all at once using a run configuration like this one:
Many of my tests use a base class that runs with RobolectricTestRunner. This base class looks like this:
#RunWith(RobolectricTestRunner::class)
#Config(application = AndroidTest.ApplicationStub::class,
manifest = Config.NONE,
sdk = [21])
abstract class AndroidTest {
#Suppress("LeakingThis")
#Rule #JvmField val injectMocks = InjectMocksRule.create(this#AndroidTest)
fun application(): Application = ApplicationProvider.getApplicationContext()
internal class ApplicationStub : Application()
}
**When running these tests using the above config, I get the error **
[Robolectric] NOTICE: legacy resources mode is deprecated; see http://robolectric.org/migrating/#migrating-to-40
This makes many of my tests fail with ResourceNotFoundException
However, when I run tests only in a specific module, everything passes. This is because Robolectric now uses Binary resources:
[Robolectric] sdk=21; resources=BINARY
I have followed the migration instructions in build.gradle files for each module, having added the following in each android block:
testOptions {
unitTests {
includeAndroidResources = true
returnDefaultValues = true
}
}
One clue I have found but have been unable to fix is this when I run the ALL UNIT TEST task:
WARNING: No manifest file found at build/intermediates/merged_manifests/debug/../../library_manifest/debug/AndroidManifest.xml.
Falling back to the Android OS resources only.
No such manifest file: build/intermediates/merged_manifests/debug/../../library_manifest/debug/AndroidManifest.xml
To remove this warning, annotate your test class with #Config(manifest=Config.NONE).
I have tried, as you have seen, to add the manifest=Config.NONE, which had no effect (and is now deprecated anyway).
Edit: Also tried android.enableUnitTestBinaryResources = true in settings.gradle, but this prevents the app from building due to it being a deprecated flag in the current gradle tools.
Thanks for any help provided!
So with the default unit test run platform being changed to Gradle in Android Studio, I managed to figure out a way to run unit tests in multiple modules all at once, circumventing the Robolectric bug.
First, go into run configurations and create a new Gradle Config.
Then, as the gradle project, select the top level project.
For arguments, use --tests "*"
And now for the gradle tasks, this is a little bit more error-prone. Here is an example of how I have it setup for my project:
:androidrma:cleanTestGoogleDebugUnitTest :androidrma:testGoogleDebugUnitTest
:calendar:cleanTestDebugUnitTest :calendar:testDebugUnitTest
:gamification:cleanTest :gamification:test
:player:cleanTest :player:test
:playlists:cleanTest :playlists:test
:sleepjournal:cleanTest :sleepjournal:test
:sound-content-filters:cleanTest :sound-content-filters:test
Please note that I inserted new lines between each module for more clarity here, in the tasks, just separate each entry with a space.
For your app module, in my case named androidrma, you must use your build variants name in the cleanTestUnitTest and testUnitTest , in my case being GoogleDebug.
If we look at the calendar module, it is an android module, , so it still operates with the same logic as the appModule.
However, if you look at player, playlists, sleepjournal, etc. those are pure kotlin modules. The tasks thus differ in their syntax.
Once you have entered all this information and everything is functioning, I recommend checking "store as project file" checkbox at the top right of the run config setup screen.
This works in Android Studio 4.2 as well as Arctic Fox, haven't tested on other versions.

Which unit test is currently running?

This question is related to the problem where my unit test procedure is crashing but I don't know on which unit test.
I realise this question is a duplicate of How can I find out which test method in a batch of test methods fails to run? but I need to try anyway:
I'm running some unit tests (about 118 of them), but one of them seems to make the Visual Studio unit test environment crash. This is what I see in the "output" window's "Tests" tab:
[5/02/2018 11:13:18 Informational] ------ Run test started ------
[5/02/2018 11:13:38 Error] The active Test Run was aborted because the execution process exited unexpectedly. The test execution process crashed while running the tests. To investigate further, open file:///C:/Users/DominiqueDS/AppData/Local/CrashDumps/vstest.executionengine.x86.exe.19136.dmp file in Visual Studio and choose "Debug in mixed mode".
[5/02/2018 11:13:38 Informational] ========== Run test finished: 65 run (0:00:19,6926824) ==========
The reason that I have that dump file is thanks to my Windows configuration which automatically creates such dumpfiles in case of a crashing application. (The procedure for this configuration is found under this URL)
Examining the dump file gives me more information on the test, which is failing: I have a hunch where I can find it, and in another thread, I find a function, calling a method which seems to contain the name of the unit test I'm running.
Although I know which test is failing, it's based on hunches and on dumpfiles I should even not generate, so I'm looking for another solution:
In the function TEST_METHOD in the file "c:\Program Files (x86)\Microsoft Visual Studio\2017\Professional\VC\Auxiliary\VS\UnitTest\include\CppUnitTest.h", I've tried to add the following lines:
cout << "Test : ";\
cout << methodName;\
cout << "\n";\
I was hoping that this would show every test I wanted to run, but instead I get the error message that this is wrong (also using std::cout and OutputDebugString() function seems not to be allowed).
Hence my question: does anybody know a way to find out, in case one of my unit tests makes my test environment crash, how can I find out which test this is, without needing to generate and debug the Visual Studio dumpfile?
For your information: looking in the "Test Explorer" is not helping: I have a list of disabled tests, a list of successful ones, and a list of some which are not finished, and my failing test is not the first of the "not finished" ones.
Found it: I've added the following line in the definition of TEST_METHOD():
Microsoft::VisualStudio::CppUnitTestFramework::Logger::WriteMessage(L#methodName);\

vs 2013 : breakpoint into cppunittest test throws exception

I am testing my c++11 static lib in vs 2013 environment.
I followed this nice tutorial and testing + code coverage are working fine.
Now I need step-by-step : I added breakpoints and executed the "Debug Selected Test" command and I get the following message : vstest.executionengine.exe has triggered a breakpoint and the callstack brings me at CppUnitTest.h:465 : (static_cast<ThisClass *>(this)->*method2)();, eg at the root call of the method I want to break into. No way to see the code inside this call.
My question : how to break into my code during debugging Ms cpp unit test ?
I found the problem. Some breakpoints were activated inside the static lib used by the test dll. It seems breakpoints inside static lib generates something wrong for the test framework.
To reproduce :
create c++ static lib project with a simple function, int foo(){ return 0;}
create a test-dll project
add a test function which calls foo
set a breakpoint in foo
execute "run selected test" : it will work
execute "debug selected test" : it will block just before calling the test-method test-generated class.
I am still interested in explaining this non-intuitive behaviour.

NetBeans generated Makefile ignores test return codes

I have a C++ project in NetBeans using generated Makefiles. I set up a job in Jenkins (continuous integration server) to run the tests configured in NetBeans. Now Jenkins runs the tests and captures their output, but it considers the build successful even when a test fails.
I'm using the Boost Unit Test Framework which of course returns a non-zero code on failure as any proper *nix program would. So I wondered why Jenkins didn't understand when a test failed. Then I found this in the generated Makefile-Debug.mk from NetBeans:
# Run Test Targets
.test-conf:
#if [ "${TEST}" = "" ]; \
then \
${TESTDIR}/TestFiles/f1 || true; \
${TESTDIR}/TestFiles/f2 || true; \
else \
./${TEST} || true; \
fi
So it seems like they deliberately ignore the return value of all tests. But this doesn't make sense, because then what are your tests testing?
I tried to find a setting in NetBeans to say "Let failing tests break the build" but didn't find anything. I also tried to find a bug in the NetBeans tracker for this but didn't see any in my brief search.
Is there any other reasonable solution? I want Jenkins to fail my build if any test fails. Right now it only fails if a test fails to build, but if it builds and fails to run, success is reported.
It turns out that NetBeans (up to version 8 at least) cannot support this. What I did to work around it is to do make build-tests rather than make test in Jenkins, followed by a loop over all the generated test files (TestFiles/f* in the build directory) to run them.
This is a major shortcoming in NetBeans' Makefile generator, as it is fundamentally incompatible with running tests outside of NetBeans itself. Thanks to #HEKTO for the link which led me to this page about writing NetBeans testing plugins: http://wiki.netbeans.org/CND69UnitTestsPluginTutotial
What that page tells you is basically that NetBeans relies on parsing the textual output of tests to determine success or failure. What it doesn't tell you is that NetBeans generates defective Makefiles which ignore critical failures in tests, including aborts, segmentation faults, assertion failures, uncaught exceptions, etc. It assumes you will use a test framework that it knows about (which is only CppUnit), or manually write magic strings at the right moments in your test programs.
I thought about taking the time to write a NetBeans unit test plugin for the Boost Unit Test Framework, but it won't help Jenkins at all: the plugins are only used when tests are run inside NetBeans itself, to display pretty status indicators.