I realize it may sound like an odd request, and it certainly will not do wonders for test performance, but it's critical that I get a new AppDomain for the start of each unit test.
Currently I'm using xUnit and Resharper as the test runner. But I'm willing to change if there's a different framework that would yield the behaviour that I need.
The xunit resharper runner doesn't have this kind of functionality, and I don't know any test framework that does this out of the box. If you need each test to run in a new AppDomain, I'd write it so that each test created a new AppDomain and ran some custom code in there.
You could probably use some of xunit's features to make this a little easier - the BeforeAfterTestAttribute allows you to run code before and after, or you could pass in a fixture that provides functionality to setup/teardown the AppDomain.
Related
I have my own little cross-platform C++ unit testing framework where unit tests look like this:
#include "Test.h"
DEFINE_TEST(myTest) {
AssertEqual(2+2, 4);
}
and are in .cpp files.
(Seems similar to the way Catch does it, among others I'm sure)
I'd like to integrate this with Xcode, so I can run my tests using Xcode's Run Tests command and utilize other tools which depend on unit tests being run that way. Ideally, I'd like each of my test cases to be a XCTest test case (though I'm fine with manually setting that up for each test) and I'd like my assertions (AssertEqual) to behave like XCTest's (XCTAssert).
Is this possible? If so, how would I do it?
(Note: not switching to just using XCtest because I'd like my tests to work on Windows as well)
Integrating with Xcode basically means ensuring:
Each one of those tests ends up as a test method on a test case class.
Every failing assertion sends -[XCTestCase recordFailure:withDescription:inFile:atLine:expected:] to the currently executing test case class instance.
Xcode will then populate the Test Navigator with all the runtime-discovered tests, organized under their test case class, after the first test run.
You might be able to do a large chunk of the work with macros to let the preprocessor build your test case class/es and methods at compile-time. Otherwise, you'd need to go through and hook everything up at runtime, before the test runner polls the runtime to discover all the XCTestCase subclasses and their test methods.
I've got some unit tests (c++) running in the Visual Studio 2012 test framework.
From what I can tell, the tests are running in parallel. In this case the tests are stepping on each other - I do not want to run them in parallel!
For example, I have two tests in which I have added breakpoints and they are hit in the following order:
Test1 TEST_CLASS_INITIALIZE
Test2 TEST_CLASS_INITIALIZE
Test2 TEST_METHOD
Test1 TEST_METHOD
If the init for Test1 runs first then all of its test methods should run to completion before anything related to Test2 is launched!
After doing some internet searches I am sufficiently confused. Everything I am reading says Visual Studio 2012 does not run tests concurrently by default, and you have to jump through hoops to enable it. We certainly have not enabled it in our project.
Any ideas on what could be happening? Am I missing something fundamental here?
Am I missing something fundamental here?
Yes.
Your should never assume that another test case will work as expected. This means that it should never be a concern if the tests execute synchronously or asynchronously.
Of course there are test cases that expect some fundamental part code to work, this might be own code or a part of the framework/library you work with. When it comes to this, the programmer should know what data or object to expect as a result.
This is where Mock Objects come into play. Mock objects allow you to mimic a part of code and assure that the object provides exactly what you expect, so you don't rely on other (time consuming) services, such as HTTP requests, file stream etc.
You can read more here.
When project becomes complex, the setup takes a fair number of lines and code starts duplicating. Solution to this are Setup and TearDown methods. The naming convention differs from framework to framework, Setup might be called beforeEach or TestInitialize and TearDown can also appear as afterEach or TestCleanup. Names for NUnit, MSTest and xUnit.net can be found on xUnit.net codeplex page.
A simple example application:
it should read a config file
it should verify if config file is valid
it should update user's config
The way I would go about building and testing this:
have a method to read config and second one to verify it
have a getter/setter for user's settings
test read method if it returns desired result (object, string or however you've designed it)
create mock config which you're expecting from read method and test if method accepts it
at this point, you should create multiple mock configs, which test all possible scenarios to see if it works for all possible scenarios and fix it accordingly. This is also called code coverage.
create mock object of accepted config and use the setter to update user's config, then use to check if it was set correctly
This is a basic principle of Test-Driven Development (TDD).
If the test suite is set up as described and all tests pass, all these parts, connected together, should work perfectly. Additional test, for example End-to-End (E2E) testing isn't necessarily needed, I use them only to assure that whole application flow works and to easily catch the error (e.g. http connection error).
I have log4net which writes entries like:
<conversionPattern value="[%date{yyyy-MM-dd HH:mm:ss}] [%property{machineName}] [%property{pid}] [%thread] [%-5level]: %message%newline"/>
It all works fine except when running unit tests. If I do not mock the logger and the tests use the real object then instead of a threadId I get
Agent: adapter run thread for test 'Log4NetLogger_TestLoggingMachineNamePrinted' with id '84e27809-f2b8-45b4-a2e1-ce305d20bc0c'
So obviously log4net gets confused when it is being used from a test runner. If I run the app normally then I get a normal thread id.
Anyone knows a workaround for that? I am using MSTest. Same behaviour happens with the MSTest test runner and the R# test runner.
Thank you in advance for reading my question.
George
Adding a reference to log4net in the unit tests project may do the trick (see this answer).
Having said that, you probably don't need logging in this case (unless these are really Integration tests), so it is best to use a Stub instead of your real logger object.
I am unit testing a StateMachineWorkflow and I create my test methods by clicking in my test project and I make Add - UnitTest. In the project window I select the workflow that I want to test and all the methods in it.
Visual Studio generated a Test Reference folder in my Test Project with an accessor to the workflow. It also generated all the TestMethod() necessary for the testing. All test Methods use a MyWorkflow_Accessor target = new MyWorkflow_Accessor(). When I need to call a function I just do something like target.SendEmail().
Everything works fine, except for one thing: I can't use WorkflowInstanceId of the Workflow, when the code reach a line that uses this it throws an exception in the Workflow, "This is an invalid design time operation. You can only perform the operation at runtime."
Is it possible to inject the WorkflowID by code? Is there any workaround to this situation? I use the WorkflowInstanceId in a lot of functions and changing the Workflow code to match my test doesn't seem like a good idea because I believe the problem is in the test and not in the workflow.
It's not clear from your question if you're using WF 3.5 or WF4 with the state machine update. For the latter, you can use Microsoft.Activities.UnitTesting to test workflows.
It sounds like you're using WF 3.5, though. If this is new development, I would seriously consider moving to WF4. Microsoft basically rewrote WF, and the sooner you switch, the easier your migration path will be.
Otherwise, there is some information on testing with WF 3.5 on MSDN.
I have a unit tests for Zend Framework controllers extending Zend_Test_PHPUnit_ControllerTestCase.
The tests are dispatching an action, which forwards to another action, like this:
// AdminControllerTest.php
public testAdminAction()
$this->dispath('/admin/index/index');
// forwards to login page
$this->assertModule('user');
$this->assertController('profile');
$this->assertController('login');
$this->assertResponseCode(401);
}
// NewsControllerTest.php
public testIndexAction()
{
$this->dispatch('/news/index/index');
$this->assertModule('news');
$this->assertController('index');
$this->assertController('index');
$this->assertResponseCode(200);
}
Both of the tests are passing when they are run as a seperate tests.
When I run them in the same test suite, the second one fails.
Instead dispatching /news/index/index the previous request is dispatched (user module).
How to trace this bug? Looks like I have some global state somewhere in the application, but I'm unable do debug this. How can I dump the objects between the tests in the suite? setUpBefore/AfterClass are static, so there are no so many data about the object instances.
I know this is a kind of guess what question. It's hard to provide reliable data here, because they would took to much place, so feel free to ask for details.
The whole unit test setup is more or less like described in: Testing Zend Framework MVC Applications - phly, boy, phly or Testing Zend Framework Controllers « Federico Cargnelutti.
Solution:
I've determined the issue (after a little nap). The problem was not in unit test setup, but in the tested code.
I use different ACL objects based on module name. Which one to use was determined by static call to action helper, which cached the result in a private static variable to speed things up. This cache was executed only when run in a test suite. I just need more unit tests for this code :)
(I'm sorry for such a rubbish post, but I've stuck with this for a day and I hoped someone else experienced similar kind of this Heisenbug with unit tests in general)
You may try clearingrequest and response objects before dispatching each action, like this:
$this->resetRequest()
->resetResponse()
->dispatch('/news/index/index');