I am using the unit testing features in Visual Studio 2013 for my application.
I am trying to write a test for a class whereby you pass in a specific object to the constructor, and depending on the state of the object passed, an exception may be thrown.
I have written stubs for each object state, and have written test cases for the scenarios where the constructor will throw an exception as follows:
TEST_METHOD(constructor_ExceptionRaised)
{
// arrange
const InvalidStub stub;
// act
auto act = [stub] { const Foo foo(stub); };
// assert
Microsoft::VisualStudio::CppUnitTestFramework::Assert::ExpectException
<MyException>(act);
}
How should I approach a scenario where I want to pass a valid stub and simply assert that no exception was raised? I want to be purely concerned with a specific MyException not being thrown (rather than any exception).
I have hacked together a test method as follows but not sure if there is a simply "1 line" approach that would fit my needs:
TEST_METHOD(constructor_NoException)
{
// arrange
const ValidStub stub;
try
{
// act
const Foo foo(stub);
}
// assert
catch (MyException e)
{
Microsoft::VisualStudio::CppUnitTestFramework::Assert::Fail();
}
catch (...)
{
Microsoft::VisualStudio::CppUnitTestFramework::Assert::Fail();
}
}
I am not confident I need to also fail "any exception" being raised, as this should(?) be picked up by the test runner (i.e. fail the test). Along the same reasoning, would the following essentially be the same test:
TEST_METHOD(constructor_NoException)
{
// arrange
const ValidStub stub;
// act
const Foo foo(stub);
// assert
// no exception
}
I used the following test method to show that a constructor doesn't throw an exception:
TEST_METHOD(constructor_NoException)
{
// arrange
const ValidStub stub;
// act
const Foo foo(stub);
// assert
Microsoft::VisualStudio::CppUnitTestFramework::Assert::IsTrue(true);
}
When an exception is raised the test automatically fails. The exception details are given in the failure message.
When no exception is raised the test will pass as I am asserting true == true.
Related
I'm unable to check the exception throw by my code in gtests. Here's a snippet of the test suite which runs the test:
EXPECT_THROW({
try{
// Insert a tuple with more target columns than values
rows_changed = 0;
query = "INSERT INTO test8(num1, num3) VALUES(3);";
txn = txn_manager.BeginTransaction();
plan = TestingSQLUtil::GeneratePlanWithOptimizer(optimizer, query, txn);
EXPECT_EQ(plan->GetPlanNodeType(), PlanNodeType::INSERT);
txn_manager.CommitTransaction(txn);
TestingSQLUtil::ExecuteSQLQueryWithOptimizer(
optimizer, query, result, tuple_descriptor, rows_changed, error_message);
}
catch (CatalogException &ex){
EXPECT_STREQ("ERROR: INSERT has more target columns than expressions", ex.what());
}
}, CatalogException);
I'm pretty sure that CatalogException is thrown. I even tried getting the details of the thrown exception by outputting it to cerr, and it showed Exception Type: Catalog.
This is not a duplicate question, I searched for answers on SO and I'm not using new in my code which throws the error. Here's the snippet which does that:
if (columns->size() < tup_size)
throw CatalogException(
"ERROR: INSERT has more expressions than target columns");
Finally, here's the definition of CatalogException:
class CatalogException : public Exception {
CatalogException() = delete;
public:
CatalogException(std::string msg) : Exception(ExceptionType::CATALOG, msg) {}
};
The idea from EXPECT_THROW is, that the macro catches the exception. If you catch the exception by yourself, gmock don't now anything about a thrown exception.
I suggest to just write the statement into the EXPECT_THROW, which actually trigger the exception. Everything else can be written before.
For example:
TEST(testcase, testname)
{
//arrange everything:
//...
//act + assert:
EXPECT_THROW(TestingSQLUtil::ExecuteSQLQueryWithOptimizer( optimizer, query, result,
tuple_descriptor, rows_changed, error_message)
,CatalogException);
}
I assume, that TestingSQLUtil::ExecuteSQLQueryWithOptimizer is trigger the thrown exception.
addition:
I tried to rebuild your exception hierarchy. This example works for me very well. The test passes, which means the exception is thrown.
enum class ExceptionType
{
CATALOG
};
class Exception {
public:
Exception(ExceptionType type, std::string msg) {}
};
class CatalogException : public Exception {
CatalogException() = delete;
public:
CatalogException(std::string msg) : Exception(ExceptionType::CATALOG, msg) {}
};
void testThrow() {
throw CatalogException( "ERROR: INSERT has more expressions than target columns");
}
TEST(a,b) {
EXPECT_THROW( testThrow(), CatalogException);
}
I'm new to using test projects in VS, and am using VS 2012. Lets say I have a WCF service that performs a read on a database and returns an object with a status code (something numeric) and a status message ("Success" or "Danger Will Robinson").
As part of a unit test, wouldn't I want the ability to test that these error codes and messages return appropriately? If so, how would I go about testing for these exception circumstances?
There are two types of errors that you want to test. Those returned as part of a result and those that occur as exceptions. You could test the first type like:
[UnitTest]
public void MethodUnderTest_ScenarioBeingTested_ResultOfTest()
{
// Arrange
string expectedError = "Danger Will Robinson";
IMySpecialService service = new FakeSpecialService();
// Act
var result = service.CallOperation(thisArgumentCausesError);
// Assert
Assert.AreEqual(expectedError, result.ReturnMessage);
}
The above will test an error message returned from some fictive object and will be compared against an expected error message "Danger Will Robinson";
When you want to test exceptions you use similar approach but decorate your unit test class with a special attribute:
[ExpectedException(typeof(System.DivideByZeroException))]
[UnitTest]
public void MethodUnderTest_ScenarioBeingTested_ExpectedResult()
{
// Arrange
int i = 0;
int j = 0;
IMathService service = new FakeService();
// Act
decimal result = service.PerformDivision(i, j);
// Assert
// This part is never reached because Assertion is handled by ExpectedException
}
In this case you know that an exception will be raised and which will not be handled. Here we are testing some fictive service that performs some math operations, like division.
This should give you an idea how to approach your unit tests.
I am trying to test a class similar to the example below:
public class Service : IService
{
public string A(string input)
{
int attemptCount = 5;
while (attemptCount > 0)
{
try
{
return TryA(input);
}
catch (ArgumentOutOfRangeException)
{
attemptCount--;
if (attemptCount == 0)
{
throw;
}
// Attempt 5 more times
Thread.Sleep(1000);
}
}
throw new ArgumentOutOfRangeException();
}
public string TryA(string input)
{
// try actions, if fail will throw ArgumentOutOfRangeException
}
}
[TestMethod]
public void Makes_5_Attempts()
{
// Arrange
var _service = MockRepository.GeneratePartialMock<Service>();
_service.Expect(x=>x.TryA(Arg<string>.Is.Anything)).IgnoreArguments().Throw(new ArgumentOutOfRangeException());
// Act
Action act = () => _service.A("");
// Assert
// Assert TryA is attempted to be called 5 times
_service.AssertWasCalled(x => x.TryA(Arg<string>.Is.Anything), opt => opt.Repeat.Times(5));
// Assert the Exception is eventually thrown
act.ShouldThrow<ArgumentOutOfRangeException>();
}
The partial mocking doesn't seem to accept my expectation. When I run the test I receive an error about the input. When I debug, I see that the actual implementation of the method is being executed instead of the expectation.
Am I doing this test correctly? According to the documentation ( http://ayende.com/wiki/Rhino%20Mocks%20Partial%20Mocks.ashx ): "A partial mock will call the method defined on the class unless you define an expectation for that method. If you have defined an expectation, it will use the normal rules for this."
It's is important to note that mocking frameworks like Rhinomocks, Moq and NSubstitute use a feature in .NET called DynamicProxy that dynamically generates a derived class of the mock in memory. Classes must:
be an interface; or
non-sealed class with parameterless constructor; or
derive from MarshalByRefObject (moq has moved away from this feature)
Methods must be part of the interface or made virtual so that alternate behaviors can be substituted at runtime.
I have a DTO which I'm populating from the request object, and the request object has many fields. I want to write a test to check if the populateDTO() method is putting values in the right places or not. If I follow the rule of one assert per test, I would have to write a large number of tests, to test each field. The other approach would be to write multiple asserts in a single test. Is it really recommended to follow one assert per test rule or can we relax in these cases. How do I approach this problem?
Keep them separate. A unit test is supposed to tell you which unit failed. Keeping them separate also allows you to isolate the problem quickly w/o requiring you to go through a lengthy debug cycle.
Is it really recommended to have only one assert per unit test? Yes it is, there are people who make that recommendation. Are they right? I don't think so. I find it hard to believe such people have actually worked on real code for a long time.
So, imangine you have a mutator method you want to unit test. The mutator has some kind of effect, or effects, which you want to check. Typically the expected effect of a mutator are few in number, because many effects suggests an overly complicated design for the mutator. With one assert per effect and one test case per assert, you will not need many test cases per mutator, so the recommendation does not seem so bad.
But the flaw in this reasoning is that those tests are looking at only the expected effects of the mutator. But if the mutator has a bug in it, it might have unexpected faulty side effects. The tests are making the foolish assumption that the code does not have a whole class of bugs, and that no future refactoring will introduce such bugs. When the method was originally written it might be obvious to the author that particular side effects were impossible, but refactoring and addition of new functionality might make such side effects possible.
The only safe way to test long lived code is to check that the mutators do not have unexpected side effects. But how can you test for those? Most classes have some invariants: things that no mutator can ever change. The size method of a container will never return a negative value, for example. Each invariant is, in effect, a post condition for every mutator (and also the constructor). Each mutator also typically has a set of invariants that describe what kind of changes it does not make. A sort method does not change the length of the container, for example. The class and mutator invariants are, in effect, post conditions for every mutator call. Adding assertions for all them is the only way of checking for unexpected side effects.
So, just add more test cases? In practice the number of invariants multiplied by the number of mutators to test is large, so one assertion per test leads to many test cases. And the information about your invariants is scattered over many test cases. A design change to tweak one invariant will require alteration of many test cases. It becomes impractical. Its better to have parameterised test cases for a mutator, which check several invariants for the mutator, using several assertions.
And the authors of JUnit5 seem to agree. They provide an assertAll for checking several assertions in one test-case.
this construction help you to have 1 big assert (with small asserts inside)
import static org.junit.jupiter.api.Assertions.assertAll;
assertAll(
() -> assertThat(actual1, is(expected)),
() -> assertThat(actual2, is(expected))
);
You can have a parameterized test where the 1st parameter is the propertyname and the second the expected value.
Is that rule extended to being in a loop? Consider this
Collection expectedValues = // populate expected values
populateDTO();
for(DTO dto : myDtoContainer)
assert_equal(dto, expectedValues.get(someIndexRelatedToDto))
Now I'm not so big on the exact syntax, but this is just the notion I'm looking at.
EDIT:
After the comments...
The answer is ... Nope!
The reason the principle exists is so you can identify which parts of the object fail. If you have them in one method, you're going to run into only one assertion, then the next, then the next, and you won't see them all.
So you can have it one of two ways:
One method, less boilerplate code.
Many methods, better reporting on the test run
It's up to you, both have ups and downs.
3. List item
[caveat: I'm very "unfluent" in Java/JUnit, so beware of errors in the details below]
There's a couple of ways to do this:
1) Write multiple assertions in the same test. This should be ok if you are only testing the DTO generation once. You could start here, and move to another solution when this starts to hurt.
2) Write a helper assertion, e.g. assertDtoFieldsEqual, passing in the expected and actual DTO. Inside the helper assertion you assert each field separately. This at least gives you the illusion of only one assert per test and will make things clearer if you test DTO generation for multiple scenarios.
3) Implement equals for the object that check each property and implement toString so that you at least can inspect the assertion result manually to find out what part is incorrect.
4) For each scenario where the DTO is generated, create a separate test fixture that generates the DTO and initializes the expected properties in the setUp method. The create a separate test for testing each of the properties. This also results in a lot of tests, but they will at least be one-liners only. Example in pseudo-code:
public class WithDtoGeneratedFromXxx : TestFixture
{
DTO dto = null;
public void setUp()
{
dto = GenerateDtoFromXxx();
expectedProp1 = "";
...
}
void testProp1IsGeneratedCorrectly()
{
assertEqual(expectedProp1, dto.prop1);
}
...
}
If you need to test the DTO generation under different scenarios and choose this last method it could soon become tedious to write all those tests. If this is the case you could implement an abstract base fixture that leaves out the details on how to create the DTO and setup the expected properties to derived classes. Pseudo-code:
abstract class AbstractDtoTest : TestFixture
{
DTO dto;
SomeType expectedProp1;
abstract DTO createDto();
abstract SomeType getExpectedProp1();
void setUp()
{
dto = createDto();
...
}
void testProp1IsGeneratedCorrectly()
{
assertEqual(getExpectedProp1(), dto.prop1);
}
...
}
class WithDtoGeneratedFromXxx : AbstractDtoTest
{
DTO createDto() { return GenerateDtoFromXxx(); }
abstract SomeType getExpectedProp1() { return new SomeType(); }
...
}
Or you can do some workaround.
import junit.framework.Assert;
import org.junit.After;
import org.junit.AfterClass;
import org.junit.Before;
import org.junit.BeforeClass;
import org.junit.Test;
public class NewEmptyJUnitTest {
public NewEmptyJUnitTest() {
}
#BeforeClass
public static void setUpClass() throws Exception {
}
#AfterClass
public static void tearDownClass() throws Exception {
}
#Before
public void setUp() {
}
#After
public void tearDown() {
}
#Test
public void checkMultipleValues() {
String errMessages = new String();
try{
this.checkProperty1("someActualResult", "someExpectedResult");
} catch (Exception e){
errMessages += e.getMessage();
}
try{
this.checkProperty2("someActualResult", "someExpectedResult");
} catch (Exception e){
errMessages += e.getMessage();
}
try{
this.checkProperty3("someActualResult", "someExpectedResult");
} catch (Exception e){
errMessages += e.getMessage();
}
Assert.assertTrue(errMessages, errMessages.isEmpty());
}
private boolean checkProperty1(String propertyValue, String expectedvalue) throws Exception{
if(propertyValue == expectedvalue){
return true;
}else {
throw new Exception("Property1 has value: " + propertyValue + ", expected: " + expectedvalue);
}
}
private boolean checkProperty2(String propertyValue, String expectedvalue) throws Exception{
if(propertyValue == expectedvalue){
return true;
}else {
throw new Exception("Property2 has value: " + propertyValue + ", expected: " + expectedvalue);
}
}
private boolean checkProperty3(String propertyValue, String expectedvalue) throws Exception{
if(propertyValue == expectedvalue){
return true;
}else {
throw new Exception("Property3 has value: " + propertyValue + ", expected: " + expectedvalue);
}
}
}
Maybe not the best approach and if overused than can confuse... but it is a possibility.
I've been working on a Java application where I have to use JUnit for testing. I am learning it as I go. So far I find it to be useful, especially when used in conjunction with the Eclipse JUnit plugin.
After playing around a bit, I developed a consistent method for building my unit tests for functions with no return values. I wanted to share it here and ask others to comment. Do you have any suggested improvements or alternative ways to accomplish the same goal?
Common Return Values
First, there's an enumeration which is used to store values representing test outcomes.
public enum UnitTestReturnValues
{
noException,
unexpectedException
// etc...
}
Generalized Test
Let's say a unit test is being written for:
public class SomeClass
{
public void targetFunction (int x, int y)
{
// ...
}
}
The JUnit test class would be created:
import junit.framework.TestCase;
public class TestSomeClass extends TestCase
{
// ...
}
Within this class, I create a function which is used for every call to the target function being tested. It catches all exceptions and returns a message based on the outcome. For example:
public class TestSomeClass extends TestCase
{
private UnitTestReturnValues callTargetFunction (int x, int y)
{
UnitTestReturnValues outcome = UnitTestReturnValues.noException;
SomeClass testObj = new SomeClass ();
try
{
testObj.targetFunction (x, y);
}
catch (Exception e)
{
UnitTestReturnValues.unexpectedException;
}
return outcome;
}
}
JUnit Tests
Functions called by JUnit begin with a lowercase "test" in the function name, and they fail at the first failed assertion. To run multiple tests on the targetFunction above, it would be written as:
public class TestSomeClass extends TestCase
{
public void testTargetFunctionNegatives ()
{
assertEquals (
callTargetFunction (-1, -1),
UnitTestReturnValues.noException);
}
public void testTargetFunctionZeros ()
{
assertEquals (
callTargetFunction (0, 0),
UnitTestReturnValues.noException);
}
// and so on...
}
Please let me know if you have any suggestions or improvements. Keep in mind that I am in the process of learning how to use JUnit, so I'm sure there are existing tools available that might make this process easier. Thanks!
It is true that if you are using JUnit 3, and you are testing whether a particular exception is thrown or not thrown within a method, you will need to use something like the try-catch pattern you define above.
However:
1) I'd argue that there is a lot more to testing a method with a void return value then checking for exceptions: is your method making the correct calls to (presumably mocked) dependencies; does it behave differently when the class is initialized with a different context or different sets of dependencies, etc. By wrapping all calls to that method, you make it hard to change other aspects of your test.
I'm also generally opposed to adding code and adding complexity if it can be avoided; I don't think it's a burden to have to put a try/catch in a given test when it's checking for exceptions.
2) Switch to JUnit 4! It makes it easy to check for expected exceptions:
#Test(expected=IndexOutOfBoundsException.class)
public void testIndexOutOfBoundsException() {
ArrayList emptyList = new ArrayList();
Object o = emptyList.get(0);
}
If you have the possibility, you should upgrade to JUnit 4.x.
Then your first example can be rewritten to:
#Test(expected=RuntimeException.class)
public void testTargetFunction() {
testObj.targetFunction (x, y);
}
The advantage here is that you can remove you the private UnitTestReturnValues callTargetFunction (int x, int y) method and use JUnit's built in support for expecting exceptions.
You should also test for specific exceptions instead.
Looks like you reimplemented most of JUnit :) In general you don't need to do it. You just call the function you want to call and compare results. If it throws an exception, JUnit will catch if for you and fail the test. If you expect an exception, either you can use the explicit annotation if you are using JUnit 4, or you can use the following pattern:
public void testThrows()
{
try {
obj.DoSth(); //this should throw MyException
assertFail("Expected exception");
} catch (MyException e) {
//assert the message etc
}
}
again, if obj.DoSth() throws a different exception JUnit will fail the test.
So to sum up, I am afraid I believe your approach is overcomplicated, sorry.
please correct me if I am wrong. As I understood from the provided code you're only checking if there may be an exception while executing the function. But you're actually not verifying, if the called functions "works" correctly unless the only way to end in case of an error would be an exception. I suggest writing additional tests like this:
public void testTargetFunctionSomeValue() {
int someValue = 0;
callTargetFunction(someValue, someValue);
assertTrue(verifyTargetFunction(someValue, someValue));
}
public boolean verifyTargetFucntion(int someValue, int someValue) {
// verify that execution of targetFunction made expected changes.
. . . . .
}
and the verifyTargetFunction would acutally check, if calling targetFunction would have made the expected changes - let's say to a database table by returning true or false.
Hope that helps.
Cheers,
Markus