Say I have the following two functions:
add_five (number) -> number + 2
add_six (number) -> add_five(number) + 1
As you can see, add_five has a bug.
If I now test add_six it would fail because the result is incorrect, but the code is correct.
Imagine you have a large tree of functions calling each other, it would be hard to find out which function contains the bug, because all the functions will fail (and not only the one with the bug).
So my question is: should unit tests fail because of incorrect behaviour (wrong results) or because of incorrect code (bugs).
should unit tests fail because of incorrect behaviour (wrong results) or because of incorrect code (bugs)?
Unit tests usually fail because of wrong results. That's what you write in assertions: you call a method and you define the expected result.
Unit tests cannot identify incorrect code. If the operation is return number+5 and your CPU or your RAM has a hardware problem and return something different, then the test will fail as well, even if the code is correct.
Also consider:
public int add_five(int number)
{
Thread.Sleep(5000);
return number+5;
}
How shall the unit test know whether the Sleep is intended or not?
So, if any unit test fails, it's your job to look at it, find out why it fails and if it fails in a different method, write a new unit test for that method so you can exclude that method next time.
Presumably you have a test for add_five/1 and a test for add_six/1. In this case, the test for add_six/1 will fail alongside the test for add_five/1.
Let's assume you decide to check add_six/1 first. You see that it depends on add_five/1, which is also failing. You can immediately assume that the bug in add_five/1 is cascading up.
Your module dependencies form a directed (hopefully acyclic) graph. If a dependency of you function or module is broken, that should be what you target for debugging first.
Another option is to mock out the add_five function when testing your add_sixfunction, but this quickly creates a lot of extra typing and logic duplication. If you change the spec of add_five, you have to change every place you reimplemented it as a mock.
If you use a quickcheck-style testing library, you can test for certain logic errors based on the properties of what you're testing. These bugs are detected using randomly generated cases that produce incorrect results, but all you as a tester write are library-specific definitions of the properties you're testing for. However, this will also suffer from dependency breakages unless you've mocked out dependent modules/functions.
I need to test code that fetches and processes some data from standard C library. However, this data may not be installed at any particular system. So at the start of each test I detect whether the data is installed and skip the test.
It's easy to wrap the test in an if, but since the code that tests whether the test is possible may itself fail, I need to at least know that the tests were skipped. I know I can't simply use println! and I don't want to have to remember to pass --nocapture every time I test (and I want to see the warnings in the Travis log; Travis virtuals don't have all the data).
A lesser-known fact is that println! uses the thread-local "standard out", which can be set to other output types. During tests, this is set to a capturing type. However, you can still get direct access to the true standard out and use that:
use std::io::{self,Write};
#[test]
fn a() {
println!("Hi!"); // Will be captured / hidden by default
write!(&mut io::stdout(), "Hello!").unwrap(); // Will be shown
}
Has output like:
running 1 test
Hello!test a ... ok
Suppose I am writing a template library consisting of a function template
template<T> void f(T);
with the requirement that it works with a predefined set of classes A, B, C, and D, e.g., the following must compile:
template<> void f(A);
template<> void f(B);
template<> void f(C);
template<> void f(D);
Which test framework can I use to write test cases that captures this requirement at runtime instead of failing at compilation of the test code? In another word, I would like the framework to instantiate the templates at runtime and produce a nicely formatted error report if a subset of them fails.
I know I can forego test frameworks altogether and simply write a simple cc file containing the 4 lines above. But I was hoping I could incorporate this requirement into regular, standard test cases for generation of test status reports. For example,
test f works with A: passed.
test f works with B: passed.
test f works with C: failed! Cannot cast type C!
test f works with D: passed.
3 of 4 tests passed.
1 of 4 tests failed.
Write a test case that spawns the compiler... that's how e.g. autoconf tests for existence of features.
I don't understand why failing at runtime is preferable to failing at compile time. The earlier you fail in the unit testing process the better. It is preferable to have your unit tests not compile than fail. Its even easier to fix, In fact it probably won't even be committed to source control. Your unit test should just include those four lines and assert true at the end. Note this isn't the way I would go about doing it myself.
C++ templates are a compile time feature. In many cases they will fail at compile time, by design. You simply can't get around this without doing something really crazy.
However, you're also going to want to know that your template specializations are correct, because the specializations override the behavior you would otherwise get from the template. So test the specializations. But realize you will never get around the compile time aspects of templates.
Based on what you're trying to test here, checking if the thing can compile is the only sensible test you can perform.
Testing should not be for the sake of testing, but to ensure functional correctness. If you want to have proper tests around your class, you should write tests that verify the functionality of your template with all of the 4 different classes it can be compiled with.
I'm trying to write a test to verify a compiling error. It's about assigning a number to a String type property. But since it's a compiling error, the unit test code won't even be compiled at the first place. So anyone have any suggestion on how to do this?
I'm thinking maybe I can assign the number in runtime and check to see if certain exception got thrown? But I'm not sure how exactly to do that.
Thanks in advance!
If I understand you correctly, you have some piece of code that may not compile and you want to write the unit test that fails if the code really doesn't compile. If that is the case, then you should not write any unit tests. You should understand that you should write unit tests only for your code, not the code that somebody else wrote.
You didn't specify the programming language, so I will do some pseudo-code. Say you are writing a function to add two numbers:
function add(a, b) {
return a + b;
}
Very simple. You should unit test this, e.g. by making tests like the below:
function testAdd() {
assertEquals(4, add(2, 2));
assertEquals(46, add(12, 34));
}
However, you should not write a test that tests whether + operator is working fine. This is the job of whoever wrote the library that implements how + operator really works.
So, if that's the case, don't write any unit tests. Compiling your code is job of your compiler. The compiler should report that there is a compilation error in a proper way. You should not test whether compiler does its job correctly - testing that is the job of the people who are writing the compiler.
You did not specify the language
Ruby, Python -- dynamic & strong typesystem. This means that the types deduced during runtime (dynamic), but implicit conversions between types are prohibited
Js, Perl -- dynamic & weak typesystem.
C++ -- static and strong typesystem.
Let's assume we are talking about C++. Moreover I can create more really example. Imagine that you implement static assert for your project which doesn't not use c++11 compiler
template <bool>
struct my_static_assert;
template <>
struct my_static_assert<true> {};
If you want to check that such mechanism works ok. You should create unittest function which do the follow steps:
Create some file to compiler
Create external process of the compiler and pass test compile unit to it
Wait for compiler process completion
Retrive return code from the compiler process
Your function check return code from 4.
I checked google-test guide but it seems that they doesn't support such concept https://github.com/google/googletest/blob/master/docs/advanced.md
Regarding the classic test pattern of Arrange-Act-Assert, I frequently find myself adding a counter-assertion that precedes Act. This way I know that the passing assertion is really passing as the result of the action.
I think of it as analogous to the red in red-green-refactor, where only if I've seen the red bar in the course of my testing do I know that the green bar means I've written code that makes a difference. If I write a passing test, then any code will satisfy it; similarly, with respect to Arrange-Assert-Act-Assert, if my first assertion fails, I know that any Act would have passed the final Assert - so that it wasn't actually verifying anything about the Act.
Do your tests follow this pattern? Why or why not?
Update Clarification: the initial assertion is essentially the opposite of the final assertion. It's not an assertion that Arrange worked; it's an assertion that Act hasn't yet worked.
This is not the most common thing to do, but still common enough to have its own name. This technique is called Guard Assertion. You can find a detailed description of it on page 490 in the excellent book xUnit Test Patterns by Gerard Meszaros (highly recommended).
Normally, I don't use this pattern myself, since I find it more correct to write a specific test that validates whatever precondition I feel the need to ensure. Such a test should always fail if the precondition fails, and this means that I don't need it embedded in all the other tests. This gives a better isolation of concerns, since one test case only verifies one thing.
There may be many preconditions that need to be satisfied for a given test case, so you may need more than one Guard Assertion. Instead of repeating those in all tests, having one (and one only) test for each precondition keeps your test code more mantainable, since you will have less repetition that way.
It could also be specified as Arrange-Assume-Act-Assert.
There is a technical handle for this in NUnit, as in the example here:
http://nunit.org/index.php?p=theory&r=2.5.7
Here's an example.
public void testEncompass() throws Exception {
Range range = new Range(0, 5);
assertFalse(range.includes(7));
range.encompass(7);
assertTrue(range.includes(7));
}
It could be that I wrote Range.includes() to simply return true. I didn't, but I can imagine that I might have. Or I could have written it wrong in any number of other ways. I would hope and expect that with TDD I actually got it right - that includes() just works - but maybe I didn't. So the first assertion is a sanity check, to ensure that the second assertion is really meaningful.
Read by itself, assertTrue(range.includes(7)); is saying: "assert that the modified range includes 7". Read in the context of the first assertion, it's saying: "assert that invoking encompass() causes it to include 7. And since encompass is the unit we're testing, I think that's of some (small) value.
I'm accepting my own answer; a lot of the others misconstrued my question to be about testing the setup. I think this is slightly different.
An Arrange-Assert-Act-Assert test can always be refactored into two tests:
1. Arrange-Assert
and
2. Arrange-Act-Assert
The first test will only assert on that which was set up in the Arrange phase, and the second test will only assert for that which happened in the Act phase.
This has the benefit of giving more precise feedback on whether it's the Arrange or the Act phase that failed, while in the original Arrange-Assert-Act-Assert these are conflated and you would have to dig deeper and examine exactly what assertion failed and why it failed in order to know if it was the Arrange or Act that failed.
It also satisfies the intention of unit testing better, as you are separating your test into smaller independent units.
I am now doing this. A-A-A-A of a different kind
Arrange - setup
Act - what is being tested
Assemble - what is optionally needed to perform the assert
Assert - the actual assertions
Example of an update test:
Arrange:
New object as NewObject
Set properties of NewObject
Save the NewObject
Read the object as ReadObject
Act:
Change the ReadObject
Save the ReadObject
Assemble:
Read the object as ReadUpdated
Assert:
Compare ReadUpdated with ReadObject properties
The reason is so that the ACT does not contain the reading of the ReadUpdated is because it is not part of the act. The act is only changing and saving. So really, ARRANGE ReadUpdated for assertion, I am calling ASSEMBLE for assertion. This is to prevent confusing the ARRANGE section
ASSERT should only contain assertions. That leaves ASSEMBLE between ACT and ASSERT which sets up the assert.
Lastly, if you are failing in the Arrange, your tests are not correct because you should have other tests to prevent/find these trivial bugs. Because for the scenario i present, there should already be other tests which test READ and CREATE. If you create a "Guard Assertion", you may be breaking DRY and creating maintenance.
I don't use that pattern, because I think doing something like:
Arrange
Assert-Not
Act
Assert
May be pointless, because supposedly you know your Arrange part works correctly, which means that whatever is in the Arrange part must be tested aswell or be simple enough to not need tests.
Using your answer's example:
public void testEncompass() throws Exception {
Range range = new Range(0, 5);
assertFalse(range.includes(7)); // <-- Pointless and against DRY if there
// are unit tests for Range(int, int)
range.encompass(7);
assertTrue(range.includes(7));
}
Tossing in a "sanity check" assertion to verify state before you perform the action you're testing is an old technique. I usually write them as test scaffolding to prove to myself that the test does what I expect, and remove them later to avoid cluttering tests with test scaffolding. Sometimes, leaving the scaffolding in helps the test serve as narrative.
I've already read about this technique - possibly from you btw - but I do not use it; mostly because I'm used to the triple A form for my unit tests.
Now, I'm getting curious, and have some questions: how do you write your test, do you cause this assertion to fail, following a red-green-red-green-refactor cycle, or do you add it afterwards ?
Do you fail sometimes, perhaps after you refactor the code ? What does this tell you ? Perhaps you could share an example where it helped. Thanks.
I have done this before when investigating a test that failed.
After considerable head scratching, I determined that the cause was the methods called during "Arrange" were not working correctly. The test failure was misleading. I added a Assert after the arrange. This made the test fail in a place which highlighted the actual problem.
I think there is also a code smell here if the Arrange part of the test is too long and complicated.
In general, I like "Arrange, Act, Assert" very much and use it as my personal standard. The one thing it fails to remind me to do, however, is to dis-arrange what I have arranged when the assertions are done. In most cases, this doesn't cause much annoyance, as most things auto-magically go away via garbage collection, etc. If you have established connections to external resources, however, you will probably want to close those connections when you're done with your assertions or you many have a server or expensive resource out there somewhere holding on to connections or vital resources that it should be able to give away to someone else. This is particularly important if you're one of those developers who does not use TearDown or TestFixtureTearDown to clean up after one or more tests. Of course, "Arrange, Act, Assert" is not responsible for my failure to close what I open; I only mention this "gotcha" because I have not yet found a good "A-word" synonym for "dispose" to recommend! Any suggestions?
Have a look at Wikipedia's entry on Design by Contract. The Arrange-Act-Assert holy trinity is an attempt to encode some of the same concepts and is about proving program correctness. From the article:
The notion of a contract extends down to the method/procedure level; the
contract for each method will normally contain the following pieces of
information:
Acceptable and unacceptable input values or types, and their meanings
Return values or types, and their meanings
Error and exception condition values or types that can occur, and their meanings
Side effects
Preconditions
Postconditions
Invariants
(more rarely) Performance guarantees, e.g. for time or space used
There is a tradeoff between the amount of effort spent on setting this up and the value it adds. A-A-A is a useful reminder for the minimum steps required but shouldn't discourage anyone from creating additional steps.
Depends on your testing environment/language, but usually if something in the Arrange part fails, an exception is thrown and the test fails displaying it instead of starting the Act part. So no, I usually don't use a second Assert part.
Also, in the case that your Arrange part is quite complex and doesn't always throw an exception, you might perhaps consider wrapping it inside some method and writing an own test for it, so you can be sure it won't fail (without throwing an exception).
If you really want to test everything in the example, try more tests... like:
public void testIncludes7() throws Exception {
Range range = new Range(0, 5);
assertFalse(range.includes(7));
}
public void testIncludes5() throws Exception {
Range range = new Range(0, 5);
assertTrue(range.includes(5));
}
public void testIncludes0() throws Exception {
Range range = new Range(0, 5);
assertTrue(range.includes(0));
}
public void testEncompassInc7() throws Exception {
Range range = new Range(0, 5);
range.encompass(7);
assertTrue(range.includes(7));
}
public void testEncompassInc5() throws Exception {
Range range = new Range(0, 5);
range.encompass(7);
assertTrue(range.includes(5));
}
public void testEncompassInc0() throws Exception {
Range range = new Range(0, 5);
range.encompass(7);
assertTrue(range.includes(0));
}
Because otherwise you are missing so many possibilities for error... eg after encompass, the range only inlcudes 7, etc...
There are also tests for length of range (to ensure it didn't also encompass a random value), and another set of tests entirely for trying to encompass 5 in the range... what would we expect - an exception in encompass, or the range to be unaltered?
Anyway, the point is if there are any assumptions in the act that you want to test, put them in their own test, yes?
I use:
1. Setup
2. Act
3. Assert
4. Teardown
Because a clean setup is very important.