I'm writing a Go package that needs to interact with the Linux command line. Since the CLI tools that my code is interacting with are Linux specific, I first need to make sure that the OS is Linux, and then I can check for the presence of the CLI tools.
My function to test the OS is as follows:
func isLinux() bool {
return runtime.GOOS == "linux"
}
Realistically, the statement within the function can only return a boolean value. The result of the boolean being true or false seems to be wholly dependent on the testing environment too - so unless I run the tests across different OSes, I have to just assume that the function would return false in a Windows environment.
Is there a way for me to temporarily set the value of runtime.GOOS for the purpose of testing?
Would passing runtime.GOOS to the function as a string parameter be worthwhile? It would allow me to test the function more easily (because I could pass in string literals for different OSes), but since the function is so specific, and the snippet is so small, I'm not sure whether parameterising it would be worthwhile.
EDIT: It's more of a hypothetical question, I'm not really sure whether it's even worth testing given the size of the function. But, when I write code that's reliant on the current OS, how should I write it to facilitate testing - should I always to take the OS as a parameter so that I can easily mock it, or is there a way for me to temporarily override the value of runtime.GOOS?
How should I handle testing of a Go function that uses runtime.GOOS?
Either not at all or on a machine with the appropriate OS.
It really is that simple.
I have a class A where I deleted the default constructor.
class A {
public:
A() = delete;
A(int a): m_myInt(a) {}
private:
const int m_myInt;
};
int main () {
A foo(1); // works perfect
A bar; // won't compile
}
How do I write a good unit test ensuring that A bar; remains not valid? I could write a not compiling test and take the compile error as a test requirement. I wonder, if there is a better way to write a unit test?
2) If std::is_trivially_constructible<T>::value is true, provides the member constant value equal to true, otherwise value is false.
http://en.cppreference.com/w/cpp/types/is_default_constructible
There is nothing wrong with a unit test that hinges on confirming that some code constructs fail to compile. The purpose of unit testing is determining if one or more sections of code are collectively "fit for use" - and the criteria can represent functional or non-functional (e.g. quality) attributes. If a "fit for use" criterion is "code will not compile if ....." then an obvious approach is to write a unit test that deliberately seeks to cause a failed compilation.
This is certainly a valid unit testing approach for a compiler - in which testing how the compiler responds to samples of bad code is completely appropriate as a unit test, or set of unit tests.
Given that the requirement is that default-instantiation of an A (to use the OP's words) "remains not valid", the basic criterion for a passed unit test would be a failed compilation of code which performs
A bar;
Depending on requirement, such code may need to be tested in different contexts (e.g. within an instrumented function that is a member or friend of A, in a unrelated function, at file scope, etc). So this requirement may require a set of unit tests.
To implement such a test, it would be necessary for the build process to capture whether compilation fails. A successful compilation would need to be captured as a failed test and, conversely, a failed compilation would need to be captured as a passed unit test.
Depending on how much evidence needs to be captured to justify a claim of a passed (or failed) unit test, the build process may need to capture and parse the compiler diagnostics - for example, to determine which lines in a source file actually cause compilation to fail.
Equally, there would probably need to be a unit test, or set of unit tests, that check some code constructs do compile.
You might use traits for that:
static_assert(!std::is_constructible<A>::value, "!");
Say I have the following two functions:
add_five (number) -> number + 2
add_six (number) -> add_five(number) + 1
As you can see, add_five has a bug.
If I now test add_six it would fail because the result is incorrect, but the code is correct.
Imagine you have a large tree of functions calling each other, it would be hard to find out which function contains the bug, because all the functions will fail (and not only the one with the bug).
So my question is: should unit tests fail because of incorrect behaviour (wrong results) or because of incorrect code (bugs).
should unit tests fail because of incorrect behaviour (wrong results) or because of incorrect code (bugs)?
Unit tests usually fail because of wrong results. That's what you write in assertions: you call a method and you define the expected result.
Unit tests cannot identify incorrect code. If the operation is return number+5 and your CPU or your RAM has a hardware problem and return something different, then the test will fail as well, even if the code is correct.
Also consider:
public int add_five(int number)
{
Thread.Sleep(5000);
return number+5;
}
How shall the unit test know whether the Sleep is intended or not?
So, if any unit test fails, it's your job to look at it, find out why it fails and if it fails in a different method, write a new unit test for that method so you can exclude that method next time.
Presumably you have a test for add_five/1 and a test for add_six/1. In this case, the test for add_six/1 will fail alongside the test for add_five/1.
Let's assume you decide to check add_six/1 first. You see that it depends on add_five/1, which is also failing. You can immediately assume that the bug in add_five/1 is cascading up.
Your module dependencies form a directed (hopefully acyclic) graph. If a dependency of you function or module is broken, that should be what you target for debugging first.
Another option is to mock out the add_five function when testing your add_sixfunction, but this quickly creates a lot of extra typing and logic duplication. If you change the spec of add_five, you have to change every place you reimplemented it as a mock.
If you use a quickcheck-style testing library, you can test for certain logic errors based on the properties of what you're testing. These bugs are detected using randomly generated cases that produce incorrect results, but all you as a tester write are library-specific definitions of the properties you're testing for. However, this will also suffer from dependency breakages unless you've mocked out dependent modules/functions.
I've got a problem when unit testing my program.
The problem is simple but i'm not sure why this is not working.
1 -> i build all my program
2 -> i build my unitTest
3 -> the test is running.
All is ok when it is not about getting global data from the data segment. It seems as if the variable are not initialized / or simply found. So of course all my tests become wrong.
My question is:
Is it totally wrong to build an executable, then running the test on it? Or should i must compile all my code + the unit test in the same time, and then running it? Or is it just a lack of SenTesting framework?
I forgot to mention that this is a C++ const string. Dunno if that change something.
*EDIT***
My assumption was wrong, but i still don't understand the magic beyond! Seems a C++ magic hoydi hoo?
char cstring[] = "***";
std::string cppString = "***";
NSString* nstring = #"***";
- (void)testSync{
STAssertNotNil(nstring, nil); // fine
STAssertNotNil((id)strlen(bbb), nil); // fine
STAssertNotNil((id)cppString.size(), nil); // failed
}
EDIT 2**
Actually this is normal that the C++ is not initialized at this part of the code. If i do a nm on my executable, it appears that my C and Obj-C global are put into the dataSegment. I thought my C++ string was in the same case, but it is actually put into the bss segment. That's means it's uninitialized. The fact is the C++ compiler do some magic beyond and the C++ string is initialized after the main() call and act like if it were into the dataSegment.
I didn't know that testSuit doesn't have main() call, so the C++ object are never initialized. There is some technique in order to call the .ctor before the testSuit. But i am too lazy too explain and it's some kind of topic. I have just replaced my C++ string with a simple char array, and it work perfectly since my value are now POD.
By the way there is no devil in global variable if they are just read-only. ;)
OK, I can see a few faults here.
First of all, this code gives errors on my environment (Xcode 5) and for good reasons (with ARC enabled). I don't know how you got the thing to compile. The reason is that you are casting an integer (or long) to an object, and this will result in many errors, as it is normally an invalid operation. So, the real question is not why the third "assert" failed, but why the second one succeeded.
As far as the second part of your question is concerned, I have to admit that I do not completely understand your question, and you may have to explain it more thoroughly.
In general, unit testing is testing specific parts of your code. Therefore, you typically don't perform the tests on an actual final executable (this is not called unit testing, I believe), nor do you have to compile "all your c++ code + your unit tests at the same time".
Since you are using Xcode, I will give you some indications.
Write your application (or at least a part of it), and find the aspects / functions / objects you want to perform unit tests on.
In separate files, write unit tests that instantiate these objects and test their methods, call them and compare the inputs and outputs.
You should have a second target in your application, that will compile only the unit test source code and the relevant main program code.
Build this target, or press command-U and it will report successes and failures.
So, you need to separate your source code and isolate your classes / methods to make them testable like this. This needs a good architecture and design of the application on your part, and you may need to make some compromises in flexibility (that is up to you to decide). Oh, and I believe that in a testable code you should avoid global variables in general, for various reasons. Global variables are helpful sometimes, but they generally make unit testing really difficult, (and if misused may lead to spaghetti code, but this is a whole different story)
I hope I helped, even without fully understanding the second part of your post.
Regarding the classic test pattern of Arrange-Act-Assert, I frequently find myself adding a counter-assertion that precedes Act. This way I know that the passing assertion is really passing as the result of the action.
I think of it as analogous to the red in red-green-refactor, where only if I've seen the red bar in the course of my testing do I know that the green bar means I've written code that makes a difference. If I write a passing test, then any code will satisfy it; similarly, with respect to Arrange-Assert-Act-Assert, if my first assertion fails, I know that any Act would have passed the final Assert - so that it wasn't actually verifying anything about the Act.
Do your tests follow this pattern? Why or why not?
Update Clarification: the initial assertion is essentially the opposite of the final assertion. It's not an assertion that Arrange worked; it's an assertion that Act hasn't yet worked.
This is not the most common thing to do, but still common enough to have its own name. This technique is called Guard Assertion. You can find a detailed description of it on page 490 in the excellent book xUnit Test Patterns by Gerard Meszaros (highly recommended).
Normally, I don't use this pattern myself, since I find it more correct to write a specific test that validates whatever precondition I feel the need to ensure. Such a test should always fail if the precondition fails, and this means that I don't need it embedded in all the other tests. This gives a better isolation of concerns, since one test case only verifies one thing.
There may be many preconditions that need to be satisfied for a given test case, so you may need more than one Guard Assertion. Instead of repeating those in all tests, having one (and one only) test for each precondition keeps your test code more mantainable, since you will have less repetition that way.
It could also be specified as Arrange-Assume-Act-Assert.
There is a technical handle for this in NUnit, as in the example here:
http://nunit.org/index.php?p=theory&r=2.5.7
Here's an example.
public void testEncompass() throws Exception {
Range range = new Range(0, 5);
assertFalse(range.includes(7));
range.encompass(7);
assertTrue(range.includes(7));
}
It could be that I wrote Range.includes() to simply return true. I didn't, but I can imagine that I might have. Or I could have written it wrong in any number of other ways. I would hope and expect that with TDD I actually got it right - that includes() just works - but maybe I didn't. So the first assertion is a sanity check, to ensure that the second assertion is really meaningful.
Read by itself, assertTrue(range.includes(7)); is saying: "assert that the modified range includes 7". Read in the context of the first assertion, it's saying: "assert that invoking encompass() causes it to include 7. And since encompass is the unit we're testing, I think that's of some (small) value.
I'm accepting my own answer; a lot of the others misconstrued my question to be about testing the setup. I think this is slightly different.
An Arrange-Assert-Act-Assert test can always be refactored into two tests:
1. Arrange-Assert
and
2. Arrange-Act-Assert
The first test will only assert on that which was set up in the Arrange phase, and the second test will only assert for that which happened in the Act phase.
This has the benefit of giving more precise feedback on whether it's the Arrange or the Act phase that failed, while in the original Arrange-Assert-Act-Assert these are conflated and you would have to dig deeper and examine exactly what assertion failed and why it failed in order to know if it was the Arrange or Act that failed.
It also satisfies the intention of unit testing better, as you are separating your test into smaller independent units.
I am now doing this. A-A-A-A of a different kind
Arrange - setup
Act - what is being tested
Assemble - what is optionally needed to perform the assert
Assert - the actual assertions
Example of an update test:
Arrange:
New object as NewObject
Set properties of NewObject
Save the NewObject
Read the object as ReadObject
Act:
Change the ReadObject
Save the ReadObject
Assemble:
Read the object as ReadUpdated
Assert:
Compare ReadUpdated with ReadObject properties
The reason is so that the ACT does not contain the reading of the ReadUpdated is because it is not part of the act. The act is only changing and saving. So really, ARRANGE ReadUpdated for assertion, I am calling ASSEMBLE for assertion. This is to prevent confusing the ARRANGE section
ASSERT should only contain assertions. That leaves ASSEMBLE between ACT and ASSERT which sets up the assert.
Lastly, if you are failing in the Arrange, your tests are not correct because you should have other tests to prevent/find these trivial bugs. Because for the scenario i present, there should already be other tests which test READ and CREATE. If you create a "Guard Assertion", you may be breaking DRY and creating maintenance.
I don't use that pattern, because I think doing something like:
Arrange
Assert-Not
Act
Assert
May be pointless, because supposedly you know your Arrange part works correctly, which means that whatever is in the Arrange part must be tested aswell or be simple enough to not need tests.
Using your answer's example:
public void testEncompass() throws Exception {
Range range = new Range(0, 5);
assertFalse(range.includes(7)); // <-- Pointless and against DRY if there
// are unit tests for Range(int, int)
range.encompass(7);
assertTrue(range.includes(7));
}
Tossing in a "sanity check" assertion to verify state before you perform the action you're testing is an old technique. I usually write them as test scaffolding to prove to myself that the test does what I expect, and remove them later to avoid cluttering tests with test scaffolding. Sometimes, leaving the scaffolding in helps the test serve as narrative.
I've already read about this technique - possibly from you btw - but I do not use it; mostly because I'm used to the triple A form for my unit tests.
Now, I'm getting curious, and have some questions: how do you write your test, do you cause this assertion to fail, following a red-green-red-green-refactor cycle, or do you add it afterwards ?
Do you fail sometimes, perhaps after you refactor the code ? What does this tell you ? Perhaps you could share an example where it helped. Thanks.
I have done this before when investigating a test that failed.
After considerable head scratching, I determined that the cause was the methods called during "Arrange" were not working correctly. The test failure was misleading. I added a Assert after the arrange. This made the test fail in a place which highlighted the actual problem.
I think there is also a code smell here if the Arrange part of the test is too long and complicated.
In general, I like "Arrange, Act, Assert" very much and use it as my personal standard. The one thing it fails to remind me to do, however, is to dis-arrange what I have arranged when the assertions are done. In most cases, this doesn't cause much annoyance, as most things auto-magically go away via garbage collection, etc. If you have established connections to external resources, however, you will probably want to close those connections when you're done with your assertions or you many have a server or expensive resource out there somewhere holding on to connections or vital resources that it should be able to give away to someone else. This is particularly important if you're one of those developers who does not use TearDown or TestFixtureTearDown to clean up after one or more tests. Of course, "Arrange, Act, Assert" is not responsible for my failure to close what I open; I only mention this "gotcha" because I have not yet found a good "A-word" synonym for "dispose" to recommend! Any suggestions?
Have a look at Wikipedia's entry on Design by Contract. The Arrange-Act-Assert holy trinity is an attempt to encode some of the same concepts and is about proving program correctness. From the article:
The notion of a contract extends down to the method/procedure level; the
contract for each method will normally contain the following pieces of
information:
Acceptable and unacceptable input values or types, and their meanings
Return values or types, and their meanings
Error and exception condition values or types that can occur, and their meanings
Side effects
Preconditions
Postconditions
Invariants
(more rarely) Performance guarantees, e.g. for time or space used
There is a tradeoff between the amount of effort spent on setting this up and the value it adds. A-A-A is a useful reminder for the minimum steps required but shouldn't discourage anyone from creating additional steps.
Depends on your testing environment/language, but usually if something in the Arrange part fails, an exception is thrown and the test fails displaying it instead of starting the Act part. So no, I usually don't use a second Assert part.
Also, in the case that your Arrange part is quite complex and doesn't always throw an exception, you might perhaps consider wrapping it inside some method and writing an own test for it, so you can be sure it won't fail (without throwing an exception).
If you really want to test everything in the example, try more tests... like:
public void testIncludes7() throws Exception {
Range range = new Range(0, 5);
assertFalse(range.includes(7));
}
public void testIncludes5() throws Exception {
Range range = new Range(0, 5);
assertTrue(range.includes(5));
}
public void testIncludes0() throws Exception {
Range range = new Range(0, 5);
assertTrue(range.includes(0));
}
public void testEncompassInc7() throws Exception {
Range range = new Range(0, 5);
range.encompass(7);
assertTrue(range.includes(7));
}
public void testEncompassInc5() throws Exception {
Range range = new Range(0, 5);
range.encompass(7);
assertTrue(range.includes(5));
}
public void testEncompassInc0() throws Exception {
Range range = new Range(0, 5);
range.encompass(7);
assertTrue(range.includes(0));
}
Because otherwise you are missing so many possibilities for error... eg after encompass, the range only inlcudes 7, etc...
There are also tests for length of range (to ensure it didn't also encompass a random value), and another set of tests entirely for trying to encompass 5 in the range... what would we expect - an exception in encompass, or the range to be unaltered?
Anyway, the point is if there are any assumptions in the act that you want to test, put them in their own test, yes?
I use:
1. Setup
2. Act
3. Assert
4. Teardown
Because a clean setup is very important.