I think this is poorly documented (if not a bug).
Dart first completes the code in main(), then the tests in functions test().
This code explains:
import 'package:test/test.dart';
void main() {
test('',() {print('1');});
print('2');
test('',() {print('3');});
}
Output: 213
Is there a way to return an error or suppress the execution of spurious code not pertaining to tests? Or maybe a way to have a lint warning from the analyzer?
Related
If you have 100% test coverage and all tests pass does that mean that the code is guaranteed to be correct and writing more tests is pointless?
It's only correct as far as the logic of your testing is correct.
I'll give the most stupid example possible...
If I for example have a class (Java):
public class Example {
public int timesTwo(int x){
return x*2;
}
}
with the following test (which you can see it being illogical and useless)
public class ExampleTest {
#Test
public void timesTwo() {
new Example().timesTwo(5);
assertTrue(true);
}
}
Most coverage tools will say that this class has been tested 100%!
So no, coverage is a good indicator of what needs to be tested; but the test logic itself isn't assured.
If you have 100% test coverage and all tests pass does that mean that the code is guaranteed to be correct and writing more tests is pointless?
No. Counter-example: For a division-function a testcase with 0 as divisor can be missing (example taken from here: https://stackoverflow.com/a/555824/3905529 ). The tests verify that the function is working with the inputs used in the unit-tests. But there may be edge-cases where the code throws exceptions and there is maybe no unit-test which asserts exactly this, even with a coverage of 100%.
Another problem I have often seen is that for a condition the if-block will be handled and tested but else-block is missing and there is also no test that verifies that the else-case will be handled in the function. This is else-part may be missing completely. So the test is green, but the function has semantical flaws due to missing code. But the coverage can still be 100% because there are only tests for the existing code, but not for the missing parts of it.
Beside this there may be other issues, see:
https://sqa.stackexchange.com/a/36940
I have a following function which takes a reference parameter:
#include <iostream>
class A { static void TestA(const int &y) };
void A::TestA(const int &y) { std::cout << y; }
int main()
{
A::TestA(2);
return 0;
}
In my (lcov) code coverage with google unit tests, it is saying missing a branch with TestA() function, and symbols list have a stack_chk_fail symbol added. If I change the function parameter to non-reference then coverage is 100%.
I am using g++ compiler.
Am I missing anything ?
Thanks
The compiler inlines Test into main (because that's what a good compiler does). However, it also has to create code for Test because it has external linkage. Effectively, the code for the function exists twice: Once inlined into main and once in the code for Test that the linker can link with other compilation units.
If your compiler is bad with code attribution (in the debug symbols) for inlined functions (hello MSVC?) then your profiler will give you exactly the result you see: Executing the program does not lead to coverage for Test because no piece of the binary that is executed (i.e. main) has any line attribution into Test.
Changing the parameter type may affect inlining, but it's more likely that it changes how debug symbols are generated.
To verify this, step through the program with a debugger, with a breakpoint in Test. If that breakpoint is not hit when running main, your coverage tool won't see coverage for that line either. Or, if you really want, look into the debug symbols manually to see which lines have attribution. In Visual Studio you can also look at the disassembly while debugging (it will show the associated code lines).
For the above reasons you will generally get more reliable coverage results if you do coverage runs with debug builds (where e.g. inlining won't happen).
Here it's discussed how to catch failing assert, e.g. you setup your fixture so that assert() fails and you see nice output. But what I need is the opposite. I want to test that assert() succeeds. But in case it fails I want to have nice output. At that point it just terminates when it snags on assert().
#define LIMIT 5
struct Obj {
int getIndex(int index) {
assert(index < LIMIT);
// do stuff;
}
}
Obj obj;
TEST(Fails_whenOutOfRange) {
ASSERT_DEATH(obj->getIndex(6), "");
}
TEST(Succeeds_whenInRange) {
obj->getIndex(4);
}
Above is contrived example. I want second test not to terminate in case it fails, for example if I set LIMIT to 3. After all, ASSERT_DEATH suppresses somehow termination when assert() fails.
You should try using the command line option --gtest_break_on_failure
It is meant to run tests within a debugger, so you get a breakpoint upon test failure. If you don't use a debugger you'll just get a SEGFAULT and execution will stop.
The following is just my opinion, but it seems for me that you are either testing a wrong thing, or using a wrong tool.
Assert (C assert()) is not for verifying input, it is for catching impossible situations. It will disappear from release code, for example, so you can't rely on it.
What you should test is your function specification rather than implementation. And you should decide, what is your specification for invalid input values:
Undefined behavior, so assert is fine, but you can't test it with unit-test, because undefined behavior is, well, undefined.
Defined behavior. Then you should be consistent regardless of NDEBUG presence. And throwing exception, in my opinion, is the right thing to do here, instead of calling std::abort, which is almost useless for user (can't be intercepted and processed properly).
If assert triggers (fails) you get "nice output" (or a crash or whatever assert does in your environment). If assert does not trigger then nothing happens and execution continues.
What more do you need to know?
This (hack) adds a EXPECT_NODEATH macro to Google Test. It is the "opposite" of EXPECT_DEATH in that it will pass if the statement does not assert, abort, or otherwise fail.
The general idea was simple, but I did not take the time to make the error messages any nicer. I tried to leave Google Test as untouched as possible and just piggy-back on what is already there. You should be able to include this without any side effects to the rest of Google Test
For your case:
TEST(Succeeds_whenInRange) {
EXPECT_NODEATH(obj->getIndex(4), "");
}
GTestNoDeath.h
In the developer code, there are many places where it calls assert(xyz):
(from assert.h)
#define assert(_Expression) (void)( (!!(_Expression)) || (_wassert(_CRT_WIDE(#_Expression), _CRT_WIDE(__FILE__), __LINE__), 0) )
When I run my tests through gtest and one of these asserts fails, then my executable completely shuts down.
I want a way for gtest to just catch this assert, fail the test, and the continue execution. Is this possible?
As from google test's reference documentation
How to Write a Death Test
Google Test has the following macros to support death tests:
where statement is a statement that is expected to cause the process to die, predicate is a function or function object that evaluates an integer exit status, and regex is a regular expression that the stderr output of statement is expected to match. Note that statement can be any valid statement (including compound statement) and doesn't have to be an expression.
You can use these test macros to intercept native exit() or _exit() calls of your tested code, if these return different values from 0.
As for your comment
"What if the test itself doesn't expect it, but it happens anyway? I don't want the rest of my execution to stop. Just that test to fail, then continue on."
Sorry, you can't prevent that. That's what assert() statements are designed for, and act as a self assertion for certain functions that test the inputs or conditions they achieve.
You may try to compile your testing and under test code using the -DNDEBUG compiler option, but this will leavee you with even more obscure issues hitting undefined behavior or such.
If a test case is likely to hit an unexpected assertion, there's either something wrong with your test cases input values, or with the code tested.
So you should setup reproducible conditions, that either the test case fails with the assert (and the unit tester runnable carries on), or the whole thing blows up (exits the test runner process), which means your tested input didn't pass (and you'll need to change the testcase, or fix the the code under test).
Basically, if the code you are testing is broken, the test cannot continue.
To keep gtest from crashing, make sure the code you are testing at least compiles properly, and input it is gathering is valid.
I am saying this not to be mean, but rather out of personal experience. I use gtest and gmock for my own projects. I have been playing around with code lately that was a bit out of my league (after all, the only way to grow is to stretch beyond your perceived limits).
The code was taking data from a data file, and this was crashing my tests, not because there was anything wrong with the test, but because I wasn't doing proper error checking yet for the functions that were reading from the file, and it was throwing a wrench in things when the program was getting a string, and wanted an integer.
Believe it or not, exceptions are a GOOD thing in tests. You don't want to just ignore them and move on, you want to figure out what is causing them and make it stop. That is the entire reason for testing.
In my most C++ project I heavily used ASSERTION statement as following:
int doWonderfulThings(const int* fantasticData)
{
ASSERT(fantasticData);
if(!fantasticData)
return -1;
// ,,,
return WOW_VALUE;
}
But TDD community seems like to enjoy doing something like this:
int doMoreWonderfulThings(const int* fantasticData)
{
if(!fantasticData)
return ERROR_VALUE;
// ...
return AHA_VALUE;
}
TEST(TDD_Enjoy)
{
ASSERT_EQ(ERROR_VALUE, doMoreWonderfulThings(0L));
ASSERT_EQ(AHA_VALUE, doMoreWonderfulThings("Foo"));
}
Just with my experiences first approaches let me remove so many subtle bugs.
But TDD approaches are very smart idea to handle legacy codes.
"Google" - they compare "FIRST METHOD" to "Walk the shore with life-vest, swim ocean without any safe guard".
Which one is better?
Which one makes software robust?
In my (limited) experience the first option is quite a bit safer. In a test-case you only test predefined input and compare the outcome, this works well as long as every possible edge-case has been checked. The first option just checks every input and thus tests the 'live' values, it filters out bugs real quickly, however it comes with a performance penalty.
In Code Complete Steve McConnell learns us the first method can be used successfully to filter out bugs in a debug build. In release build you can filter-out all assertions (for instance with a compiler flag) to get the extra performance.
In my opinion the best way is to use both methods:
Method 1 to catch illegal values
int doWonderfulThings(const int* fantasticData)
{
ASSERT(fantasticData);
ASSERTNOTEQUAL(0, fantasticData)
return WOW_VALUE / fantasticData;
}
and method 2 to test edge-cases of an algorithm.
int doMoreWonderfulThings(const int fantasticNumber)
{
int count = 100;
for(int i = 0; i < fantasticNumber; ++i) {
count += 10 * fantasticNumber;
}
return count;
}
TEST(TDD_Enjoy)
{
// Test lower edge
ASSERT_EQ(0, doMoreWonderfulThings(-1));
ASSERT_EQ(0, doMoreWonderfulThings(0));
ASSERT_EQ(110, doMoreWonderfulThings(1));
//Test some random values
ASSERT_EQ(350, doMoreWonderfulThings(5));
ASSERT_EQ(2350, doMoreWonderfulThings(15));
ASSERT_EQ(225100, doMoreWonderfulThings(150));
}
Both mechanisms have value. Any decent test framework will catch the standard assert() anyway, so a test run that causes the assert to fail will result in a failed test.
I typically have a series of asserts at the start of each c++ method with a comment '// preconditions'; it's just a sanity check on the state I expect the object to have when the method is called. These dovetail nicely into any TDD framework because they not only work at runtime when you're testing functionality but they also work at test time.
There is no reason why your test package cannot catch asserts such as the one in doMoreWonderfulThings. This can be done either by having your ASSERT handler support a callback mechanism, or your test asserts contain a try/catch block.
I don't know which particlar TDD subcommunity you're refering to but the TDD patterns I've come across either use Assert.AreEqual() for positive results or otherwise use an ExpectedException mechanism (e.g., attributes in .NET) to declare the error that should be observed.
In C++, I prefer method 2 when using most testing frameworks. It usually makes for easier to understand failure reports. This is invaluable when a test months to years after the test was written.
My reason is that most C++ testing frameworks will print out the file and line number of where the assert occurred without any kind of stack trace information. So most of the time you will get the reporting line number inside of the function or method and not inside of the test case.
Even if the assert is caught and re-asserted from the caller the reporting line will be with the catch statement and may not be anywhere close to the test case line which called the method or function that asserted. This can be really annoying when the function that asserted may have been used on multiple times in the test case.
There are exceptions though. For example, Google's test framework has a scoped trace statement which will print as part of the trace if an exception occurs. So you can wrap a call to generalized test function with the trace scope and easily tell, within a line or two, which line in the exact test case failed.