I'm looking for some #define which can stop test-case execution if first test failed
TEST_F(TestInitializer, 1st-test) {
Initiator.call();
EXPECT_CALL(mock_obj, onAction(false)).Times(AtLeast(0));
// some define I want
::testing::stopIfFailed();
}
TEST_F(TestInitializer, 2nd-test) {
// this test must not be executed if first test failed
}
run binary with flag --gtest_break_on_failure
Related
I'm working on a TCL project, for which version control is based on Git. So, in order to produce good-quality code, I set up put execution of the tests in pre-commit hook.
However, even if they are executed (trace is shown in command-line), and one of the tests is failed, Git performs the commit. So I launched the hook manually to check the error code, and I figured out that it is null, explaining why Git does not stop:
$ .git/hooks/pre-commit
++++ FlattenResult-test PASSED
(...)
==== CheckF69F70 FAILED
==== Content of test case:
(...)
==== CheckF69F70 FAILED
$ echo $?
0
(Launching the tests script with tclsh also results in $? to be 0.)
So my question is about this last line: why is $? equal to 0, when one of the tcl tests is failed? And how can I achieve a simple pre-commit hook that stops on failure?
I read and reread the tcltest documentation, but saw no setting or information about this error code. And I would really like not to have to parse the tcl tests output, to check if ERROR or FAILED is present...
Edit: versions
TCL version : 8.5
tcltest version: 2.3.4
This depends on how you run your test suite. Normally you run a file called tests/all.tcl which may look something like this:
package require Tcl 8.6
package require tcltest 2.5
namespace import tcltest::*
configure -testdir [file dirname [file normalize [info script]]] {*}$argv
runAllTests
That final runAllTests returns a boolean indicating success (0) or failure (1). You can use that to generate an exit code by changing the last line to:
exit [runAllTests]
I use this redefinition in some of my test scripts:
# Exit non-zero if any tests fail.
# tcltest's `cleanupTests` resets the numTests array, so capture it first.
proc cleanupTests {} {
set failed [expr {$::tcltest::numTests(Failed) > 0}]
uplevel 1 ::tcltest::cleanupTests
if {$failed} then {exit 1}
}
After some research, I could make it work, even though several factors were against me:
I have to use an old TCL version (8.5) with tcltest version 2.3.4, in which runAllTests returns nothing;
I forgot to write cleanupTests at the end of test scripts, as the documentation is not really clear about its usage. (It is not clearer now. I just figured out it is needed if you want to get your tests run by runAllTests, which is really not obvious).
And here is my solution, mostly based on Hai's DevBits blog post:
all.tcl
package require tcltest
::tcltest::configure (...)
proc ::tcltest::cleanupTestsHook {} {
variable numTests
set ::exitCode [expr {$numTests(Total) == 0 || $numTests(Failed) > 0}]
}
::tcltest::runAllTests
exit $exitCode
Some thoughts about it:
I added $numTests(Total) == 0 as a failure condition: this means that no tests was found, which is clearly an erroneous condition;
This doesn't catch exceptions in the configuration of the tests, for instance a source command that points to a non-existing file, revealing some failure in tests scaffolding. This would be catched as error in other test framewords (ah, pytest, I miss you!)
We have thousands of tests, and one is stuck (some infinite loop probably).
I'd like to know which tests is running but sbt only displays tests that are done. It also shows the tests titles, but as I said, we have too many tests to know which title the stuck test is part of.
Try using runner arguments to configure a file reporter
-f output.txt
in combination with unformatted or "ugly" mode flag U which
Rather than attempting to make the output look as pretty and
human-readable as possible, unformatted mode will just print out
verbose information about each event as it arrives, helping you track
down the problem you are trying to debug.
and then tail the output file during test execution
tail -f output.txt
which will show events in realtime as they are happening as opposed to at the end of testing.
Now given following example
Test / testOptions += Tests.Argument("-fU", "output.txt")
class HelloSpec extends FlatSpec with Matchers {
"The Hello object" should "satisfy case 1" in {
assert(true)
}
it should "satisfy case 2" in {
assert(true)
}
it should "satisfy case 3 (stuck here)" in {
while(true) { /* do something forever */ }
assert(true)
}
it should "satisfy case 4" in {
assert(true)
}
}
then tail -f output.txt outputs
Run starting. Expected test count is: 0
Suite Starting - HelloSpec
Scope Opened - HelloSpec: The Hello object
Test Starting - HelloSpec: The Hello object should satisfy case 1
Test Succeeded - HelloSpec: The Hello object should satisfy case 1
Test Starting - HelloSpec: The Hello object should satisfy case 2
Test Succeeded - HelloSpec: The Hello object should satisfy case 2
Test Starting - HelloSpec: The Hello object should satisfy case 3 (stuck here)
where we can identify the stuck test on the last line.
I have a function that prints an error message to stdout when it encounters a certain condition.
if(!condition)
printf("ERROR: Condition fail!");
The unit test for this function is picking up the "ERROR" keyword, and while all the assertions pass, the boost unit test suite considers it an error in execution and displays it. Is there a way to specify an expected error message, or expected error string for a given unit test?
The unit test for this function is picking up the "ERROR" keyword,
It is not.
and while all the assertions pass, the boost unit test suite considers it an error in execution and displays it
Not at all. The unit test is unaware of the fact that your test prints to the console.
Your test, though, contains a line of code:
printf("ERROR: Condition fail!");
That line has the immediate effect of printing the following text to the standard output device:
ERROR: Condition fail!
This is the only reason you're seeing the text. If you inspect the report or log you'll find that (unless asserts failed) the test passed.
I am using IAR EWARM's cspybat to run some unit tests for my embedded code using Unity. I would like an easy way for my build server to determine if the unit tests passed or failed. Is there a way for CSPY to return a nonzero error code if my unit tests fail? I have tried changing the return value in main() with no change. Is there a function I can call to force an error to be returned?
My cspybat batch file looks like this:
"C:\Program Files (x86)\IAR Systems\Embedded Workbench 7.4\common\bin\cspybat" -f "C:\Work\Sandbox\ST\stmicroeval\_iar_ewarm_project\settings\Project.UnitTest.general.xcl" --backend -f "C:\Work\Sandbox\ST\stmicroeval\_iar_ewarm_project\settings\Project.UnitTest.driver.xcl"
Unfortunately, no.
I've solved this by replacing "exit" with a function that prints a specific pattern, plus the exit code. I then wrapped the call to cspybat into a script that 1) strips the output of the extra output and 2) exits with the desired exit code.
It's late 2020 and they still don't offer a mechanism to do this.
We solved it by including a macro file with the contents:
execUserExit()
{
__message "program exited with __exit_value = ", __exit_value:%d ;
}
And having our own exit variable in the code:
extern "C" int __exit_value=0xff;
That we set prior to calling exit() (though you could just write your own version of exit())
This makes the debugger always print SOMETHING, even if the program crashes on startup.
Then we parse with a python wrapper:
pattern = "__exit_value =\s([\-|0-9|a-f|A-F|x]*)"
retvalue = int(re.findall(pattern,process.stdout)[0])
I need to be able to:
fork a process and make it execvp (I did that)
check if the child process execvp was successful (don't know how)
check if the child process finished (having problems)
I'm forking a process and I don't have any way to check if the childs's execvp worked or failed. If it failed I need to be able to know that it failed. Currently I'm using
-1 != waitpid( pid, &status, WNOHANG )
But it seems that if the execv of the pid process fails the waitpid does not return -1.
How could I check that? I read the waitpid man page, but it isn't clear to me; maybe my English isn't good enough.
EDIT: in order to explain more:
I'm building my own terminal for a Home Work. I need to get as an input a command string, lets say "ls" and then I have to execute the command.
After the child forks, the child calls execvp in order to execute the command ( after I parse the string ), and the parent need to check whether there was a '&' at the end of the command or not.
if the sign '&' does not exist at the end of the command then the parent need to wait for the child to execute.
so I need to know if the execvp failed. If it didn't failed then the parent use waitpid to wait for the child to finish it execution. If it failed then the parent will not wait for the child.
A common solution to #2 is to open a pipe prior to the fork(), then write to it in the child following the exec. In the parent, a successful read means the exec failed; an unsuccessful read means the exec succeeded and the write never took place.
// ignoring all errors except from execvp...
int execpipe[2];
pipe(execpipe);
fcntl(execpipe[1], F_SETFD, fcntl(execpipe[1], F_GETFD) | FD_CLOEXEC);
if(fork() == 0)
{
close(execpipe[0]);
execvp(...); // on success, never returns
write(execpipe[1], &errno, sizeof(errno));
// doesn't matter what you exit with
_exit(0);
}
else
{
close(execpipe[1]);
int childErrno;
if(read(execpipe[0], &childErrno, sizeof(childErrno)) == sizeof(childErrno))
{
// exec failed, now we have the child's errno value
// e.g. ENOENT
}
}
This lets the parent unambiguously know whether the exec was successful, and as a byproduct what the errno value was if unsuccessful.
If the exec was successful, the child process may still fail with an exit code, and examining the status with the WEXITSTATUS macro give you that condition as well.
NOTE: Calling waitpid with the WNOHANG flag is nonblocking, and you may need to poll the process until a valid pid is returned.
An exec call shouldn't return at all if it succeeds, because it replaces the current process image with another one, so if it does it means an error has occurred:
execvp(...);
/* exec failed and you should exit the
child process here with an error */
exit(errno);
To let the parent process know if exec failed you should read the status of the child process:
waitpid(pid, &status, WNOHANG);
And then use the WEXITSTATUS(status) macro, from the man page:
WEXITSTATUS(status) returns the exit status of the child. This
consists of the least significant 8 bits of the status argument that
the child specified in a call to exit(3) or _exit(2) or as the argument for a return statement in main()
Note the last statement means if exec succeeds and runs the command you will get the exit status of the main() function of that command, in other words you can't reliably tell the difference between a failed exec and a failed command this way, so it depends if that matters to you.
Another issue:
if the sign '&' does not exist at the end of the command then the
parent need to wait for the child to execute.
You need to call wait() on the child process at some point in your program, regardless of the & to avoid leaving the child process in a zombie state,
Note: When you use the WNOHANG it means that waitpid() will return immediately if no process has changed its state, i.e. it will not block, I assume you know that, otherwise use wait() or call waitpid() as part of your main loop.