We have thousands of tests, and one is stuck (some infinite loop probably).
I'd like to know which tests is running but sbt only displays tests that are done. It also shows the tests titles, but as I said, we have too many tests to know which title the stuck test is part of.
Try using runner arguments to configure a file reporter
-f output.txt
in combination with unformatted or "ugly" mode flag U which
Rather than attempting to make the output look as pretty and
human-readable as possible, unformatted mode will just print out
verbose information about each event as it arrives, helping you track
down the problem you are trying to debug.
and then tail the output file during test execution
tail -f output.txt
which will show events in realtime as they are happening as opposed to at the end of testing.
Now given following example
Test / testOptions += Tests.Argument("-fU", "output.txt")
class HelloSpec extends FlatSpec with Matchers {
"The Hello object" should "satisfy case 1" in {
assert(true)
}
it should "satisfy case 2" in {
assert(true)
}
it should "satisfy case 3 (stuck here)" in {
while(true) { /* do something forever */ }
assert(true)
}
it should "satisfy case 4" in {
assert(true)
}
}
then tail -f output.txt outputs
Run starting. Expected test count is: 0
Suite Starting - HelloSpec
Scope Opened - HelloSpec: The Hello object
Test Starting - HelloSpec: The Hello object should satisfy case 1
Test Succeeded - HelloSpec: The Hello object should satisfy case 1
Test Starting - HelloSpec: The Hello object should satisfy case 2
Test Succeeded - HelloSpec: The Hello object should satisfy case 2
Test Starting - HelloSpec: The Hello object should satisfy case 3 (stuck here)
where we can identify the stuck test on the last line.
Related
I'm working on a TCL project, for which version control is based on Git. So, in order to produce good-quality code, I set up put execution of the tests in pre-commit hook.
However, even if they are executed (trace is shown in command-line), and one of the tests is failed, Git performs the commit. So I launched the hook manually to check the error code, and I figured out that it is null, explaining why Git does not stop:
$ .git/hooks/pre-commit
++++ FlattenResult-test PASSED
(...)
==== CheckF69F70 FAILED
==== Content of test case:
(...)
==== CheckF69F70 FAILED
$ echo $?
0
(Launching the tests script with tclsh also results in $? to be 0.)
So my question is about this last line: why is $? equal to 0, when one of the tcl tests is failed? And how can I achieve a simple pre-commit hook that stops on failure?
I read and reread the tcltest documentation, but saw no setting or information about this error code. And I would really like not to have to parse the tcl tests output, to check if ERROR or FAILED is present...
Edit: versions
TCL version : 8.5
tcltest version: 2.3.4
This depends on how you run your test suite. Normally you run a file called tests/all.tcl which may look something like this:
package require Tcl 8.6
package require tcltest 2.5
namespace import tcltest::*
configure -testdir [file dirname [file normalize [info script]]] {*}$argv
runAllTests
That final runAllTests returns a boolean indicating success (0) or failure (1). You can use that to generate an exit code by changing the last line to:
exit [runAllTests]
I use this redefinition in some of my test scripts:
# Exit non-zero if any tests fail.
# tcltest's `cleanupTests` resets the numTests array, so capture it first.
proc cleanupTests {} {
set failed [expr {$::tcltest::numTests(Failed) > 0}]
uplevel 1 ::tcltest::cleanupTests
if {$failed} then {exit 1}
}
After some research, I could make it work, even though several factors were against me:
I have to use an old TCL version (8.5) with tcltest version 2.3.4, in which runAllTests returns nothing;
I forgot to write cleanupTests at the end of test scripts, as the documentation is not really clear about its usage. (It is not clearer now. I just figured out it is needed if you want to get your tests run by runAllTests, which is really not obvious).
And here is my solution, mostly based on Hai's DevBits blog post:
all.tcl
package require tcltest
::tcltest::configure (...)
proc ::tcltest::cleanupTestsHook {} {
variable numTests
set ::exitCode [expr {$numTests(Total) == 0 || $numTests(Failed) > 0}]
}
::tcltest::runAllTests
exit $exitCode
Some thoughts about it:
I added $numTests(Total) == 0 as a failure condition: this means that no tests was found, which is clearly an erroneous condition;
This doesn't catch exceptions in the configuration of the tests, for instance a source command that points to a non-existing file, revealing some failure in tests scaffolding. This would be catched as error in other test framewords (ah, pytest, I miss you!)
I'd like to get the number of tests that were run with go test, as kind of a checksum to detect if all the tests are running. Since Go relies on filenames and method names to determine what's a test, it's easy to mistype something, which would mean the test would silently be skipped.
I think that the gotestsum tool is close to what you are looking for.
It is a wrapper around go test that prints formatted test output and a summary of the test run.
Default go test:
go test ./...
? github.com/marco-m/timeit/cmd/sleepit [no test files]
ok github.com/marco-m/timeit/cmd/timeit 0.601s
Default gotestsum:
gotestsum
∅ cmd/sleepit
✓ cmd/timeit (cached)
DONE 11 tests in 0.273s <== see here
Checkout the documentation and the built-in help, it is well-written.
In my experience, gotestsum (and the other tools by the same organization) is good. For me, it is also very important to be able to use the standard Go test package, without other "test frameworks". gotestsum allows me to do so.
On the other hand, to really satisfy your requirement (print the number of declared tests and verify that that number is actually ran), you would need something like TAP, the Test Anything Protocol, which works for any programming language:
1..4 <== see here
ok 1 - Input file opened
not ok 2 - First line of the input valid
ok 3 - Read the rest of the file
not ok 4 - Summarized correctly # TODO Not written yet
TAP actually is very nice and simple. I remember there was a Go port, tap-go, but it is now marked as archived.
Is there a way to rerun entire Test Suite if a single Particular test case fails .
So for example , a Robot Code which contain a test case which will check the cookie value , if the cookie is of a particular pattern will continue execution of rest of code , if it fails it should rerun the entire Robot Code / Test Suite and repeat this 3 times , if the cookie value is not same for three runs , let it fail the test suite completely .
You can run original test, rerun failed and merge the results of both runs. If some tests are failed in the first run and then pass in the second, you will see that in the results.
There is often a need to re-execute a subset of tests, for example,
after fixing a bug in the system under test or in the tests
themselves. This can be accomplished by selecting test cases by names
(--test and --suite options), tags (--include and --exclude), or by
previous status (--rerunfailed or --rerunfailedsuites).
robot --output original.xml tests # first execute all tests
robot --rerunfailedsuites original.xml --output rerun.xml tests # then re-execute failing
rebot --merge original.xml rerun.xml # finally merge results
You can read more about this in here https://robotframework.org/robotframework/latest/RobotFrameworkUserGuide.html#merging-re-executed-tests
To your specific example, I am not sure that you can do that. But you can save the exit code of the run and evaluate it base on that
robot "your robot options" $#
if [ $? -eq 0 ]; then
"evaluation options when passed"
fi
else
"evaluation options when failed"
fi
I've written a command line tool that I want to test (I'm not looking to run unit tests from command line). I want to map a specific set of input options to a specific output. I haven't been able to find any existing tools for this. The application is just a binary and could be written in any language but it accepts POSIX options and writes to standard output.
Something along the lines of:
For each known set of input options:
Launch application with specified input.
Pipe output to a file.
Diff output to stored (desired) output.
If diff is not empty, record error.
(Btw, is this what you call an integration test rather than a unit test?)
Edit: I know how I would go about writing my own tool for this, I don't need help with the code. What I want to learn is if this has already been done.
DejaGnu is a mature and somewhat standard framework for writing test suites for CLI programs.
Here is a sample test taken from this tutorial:
# send a string to the running program being tested:
send "echo Hello world!\n"
# inspect the output and determine whether the test passes or fails:
expect {
-re "Hello world.*$prompt $" {
pass "Echo test"
}
-re "$prompt $" {
fail "Echo test"
}
timeout {
fail "(timeout) Echo test"
}
}
Using a well-established framework like this is probably going to be better in the long run than anything you can come up with yourself, unless your needs are very simple.
You are looking for BATS (Bash Automated Testing System):
https://github.com/bats-core/bats-core
From the docs:
example.bats contains
#!/usr/bin/env bats
#test "addition using bc" {
result="$(echo 2+2 | bc)"
[ "$result" -eq 4 ]
}
#test "addition using dc" {
result="$(echo 2 2+p | dc)"
[ "$result" -eq 4 ]
}
$ bats example.bats
✓ addition using bc
✓ addition using dc
2 tests, 0 failures
bats-core
Well, I think every language should have a way of execute an external process.
In C#, you could do something like:
var p = new Process();
p.StartInfo = new ProcessStartInfo(#"C:\file-to-execute.exe");
... //You can set parameters here, etc.
p.StartInfo.RedirectStandardOutput = true;
p.StartInfo.RedirectStandardInput = true;
p.StartInfo.UseShellExecute = false;
p.Start();
//To read the standard output:
var output = p.StandardOutput.ReadToEnd();
I have never had to write to the standard input, but I believe it can be done by accessing to p.StandardInput as well. The idea is to treat both inputs as Stream objects, because that's what they are.
In Python there is the subprocess module. According to its documentation:
The subprocess module allows you to spawn new processes, connect to their input/output/error pipes, and obtain their return codes.
I had to do the same when writing unit tests for the code generation part of a compiler I write some months ago: Writing unit tests in my compiler (which generates IL)
We wrote should, a single-file Python program to test any CLI tool. The default usage is to check that a line of the output contains some pattern. From the docs:
# A .should file launches any command it encounters.
echo "hello, world"
# Lines containing a `:` are test lines.
# The `test expression` is what is found at the right of the `:`.
# Here 'world' should be found on stdout, at least in one line.
:world
# What is at the left of the `:` are modifiers.
# One can specify the exact number of lines where the test expression has to appear.
# 'moon' should not be found on stdout.
0:moon
Should can check occurrences counts, look for regular expressions, use variables, filter tests, parse json data, and check exit codes.
Sure, it's been done literally thousands of times. But writing a tool to run simple shell scripts or batch files like what you propose is a trivial task, hardly worth trying to turn into a generic tool.
It seems cake test always returns 0 to the shell, even when some tests fail. This is probably fine for human, but causes problem in continuous integration. Is there anyway to make cake test return non-0 when some test fails?
You can always use grep:
cake test | grep -qve '0 tests failed'
Replace "0 tests failed" with a string that clearly determines that all tests passed fine.
I don't know how good is an answer after 2 years, but looks like the solution to this is explained here:
https://www.hiddentao.com/archives/2011/12/21/instructing-cakefile-to-exit-with-error-if-a-vows-test-fails/
What I understand is that cake by itself is very lightweight and doesn't do anything about returning a proper exit code. But by looking for a couple of clues like presence of "✗ Broken" in the output or checking the exit code of the spec runner itself, you can instruct cake to return "1", so that your CI understands that the tests failed.