If I run the following
go test -run NotExist
The response is PASS. Seeing as my test file does not contain a test called TestNotExist I would expect the command above to return FAIL
Without the -run option go test runs all tests. You use the -run option to not run all tests; to filter out, to exclude tests (and you do this in the form of requiring names of non-excludable tests to match a regexp pattern - but this is irrelevant form the point of discussion):
Command go, Test packages:
By default, go test needs no arguments. It compiles and tests the package with source in the current directory, including tests, and runs the tests.
Description of testing flags:
-run regexp
Run only those tests and examples matching the regular expression.
It is a perfectly "normal" outcome that a filtering filters out all tests, that no tests remains in the set of tests that still need to be executed.
When no tests FAIL, it is considered as the test run PASSes. If no tests match, no tests will run and no tests will FAIL, and thus the test run will PASS.
Related
I have a Go application with a number of unit and benchmark tests both in the root and in a subfolder called "message".
I execute the following command to run all unit tests from the root including the ones in the messages and any other subfolder:
go test ./...
I want to achieve the same for the benchmark tests, i.e. run them all. The following works for the ones in the root directory:
go test -bench .
The benchmark tests in the /messages folder are ignored which is expected. So I run the following from the root:
go test -bench ./...
That's not recognised at all, Go seems to execute the unit tests that are located in the root dir. I even tried to specify the message folder in the command as follows:
go test -bench ./message
...but it also failed. Currently if I want to run the benchmark tests in the message folder I have to cd into that folder and execute
go test -bench .
like above.
So what's the correct way then? How can I tell Go to find the benchmark tests both in the root and the subfolders? How does the regexp arg work in the case of the -bench flag? Apparently it's different from the regexp for the unit test runner.
You should use ./... to bench all the files from the current working directory and all of its subdirectories. If you wish to get a more verbose output you can use the -v flag. Also it's good to list the memory allocation by using -benchmem.
go test -v ./... -bench=. -run=xxx -benchmem
-bench flag takes regex so to run all benchmarks (-bench .) in all packages: go test -bench=. ./...
I have two tests in my project, I would like to build a single test, place the resulting binary in a container, run it, and then attach a debugger.
Is this possible?
package dataplatform
import "testing"
func TestA(t *testing.T) {
// test A
}
func TestRunCommand(t *testing.T) {
// Test B
}
You may use -run <regexp> to limit (filter) the tests to run. So for example if you want to run only the TestA() test, you may do it like this:
go test -run TestA
Actually the above will run all tests whose names contain TestA, so to be explicit, it would be:
go test -run ^TestA$
To not run the tests but generate the test binary, you may use the -c option:
go test -c
This won't run the tests, but compile a binary which when executed will run the tests.
The problem is that you can't combine these options, e.g. running
go test -c -run TestA
Will generate a binary which when executed will run all tests.
The truth is that the generated binary accepts the same parameters as go test, so you may pass -run TestA to the generated binary, but you must prefix the params with test:
Each of these flags is also recognized with an optional 'test.' prefix, as in -test.v. When invoking the generated test binary (the result of 'go test -c') directly, however, the prefix is mandatory.
So if the name of the generated test binary is my.test, run it like:
./my.test -test.run TestA
For more options and documentation, run go help test, or visit the official documentation:
Command Go
And the relevant section:
Command Go: Testing flags
I have a number of unit tests in VSCode which I want to debug. The problem is I want to debug only one unit test. By default VS Code runs all the unit tests. I want to specify only one unit test to run like in Visual Studio. Is there any way to do this?
According to the unit test framework you are using you should be able to create a config.json or something similar where you can set options for your tests. Within this config you should be able to specify the path of all unit tests that shall be executed. For example when you are running javaScript unit tests with jasmine you can set something like this in your config.json:
"spec_dir": "./out/test/unittest",
"spec_files": [
"**/*[tT]est.js"
]
Here you can check out how this behavior is done with mocha. To run only one unit test you just set an absolute path. For example
"spec_files": [
"./path/to/a/single/unittest.js"
]
I am using SpecFlow to write Gherkin for automated UI tests. We have multiple tags (#smoke, #on, #off, etc.).
Test A has tags #smoke
Test B has tags #smoke #off
These are translated to vstest test categories and do not include the # symbol.
I want to run test A but not test B.
This is what is not working. It runs all #smoke tests.
Vstest.console.exe mytest.dll /TestCaseFilter:"TestCategory=smoke&TestCategory!=off"
Any ideas?
I had two issues.
I was executing the test with Thoughtworks Go which did weird things with the quotes before executing the command
I needed to surround the entire condition with parenthesis as such: Vstest.console.exe mytest.dll /TestCaseFilter:"(TestCategory=smoke&TestCategory!=off)"
I'm using CTest with CMake to run some tests. I use the enable_testing() command which provides me with a default command for make test. All of the tests in my subdirectory are accounted for (by doing an add_test command) and make test works great, except one problem.
There is a certain test, which I've named skip_test, that I do NOT want being run when I do make test. I would like to add a custom target so I can run make skip_test and it will run that test.
I can do this by doing add_custom_target(skip_test ...) and providing CTest with the -R flag and telling it to look for files containing "skip_test" in their name. This also seems to work. My problem now is: how can I get the make test command to ignore skip_test?
If I try commenting out enable_testing and adding my own add_custom_target(test ....), I get "No tests found!!!" now for either make test or make skip_test. I also tried making a Custom CTest file and adding set(CTEST_CUSTOM_TESTS_IGNORE skip_test). This worked so that now make test ignored "skip_test", but now running make skip_test responds with "no tests found!!!".
Any suggestions would be appreciated!
I actually used a different solution. Here is what I did. For the tests that I wanted to exclude, I used the following command when adding them:
"add_test( ..... CONFIGURATIONS ignore_flag)" where ignore_flag is whatever phrase you want. Then, in my CMakeLists.txt, when I define a custom target
add_custom_target( ignore_tests ...)
I give it ctest .... -C ignore_flag
Now, make test WILL skip these tests! make ignore_Tests will run the ignored tests + the un-ignored tests, which I'm okay with.
I'm not sure of a way to do this entirely via CTest, but since you've tagged this question with "googletest", I assume you're using that as your test framework. So, you could perhaps make use of Gtest's ability to disable tests and also to run disabled tests.
By changing the test(s) in question to have a leading DISABLED_ in their name(s), these won't be run by default when you do make test.
You can then add your custom target which will invoke your test executable with the appropriate Gtest flags to run only the disabled tests:
add_custom_target(skip_test
MyTestBinary --gtest_filter=*DISABLED_* --gtest_also_run_disabled_tests VERBATIM)
It's a bit of an abuse of the Gtest functionality - it's really meant to be used to temporarily disable tests while you refactor whatever to get the test passing again. This beats just commenting out the test since it continues to compile it, and it gives a nagging reminder after running the suite that you have disabled tests.