I want to group all of the known flakey jest tests in a codebase with the equivalent of a marker in pytest (docs here) as a strategy to eventually remove them.
Does jest allow for this? Is there some command line flag to do this?
Related
I'm trying to only run a certain subsection of xunit tests using the tfs Test Filter Criteria:
The problem is that this is not doing anything, it still runs every test regardless of it's name, what might I be missing here? Is there something else I need to be doing since I'm using Xunit? Or is there another method of excluding tests in the test unit step I can use.
Test filter criteria: Filters tests from within the test assembly files. For example, “Owner=james&Priority=1”. This option works the
same way as the console option /TestCaseFilter for vstest.console.exe
For more information, see
https://msdn.microsoft.com/en-us/library/jj155796.aspx
Please double check your property of FullyQualifiedName in your assembly files. I'm not sure if /TestCaseFilter is also support in xUnit. You could give a try with directly running the test using command line (vstest.console.exe). If it's not work, then this should also not work in TFS build task.
Add a related link talk about test filter for your reference: VSTS/TFS VISUAL STUDIO TEST TASK – FILTER CRITERIA
Grails 3 (at least 3.1.10) is flaky in running only specific tests. How do I get it to run a single integration test?
Here is a sample command to run a single integration test
grails test-app *LoginFunctional* -integration
If you put -integration flag before pattern, the test-app command will ignore the pattern and execute all integration tests.
The official command line syntax is grails test-app, optionally followed by a pattern to match the full namespaced class name of what you want to test such as org.myorg.ClassToTest or org.**.*, and -unit or -integration to select a specific phase. See the docs.
There are a number of quirks in Grails 3.1.10, though.
1) grails test-app won't always run the tests, probably a bug in the dependency management. If you first remove the test report at build/reports/tests/index.html grails will see that it actually needs to do something to generate a new report.
2) Sometimes things just go randomly weird. If so, do grails clean; grails test clean. (I didn't yet figure out if you really need both of them or only one of the two.)
3) The official way should work, but do (2) first if it doesn't. Also, if you want to run only a specific integration test you need to add -integration or you'll get an error. I think without such a flag Grails unconditionally first tries to run unit tests and then integration tests, and if your test pattern does not match any unit tests grails will error out. Similarly if the pattern only matches unit tests add -unit or you'll get an error, though you still get the correct test report in this case.
4) There's also an alternative way, by using the -Dtest.single=<classname> flag. This sets a system property that is picked up by gradle. I only got it working properly if I also added a -unit flag, but I didn't investigate very deep.
I usually use annotation #IgnoreRest. Remember to import spock.lang.IgnoreRest and run test on specified class.
I want to execute all tests from my application, now I do it with command:
go test ./app/...
Unfortunately it takes quite a long time, despite that single tests run quite fast. I think that the problem is that go needs to compile every package (with its dependence) before it runs tests.
I tried to use -i flag, it help a bit but still I'm not satisfied with the testing time.
go test -i ./app/...
go test ./app/...
Do you have any better idea how to efficiently test multiple packages.
This is the nature of go test: it builds a special runtime with addition code to execute (this is how it tracks code coverage).
If it isnt fast enough, you have two options:
1) use bash tooling to compile a list of packages (e.g. using ls), and then execute them each individually in parallel. There exists many ways to do this in bash.
The problem with this approach is that the output will be interleaved and difficult to track down failures.
2) use the t.Parallel() flag with each of your tests to allow the test runtime to execute in parallel. Since Go 1.5, go test runs with GOMAXPROCS set to the number of cores on your CPU which allows for concurrently running tests. Tests are still ran synchronously by default. You have to set the t.Parallel() flag for each test, telling the runtime it is OK to execute this test in parallel.
The problem with this approach being that it assumes you followed best practices and have used SoC/decoupling, don't have global states that would mutate in the middle of another test, no mutex locks (or very few of them), no race condition issues (use -race), etc.
--
Opinion: Personally, I setup my IDE to run gofmt and go test -cover -short on every Save. that way, my code is always formatted and my tests are run, only within the package I am in, telling me if something failed. The -cover flag works with my IDE to show me the lines of code that have been tested versus not tested. The -short flag allows me to write tests that I know will take a while to run, and within those tests I can check t.Short() bool to see if I should t.Skip() that test. There should be packages available for your favorite IDE to set this up (I did it in Sublime, VIM and now Atom).
That way, I have instant feedback within the package I m editing.
Before I commit the code, I then run all tests across all packages. Or, I can just have the C.I. server do it.
Alternatively, you can make use of the -short flag and build tags (e.g. go test -tags integration) to refactor your tests to separate your Unit tests from Integration tests. This is how I write my tests:
test that are fast and can run in parallel <- I make these tests run by default with go test and go test -short.
slow tests or tests that require external components, I require addition input to run, like go test -tags integration is required to run them. This pattern does not run the integration tests with a normal go test, you have to specify the additional tag. I don't run the integration tests across the board either. That's what my CI servers are for.
If you follow a consistent name scheme for your tests you can easily reduce the number of them you execute by using the -run flag.
Quoting from go help testflag:
-run regexp
Run only those tests and examples matching the regular expression.
So let's say you have 3 packages: foo, bar and qux. Tests of those packages are all named like TestFoo.., TestBar.. and TestQux.. respectively.
You could do go test -run '^Test(Foo|Bar)*' ./... to run only tests from foo and bar package.
When using WebStorms as a test runner every unit test is run. Is there a way to specify running only one test? Even only running one test file would be better than the current solution of running all of them at once. Is there a way to do this?
I'm using Mocha.
not currently possible, please vote for WEB-10067
You can double up the i on it of d on describe and the runner will run only that test/suite. If you prefix it with x it will exclude it.
There is a plugin called ddescribe that gives you a gui for this.
You can use the --grep <pattern> command-line option in the Extra Mocha options box on the Mocha "Run/Debug Configurations" screen. For example, my Extra Mocha options line says:
--timeout 5000 --grep findRow
All of your test *.js files, and the files they require, still get loaded, but the only tests that get run are the ones that match that pattern. So if the parts you don't want to execute are tests, this helps you a lot. If the slow parts of your process automatically get executed when your other modules get loaded with require, this won't solve that problem. You also need to go into the configuration options to change the every time you want to run tests matching a different pattern, but this is quick enough that it definitely saves me time vs. letting all my passing tests run every time I want to debug one failing test.
You can run the tests within a scope when you have a Mocha config setting by using .only either on the describe or on the it clauses
I had some problems getting it to work all the time, but when it went crazy and kept running all my tests and ignoring the .only or .skip I added to the extra mocha options the path to one of the files containing unit tests just like in the example for node setup and suddenly the .only feature started to work again regardless of the file the tests were situated in.
I googled and found the below helpful references. Currently I want to run all from the command-line (for easy of execution & quickness) in cases:
A specific test (ie. a test written by a method marked [TestMethod()])
All tests in a class
All impacted tests of the current TFS pending change of mine.
All tests
All tests except the ones marked as category [TestCategory("some-category")]
I'm not sure how can I write a correct command for my needs above.
References:
the MSTest.exe http://msdn.microsoft.com/en-us/library/ms182487.aspx
the MSTest.exe's detailed options http://msdn.microsoft.com/en-us/library/ms182489.aspx
obtaining the result http://msdn.microsoft.com/en-us/library/ms182488.aspx
[Edit]
After a while, I found the below useful tips.
run Visual Studio unit tests by using MSTest.exe, located at %ProgramFiles%\Microsoft Visual Studio 10.0\Common7\IDE\MSTest.exe in my case.
using /testcontainer:Path\To\Your\TestProjectAssembly.dll to indicate where your tests are coded. You can specify multiple '/testcontainer' options if required.
using /test:TestFilter to filter the tests to run. Note that this filter is applied to the full test method name (ie. FullNamespace.Classname.MethodName)
Currently I can have some answers for my needs:
A specific test (ie. a test written by a method marked [TestMethod()])
Use MSTest.exe /container:TheAssemblyContainingYourSpecificTest /test:TheSpecificTestName
All tests in a class
Use MSTest.exe /container:TheAssemblyContainingYourClass /test:TheClassNameWithFullNamespace
Note that the /test: is the filter which uses the full name of the class when filtering.
The others are still left unknown. Please disscuss if you know how.
For number 4. To run all tests in an assembly it's simply:
mstest /testcontainer:YourCompiledTestAssembly.dll
For question
5 All tests except the ones marked as category
[TestCategory("some-category")]
Use
mstest.exe /testcontainer:yourTests.dll /category:"!some-category"
If you need to exclude more than one category, use
mstest.exe /testcontainer:yourTests.dll /category:"!group1&!group2"
Reference: /category filter
You might be interested by the Gallio bundle. It provides a free common automation platform to run your tests (MSTest, MbUnit, NUnit, xUnit, etc.) with various test runners (GUI, command line, PoSh, plugins for 3rd party tools, etc.)
In particular you may want to use Gallio.Echo which is a nice command line test runner:
The Gallio test runners have also filtering capabilities to run a subset of your unit tests only (e.g. per category, per fixture, etc.)
** adding this due to errors I've encountered.
To run all just use '''vstest.console.exe .\x64\Release\UnitTesting.dll'''
vstest.console.exe is not deprecated so you will not need the /nologo suppression.
If needed it also has --TestCaseFilter|/TestCaseFilter: