I have a VS UnitTest (MSTest) to cover some multiple threading code. With out the required locks the code fails the test 15 out 30 runs - i.e half the runs. I fix the code and the tests pass 30 times.
Now I want the test framework to run the test N time and not just once to be sure that it cant pass once out of luck. There doesn't seem to be any way to do that with an attribute on the test so I put a loop INSIDE the test to run the test code body N times.
I remove the fixes to the code (locks etc) and run the test (which loops N times) - and bam it passes. I run it again (loop N times) and it fails... Im back where I started - its still failing only half the time (even though its doing N loops through test body on each test).
What I really want is not to loop inside the test but really have the test framework load the test, run test, unload test etc N times (as I did by hand originally). How to do that? (Why isn't there just a test attribute for this - like [TestRepeatCount=5]?)
What you really need is a Load Test. Add your unit test(s) and configure it how it will run.
You can set the number of total tests, the number of concurrent tests e.t.c.
Related
I find it useful to run my tests under --gtest_repeat for 100 or 1000 times after all tests pass as a final test to make sure there are no race conditions. However in situations where SetUpTestSuite / TearDownTestSuite can be run only once (e.g. tests use a singleton class that cannot be created multiple times) during program execution this is not possible.
So is there a way to repeat tests without re-running SetUpTestSuite?
If not how would you overcome this issue? I can try using ctest but that would make it difficult to drop into a debugger as I think I would need to attach to the process being run by ctest somehow.
Note: I can use any version of googletest as needed.
I have a large unit test suite written in C++ using Google Test.
I have recently made a change to the codebase which may affect different parts of the system, so various tests should now probably fail or even crash. I would like to run once the entire suite (which unfortunately takes a long time to complete), summarize the list of the failed tests, and fix them one by one.
However, when ever a test crashes (e.g. with a segmentation fault) as opposed to simply logically failing, it seems that GTest stops and executes no more tests.
I can than fix the crashed test, however rerunning the entire suite will take a long time.
Is there a way to tell GTest to resume executing the rest of the tests after a test has crashed?
Or, alternatively, at least a way to launch GTest starting from a particular test (assuming the order of the tests is always the same)?
If you are are need to test if assertion is triggered when API is used incorrectly then gtest delivers something called DEATH TEST.
If your test crashed because of Segmentation Fault you should fix this ASAP! You can disable test temporary by adding DISABLED_ prefix, or by adding GTEST_SKIP() in test boy. Alternatively there is also command line argument --gtest_filter=<colon separated positive patterns>[:-<colon separated negative patterns>]. There is no way to recover from segmentation fault, so test suite can't continue.
If you use gcc or clang (msvc has this feature experimentally) you can enable address sanitizer to quickly detect all memory issues in your tested code. You will able to faster fix those issues.
There are cool plugins to IDE to handle gtest, those should you help you track which test were run, which failed and which crashed.
Google tests are not able to do what you need. I'd suggest you write a simple test runner that:
Runs the test executable with --gtest_list_tests to get a list of all tests.
Runs a loop thru all tests that prints out the test number and runs the test executable with --gtest_filter=FooTest.Bar to invoke only one test in each loop iteration.
The loop skips the required number of iterations and runs from the number N after the test with the number N is fixed.
You only need to write such a runner script once, and it shouldn't be hard.
Ran into a really odd issue yesterday while doing my Jasmine tests (which run headless usually, but can debug in Chrome). A test that usually passes seems to fail when I reach a specific total test count (678), but succeeds again the moment I have more than that. I reduced the number of tests such that I was only running that one test suite, and could repro the same problem at 177 tests, which I did by taking a very simple non-failing test and duplicating it a bunch more times.
I'm not seeing any other issues (i.e. a page reload error), and even stranger is that the test that supposedly fails doesn't match the line number Jasmine spits out as the offending line (which is actually the following test). When I manually step through these, it's obvious that the spy IS called, and I do believe I'm handling the async stuff correctly, as the code involves promises.
I know that isn't super specific, but I'm curious if anyone has run into this before, and has ideas for how to proceed in debugging this?
Came to the same conclusion that Sulthan did in his comment above, but it turned out to be a problem with when I was calling my expects in relation to where I was calling done() in some tests that involved async calls/promises. Seems like the number of tests that would run would create timing issues that finally exposed these problems.
Running functional tests in TFS 15 with Vnext in comparison to the old system with MTM and Tests Environemnts, it is awefully slow. it takes like 10 minutes after initial test start, before the first tests are started. And while running the tests, they take longer as normal.
Distribution of tests is slightly "unhappy", tests get distributed at the beginning of the test run, but if 1 maschine is finished, while the other one has still 5 long test runs doens't make sense. Bucket size was a way more intelligent system
is there are a way to improve this? We have updated to RC2 and we are not happy with the test outcome. Feeling like test tast is a bottleneck
Ok so case is that the tests get distributed at the start of the test run and the test runner itself works different now. Unlike bucket system that distributes the tests one after another..
Also the tests only shows if failed or finished AFTER they are done, so if 200 tests get distributed it will only show the outcome when all 200 are done.
kinda awkward..
That's orthogonal why but for clarity: I created a TimeMonitor event listener that at the end of the test compares the elapsed time with a policy and fails it if the test takes longer.
It works great with one exception - from time to time the system gets in weird state and some of the tests might take longer because of that. Note my bar for unit tests is 15ms - it is not so hard to happen.
I had this before and the way I solved it was to create a record and wait until the same test exceed the them several times before I fail it. This has several flows - the major one - the need of persisting the data.
I think it will works better if I simply do two (or more) passes. At first pass I collect the tests that exceeded their time and in pass 2-N I repeat them to confirm or reject the problem.
My question is - how. What I need to do (if possible) to programmatically collect a subset of tests and rerun them. Do I need to remove test from testing::UnitTest::GetInstance() or I should create another UnitTest.
A reference to something similar would be great, like retry failed tests for example.
I know the following does not directly answer your question, but I believe that a suggestion of a different approach is justified. I would suggest doing test execution time analysis from a separate process to simplify things and avoid changing the program that runs the tests. This way, you can be certain that you have not influenced the execution time of your tests by inserting additional code that keeps track of tests whose execution time exceeds the threshold you have defined. Also, you won't be needing to modify state of UnitTest objects and other details of googletest implementation, which is harder to understand and potentially dangerous.
Output of the executable that runs your test suite already provides you with execution time for each test. Write a script that runs your test suite executable once and parses that output to determine which tests take too long to execute (this can be easily achieved in some higher level language like Python). Then, if the script has found some tests that are suspicious, it re-runs the test suite executable 2-N times by specifying --gtest_filter command line parameter to it. For example:
tests.exe --gtest_filter=*test1*:*test2*:...:*testN*
This way, only suspicious tests will be re-run and you will be able to determine if some of them is indeed problematic.
If you do not want to use the values provided by googletest, you can modify your TimeMonitor to output the test execution time and parse those values. However, maybe it would be best to remove it and be 100% sure you are not influencing the execution time of the tests.
Hope this helps!
The solution actually is simple (when you know it). Disclaimer not tested with every possible corner case.
in pseudo code:
time monitor -> just observe and create a filter for the long tests
attach time monitor
testing::InitGoogleTest(&argc, argv);
int result = RUN_ALL_TESTS();
if (result == 0 && time_monitor->has too long tests()) {
time monitor -> activate reporting errors
::testing::GTEST_FLAG(filter) = time monitor -> the filter();
result = RUN_ALL_TESTS();
}