I am noticing that the elapsed time of my unit tests in Visual Studio 2013 Pro are only consistent when they are run via the same command. When the command changes the elapsed times change dramatically.
My specific situation is this: I have 4 passing tests, when I run them all using the "Run All" command in the Test Explorer window I get:
But, when I run those same 4 tests again, this time using the Run Passed Tests command, I get this:
Are these tests run in the sequence that they are listed in the Test Explorer?
Why does test1 take 16ms when I use Run All and then 1ms when I use Run Passing and test2 takes 4ms when I use Run All and 16ms when I use Run Passing?
Related
I am writing pytests for a module and want to debug my way through the tests.
To debug my debugger issue i created a simple test:
def test_something():
pass
which takes 0.001 seconds to run in the terminal.
If i try to debug this test WITH A BREAKPOINT, using vscodes test explorer extension(Holger Benl) the same test takes 40-80 seconds to complete.
If there is NO BREAKPOINT, the test runs in 4 seconds with the debugger (hitting debug).
I cant figure out why this is. The time increase of the terminal to debug run without breakpoint is also huge, but some delay due to vscode running debug is exoected I guess.
Any suggestions?
I'm writing a project to learn how to use Rust and I'm calling my project future-finance-labs. After writing some basic functions and verifying the app can be built I wanted to include some tests, located in aggregates/mod.rs. [The tests are in the same file as the actual code as per the documentation.] I'm unable to get the tests to run despite following the documentation to the best of my ability. I have tried to build the project using PowerShell as well as Bash. [It fails to run on Fedora Linux as well]
Here is my output on Bash:
~/future-finance-labs$ cargo test -- src/formatters/mod.rs
Finished test [unoptimized + debuginfo] target(s) in 5.98s
Running target/debug/deps/future_finance_labs-16ed066e1ea3b9a1
running 0 tests
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out
Using PowerShell I get the same output with some errors like the following:
error: failed to remove C:\Users\jhale\AppData\Local\Packages\CanonicalGroupLimited.UbuntuonWindows_79rhkp1fndgsc\LocalState\rootfs\home\jhale\future-finance-labs\target\debug\build\mime_guess-890328c8763afc22\build_script_build-890328c8763afc22.build_script_build.c22di3i8-cgu.0.rcgu.o: The system cannot find the path specified. (os error 3)
After my initial excitement at the prospect of writing a few tests that passed on the first attempt, I quickly realized all the green was indicative; rather, of a failure to even run the tests. I just want to run the unit tests. Running cargo test alone without a separate and file fails as well. Why can't I run any test in this project with my current setup?
It can't find your test because the rust compiler doesn't know about it. You need to add mod aggregates to main.
mod aggregates;
fn main() {
println!("Hello, world!");
}
After you do that, you'll see that your aggregates/mod.rs doesn't compile for many reasons.
And as Mihir was trying to say, you need to use the name of the test, not the name of the file to run a specific test:
cargo test min_works
cargo test aggregates
See also:
How do I “use” or import a local Rust file?
Rust Book: Controlling How Tests Are Run
At least one of my tests hangs and causes the test run to abort eventually:
Aborting test run: test run timeout of 7200000 milliseconds exceeded.
Results File: D:\TF\2\s\TestResults\xxxxx.trx
Test Run Aborted.
Total tests: Unknown
Passed: 3261
Skipped: 1
Total time: 2,0003 Hours
##[warning]Vstest failed with error. Check logs for failures. There might be failed tests.
I'm using TFS 2018 and the Visual Studio Test task (2.3.28) to run unit tests built with Visual Studio 2019 16.4.5.
I started the test task with the additional parameters: /Blame -- RunConfiguration.TestSessionTimeout=7200000
A few days ago, the test suite ran successfully. It typically finishes within 10 minutes:
Results File: D:\TF\3\s\TestResults\xxxxxx.trx
Test Run Successful.
Total tests: 3336
Passed: 3335
Skipped: 1
Total time: 9.0574 Minutes
I examined the .trx file and found the test that was skipped.
But there was no hint about the tests that were skipped due to the abortion and what test caused the time out.
How do I find the problematic test?
Edit:
When examining the .trx file, display the Start Time column and sort it descending. The tests that most recently started may indicate which DLL may be the culprit. Alas, that doesn't provide a definite answer which test(s) were running when the timeout occurred.
If you mean you want information like the screenshot below, you need to use Visual Studio 2019 to do the tests.
The definition looks like below:
I am running tests through Jenkins on a windows box. In my "Execute Windows Batch command" portion of the project configuration I have the following command:
nosetests --nocapture --with-xunitmp --eval-attr "%APPLICATION% and priority<=%PRIORITY% and smoketest and not dev" --processes=4 --process-timeout=2000
The post build actions have "Publish JUnit test result report" with the Test report XMLs path being:
trunk\automation\selenium\src\nosetests.xml
When I do a test run, the nosetests.xml file is created, however it is empty, and I am not getting any Test Results for the build.
I am not really sure what is wrong here.
EDIT 1
I ran the tests with just --with-xunit and REM'd out the --processes and got test results. Does anyone of problems with xunitmp not working with a Windows environment?
EDIT 2
I unstalled an reinstalled nose and nose_xunitmp to no avail.
The nosetest plugin for parallelizing tests and plugin for producing xml output are incompatible. Enabling them at the same time will produce the exact result you got.
If you want to keep using nosetest, you need to execute tests sequentially or find other means of parallelizing them (e.g. by executing multiple parallel nosetest commands (which is what I do at work.))
Alternatively you can use another test runner like nose2 or py.test which do not have this limitation.
Apparently the problem is indeed Windows and how it handles threads. We attempted several tests outside of our Windows Jenkins server and they do not work either. Stupid Windows.
I've encountered an error when running unit tests. If I Debug the unit tests vstest.executionengine.x86.exe runs, then closes when the tests pass.
If I just run the tests (Even if the test is as simple as just creating a new list, with no asserts) vstest.executionengine.x86.exe doesn't close and stays running in task manager.
This is causing an issue for me when it comes to writing more complicated tests that include removing files / cleaning up sqllite databases.
Any help would be appreciated.
EDIT :
Steps to reproduce :
Create New Unit Test Project
Debug Unit Tests - vstest.executionengine.x86 opens and closes, test passes.
Run Unit Tests - vstest.executionengine.x86 opens and stays open
This is by design.
The vstest.executionengine.exe is restarted only when we detect a change in the configuration between two consecutive test runs. This helps ensure we aren't taking a perf hit on process restarts unnecessarily.
Product Update
With VS2013 we have a new menu item under Test -> Test Settings called "Keep Test Execution Engine Running". You can uncheck this to opt out of the default behavior.
I worked around this by using the following as a pre-build event on the affected test projects:
for 64-bit:
taskkill /F /IM vstest.executionengine.exe /FI "MEMUSAGE gt 1"
or for 32-bit:
taskkill /F /IM vstest.executionengine.x86.exe /FI "MEMUSAGE gt 1"
This silently kills the execution engine before building the test project. The /FI "MEMUSAGE gt 1" stops the command (and therefore the build) from failing if the execution engine isn't running.
For what its worth, I ran into this same situation and it turned out that I had a test that did not properly clean up all of its resources. In my specific case there was a background thread with a network connection open that did not get closed before the test exited. Not sure why exiting the test did not close this for me, but when i fixed my code to properly dispose of all the resources I opened, everything worked as expected. I did not have to add any hacks to kill the vstest.executionengine.exe, nor did I have to opt out of Test -> Test Settings -> Keep Test Execution Engine Running
I had this issue when running test using Resharper's test runner which doesn't seem to respect the Test-->Test Settings-->Keep Test Execution Engine Running setting. In my case it was causing the build to fail with the following error:
warning MSB3026: Could not copy "...\SQLite.Interop.dll" to "bin\Debug\x86\SQLite.Interop.dll". Beginning retry 10 in 1000ms. The process cannot access the file 'bin\Debug\x86\SQLite.Interop.dll' because it is being used by another process.
Adding a pre-build event to the test project as #HappyCat suggested worked for me. I also needed to wrap it in an if statement to prevent it from running on the build server and interfering with other jobs.
if $(ConfigurationName) == Debug (
echo "attempting to kill vstest to prevent access denied on sqlite.interop.dll"
taskkill /F /IM vstest.executionengine.x86.exe /FI "MEMUSAGE gt 1"
)
I know this is old but I thought I'd throw in something I just discovered.
A test I was running had some objects in it that implemented IDisposable, so the code analysis told me so should my test class. It took a while to realize it, but when this.Dispose(); was getting called on the implementation of that interface when I put it on my test class, it was actually throwing a StackOverflow exception. So I just yanked the interface and let CA continue to whine.
I did not need to toggle 'Keep Test Execution Engine Running'.
The easiest approach is to go to windows task manager. Look out for vstest.executionengine.exe process running in the background. Kill that process and it should work fine now.