I've encountered an error when running unit tests. If I Debug the unit tests vstest.executionengine.x86.exe runs, then closes when the tests pass.
If I just run the tests (Even if the test is as simple as just creating a new list, with no asserts) vstest.executionengine.x86.exe doesn't close and stays running in task manager.
This is causing an issue for me when it comes to writing more complicated tests that include removing files / cleaning up sqllite databases.
Any help would be appreciated.
EDIT :
Steps to reproduce :
Create New Unit Test Project
Debug Unit Tests - vstest.executionengine.x86 opens and closes, test passes.
Run Unit Tests - vstest.executionengine.x86 opens and stays open
This is by design.
The vstest.executionengine.exe is restarted only when we detect a change in the configuration between two consecutive test runs. This helps ensure we aren't taking a perf hit on process restarts unnecessarily.
Product Update
With VS2013 we have a new menu item under Test -> Test Settings called "Keep Test Execution Engine Running". You can uncheck this to opt out of the default behavior.
I worked around this by using the following as a pre-build event on the affected test projects:
for 64-bit:
taskkill /F /IM vstest.executionengine.exe /FI "MEMUSAGE gt 1"
or for 32-bit:
taskkill /F /IM vstest.executionengine.x86.exe /FI "MEMUSAGE gt 1"
This silently kills the execution engine before building the test project. The /FI "MEMUSAGE gt 1" stops the command (and therefore the build) from failing if the execution engine isn't running.
For what its worth, I ran into this same situation and it turned out that I had a test that did not properly clean up all of its resources. In my specific case there was a background thread with a network connection open that did not get closed before the test exited. Not sure why exiting the test did not close this for me, but when i fixed my code to properly dispose of all the resources I opened, everything worked as expected. I did not have to add any hacks to kill the vstest.executionengine.exe, nor did I have to opt out of Test -> Test Settings -> Keep Test Execution Engine Running
I had this issue when running test using Resharper's test runner which doesn't seem to respect the Test-->Test Settings-->Keep Test Execution Engine Running setting. In my case it was causing the build to fail with the following error:
warning MSB3026: Could not copy "...\SQLite.Interop.dll" to "bin\Debug\x86\SQLite.Interop.dll". Beginning retry 10 in 1000ms. The process cannot access the file 'bin\Debug\x86\SQLite.Interop.dll' because it is being used by another process.
Adding a pre-build event to the test project as #HappyCat suggested worked for me. I also needed to wrap it in an if statement to prevent it from running on the build server and interfering with other jobs.
if $(ConfigurationName) == Debug (
echo "attempting to kill vstest to prevent access denied on sqlite.interop.dll"
taskkill /F /IM vstest.executionengine.x86.exe /FI "MEMUSAGE gt 1"
)
I know this is old but I thought I'd throw in something I just discovered.
A test I was running had some objects in it that implemented IDisposable, so the code analysis told me so should my test class. It took a while to realize it, but when this.Dispose(); was getting called on the implementation of that interface when I put it on my test class, it was actually throwing a StackOverflow exception. So I just yanked the interface and let CA continue to whine.
I did not need to toggle 'Keep Test Execution Engine Running'.
The easiest approach is to go to windows task manager. Look out for vstest.executionengine.exe process running in the background. Kill that process and it should work fine now.
Related
I have a collection of MsTest and NUnit tests that are being run in TFS 2015 using the VS Test step. I'm using NUnit Test Adapter 3.4.1 to run the tests on the build agent.
At the end, even though the tests pass, Nunit seems to croak and the build step fails with these two errors.
2016-09-04T09:59:44.7209787Z ##[error]Error: Exception NUnit.Engine.NUnitEngineException, Exception thrown executing tests
2016-09-04T09:59:44.7209787Z ##[error]
2016-09-04T09:59:44.7209787Z ##[error]Error: Exception encountered unloading AppDomain
2016-09-04T09:59:44.7209787Z ##[error]
2016-09-04T09:59:44.7209787Z Information: NUnit Adapter 3.4.1.0: Test execution complete
2016-09-04T09:59:44.8615975Z Results File: C:\agent\_work2\1\TestResults\SRV-BLD1 2016-09-04 01_22_45.trx
2016-09-04T09:59:44.8615975Z Total tests: 139. Passed: 134. Failed: 0. Skipped: 5.
I've checked that there are indeed 139 tests in the suite, and 5 are set to ignore (2 are MSTest, and 3 are NUnit).
I'm not sure if there's a place to get more detailed explanation of the error. Searching this site and google seems to suggest that the NUnit.Engine.NUnitEngineException is linked to test discovery (here, and here for instance), but my tests are being discovered, so I'm not sure if this is related (Pretty new to NUnit, so not sure of a lot of things).
I also saw two links dealing with Adapter failures (here and here), but the errors don't quite match up, although maybe just because I don't have the same level of verbosity.
In TFS, the step doesn't have any configuration on it, just that it's enabled and the path to the DLLs.
Does anyone know what is causing the errors (which I'm assuming are causing the build to fail)? Alternatively, what should be the next steps in getting a more precise/verbose error stack to investigate the issue?
As a side note, I saw this SO answer, which states this:
MSTest.exe returned an exit code of 1 indicating that not all tests
passed.
I wasn't able to find any confirmation that VSTests fails when it encounters Skipped tests, but could this also be an issue?
Thank you for any help.
UPDATE
As suggested below, I tried running this from the IDE directly, and got this output (folders redacted)
------ Discover test started ------
NUnit Adapter 3.4.1.0: Test discovery starting
NUnit Adapter 3.4.1.0: Test discovery complete
========== Discover test finished: 139 found (0:00:00.8820879) ==========
------ Run test started ------
System.AppDomainUnloadedException: Attempted to access an unloaded AppDomain. This can happen if the test(s) started a thread but did not stop it. Make sure that all the threads started by the test(s) are stopped before completion.
NUnit Adapter 3.4.1.0: Test execution started
Running all tests in C:\agent\_work2\1\s\codePorject\bin\Debug\codeProjectTests.dll
NUnit3TestExecutor converted 37 of 37 NUnit test cases
System.AppDomainUnloadedException: Attempted to access an unloaded AppDomain. This can happen if the test(s) started a thread but did not stop it. Make sure that all the threads started by the test(s) are stopped before completion.
Exception NUnit.Engine.NUnitEngineException, Exception thrown executing tests
Exception encountered unloading AppDomain
NUnit Adapter 3.4.1.0: Test execution complete
========== Run test finished: 139 run (1:20:10.3290294) ==========
I've found a similar xUnit issue open, but it doesn't seem to have a solution.
This StackOverflow answer suggests to use a sleep timer, so I might try that.
In the end, it looks like some sort of a race condition between Firefox browser windows and nUnit. In my cleanup code, I'm killing firefox and iisexpress processes. Adding a sleep call has eliminated the issue:
public static void AssemblyCleanup()
{
foreach (var process in Process.GetProcessesByName("firefox")) process.Kill();
foreach (var process in Process.GetProcessesByName("iisexpress")) process.Kill();
System.Threading.Thread.Sleep(5000);
}
An NUnitEngineException is what it says: an exception that was discovered by the engine. It can be caused by many things and the message indicates the problem. In your case, the message says "Exception encountered unloading AppDomain" which means... well what it says.
The test adapter catches and handles the exception, producing the message you see. There is some indication that TFS also sees the exception and fails test run as a result. If you were to run this under the VS IDE, I think you would see the message from the adapter but the run would not fail. There is an NUnit3 Test Adapter issue about this, but it's not clear if the solution lies within the adapter or if it's a TFS problem.
If you do try this under the IDE I hope you will post the output window text from that run as well.
TL;DR: How can I make CLion kill off any running versions of the code before building the new one?
I've started playing with CLion for C++. I noticed that I often forget to stop the last iteration before trying to compile again, which leads to "Permission denied" errors -- when I first encountered this, it took me almost half an hour of fiddling with permission settings before realizing that it was because the old version was still running, and therefore couldn't be replaced with the new executable.
As far as I can tell, there's no way to do this in CMake without embedding a Batch (since I'm on Windows) script. I'd like to avoid that, because it'd be a lot of unnecessary complexity for... not that much reward.
In short, is there an option in CLion or something in CMake that will stop the previous iteration when running the new one?
Yes there is. Simply press ctrl + F2 or goto Run > Stop to terminate the previous iteration in case it keeps running.
Alternatively, you can set to run only single instance. This way previous instance will always be terminated before running the new one. To enable this, goto Run | Edit Configurations and select Single Instance Only.
As far as I know, this is not possible by default.
One solution I have found was to create a batch file with the following content:
#echo on
tasklist /FI "IMAGENAME eq %1" 2>NUL |find /I "%1">NUL
if "%ERRORLEVEL%"=="0" taskkill /F /im %1
(Second line checks whether the process is running - found here: How to check if a process is running via a batch script)
And edit the build configuration to make CLion call the batch file and pass the processname to it before every build.
This is how it works:
Run > Edit Configurations > select the configuration to change (in my case "Build All")
In the "Before launch: External..."-section click the green plus
Choose "Run external tool" and click the green plus in the pop-up window
Choose a name for the tool
Add the path of the batchfile in the "Program:" field
Write $ProjectName$.exe into the "Parameters:" field
Click ok until you are back in the config window
Give the script a sufficient priority by selecting it and clicking the arrow up
Now it should try to kill the running process before every build.
We are currently setting up our build-process within an automated continious integration environment and facing the fundamental question, when to run unit-tests?
One way would be to run the unit test with every build task. So as soon as one unit-test fails, the whole build fails. This has the advantage, that the developer is always forced to keep the unit-tests green, as s/he is otherwise not able to run the application. On the other hand, you are always distracted by fixing the tests during a development process - which might force you to work in very small iterations. Besides that the time to run your application always increases, as you have to wait for the tests every time.
The other way would be, to let the CI-Server run the tests after each new commit and let the developer simply know, that something went wrong. In this way the developer is pretty free, at what time to care for the unit-tests, but also other developers on the same branch might suffer, because they cannot be sure, that all parts of the software work as expected and have to check theirselves, if the failing tests might also influence their work.
So do you have any best-practices or recommendations, which would be a good time to run the tests?
BTW: of course we also run bigger integration-tests, which are handled in a seperate CI-process.
Short answer: run all unit tests on the build server for every commit, on every branch. Assuming your unit tests don't take a really long time to run, there really is no downside to this. As for running all unit tests on every build task locally, that would be a overkill. Developers should have the discipline to decide when to run the tests and when not to.
You want to know as soon as possible when something is wrong so you can fix it promptly. You also want to know all of the tests that fail rather than just the first test that fails. When there are multiple issues it would be a pretty annoying workflow to only fix the one issue and then have to commit, push, and wait for the build to run again to see if there are more issues.
Your build process should have two targets: build and test. test should be the default target when not specifying anything else. The test can't run until the project was build, so the build target is a dependancy of test. So (suppose use use make): make or make test will build and test. make build will just build the project.
Now, if you're using some IDE, you could consider doing the test in some separate way "outside" of the IDE. So, maybe add a third target ide and let the ide build that one. It could then have the build target as normal dependency and as last step spawn a new job in background to do the testing in it's own terminal window, something like (under linux): ( xterm -e ./run-tests & ).
And if you're developing outside of an ide (like I do), then just have a separate terminal run the build & test. As soon as testing starts, you know the build process finished, so you can run you application already, even thou the tests are still running.
Just to demonstrate this (and as a proof of concept for having the test run in background) I just created some trivial test case:
bodo.c:
#include <stdio.h>
int main(int argc, char * argv[]) {
printf("Hallo %s", argc > 1 ? argv[1] : "Welt");
return 0;
}
Makefile:
test: build run-tests
ide: build run-tests-background
run-tests-background:
( xterm -e ./run-tests --wait & )
run-tests:
./run-tests
build: bodo
bodo: bodo.o
bodo.o: bodo.c
.PHONY: run-tests run-tests-background
run-tests:
#! /bin/sh
retval=true
if test "$(./bodo)" != "Hallo Welt"
then
echo "Test failed []"
retval=false
fi
if test "$(./bodo Bodo)" != "Hallo Bodo"
then
echo "Test failed [Bodo]"
retval=false
fi
if test "$(./bodo Fail)" != "Hallo Bodo"
then
echo "Test failed [Fail]"
retval=false
fi
sleep 5 # Simulate some more tests
if $retval
then
echo "All tests suceeded ;)"
else
echo "Some tests failed :("
fi
if test "$1" == "--wait"
then
read -p "Press ENTER to close" enter
fi
if $retval
then
exit 0
else
exit 2
fi
Usage:
Build the project but do not run the tests
make build
Build the project and do run the test in current terminal
make
Build the project and do run the test in separate terminal. Make will return once the build process completed and the test got started
make ide
And two helpers, which are not supposed to be run by hand:
Only run the tests in current terminal (this will fail, if the project wasn't built yet)
make run-tests
Only run the tests in separate terminal (this will fail, if the project wasn't built yet). Make will return immediatelly
make run-tests-background
I am running tests through Jenkins on a windows box. In my "Execute Windows Batch command" portion of the project configuration I have the following command:
nosetests --nocapture --with-xunitmp --eval-attr "%APPLICATION% and priority<=%PRIORITY% and smoketest and not dev" --processes=4 --process-timeout=2000
The post build actions have "Publish JUnit test result report" with the Test report XMLs path being:
trunk\automation\selenium\src\nosetests.xml
When I do a test run, the nosetests.xml file is created, however it is empty, and I am not getting any Test Results for the build.
I am not really sure what is wrong here.
EDIT 1
I ran the tests with just --with-xunit and REM'd out the --processes and got test results. Does anyone of problems with xunitmp not working with a Windows environment?
EDIT 2
I unstalled an reinstalled nose and nose_xunitmp to no avail.
The nosetest plugin for parallelizing tests and plugin for producing xml output are incompatible. Enabling them at the same time will produce the exact result you got.
If you want to keep using nosetest, you need to execute tests sequentially or find other means of parallelizing them (e.g. by executing multiple parallel nosetest commands (which is what I do at work.))
Alternatively you can use another test runner like nose2 or py.test which do not have this limitation.
Apparently the problem is indeed Windows and how it handles threads. We attempted several tests outside of our Windows Jenkins server and they do not work either. Stupid Windows.
I recently found out about CMake testing possibilities. I wrote several test-clients using it, they work ok, but to perform tests I need to:
cmake .. -> make -> then run my program in the background or other terminal -> make test (which runs all test clients/test scenarios)
Lets say I want the command: make test not only to run the tests, but also to run the executable(That is being tested) in background and kill it after tests complete. How can I pass a bash command via CMakeLists? I haven't found a straightforward way to achieve what I want yet
You can do so by using ADD_CUSTOM_COMMAND. (CMake ADD_CUSTOM_COMMAND docs)
There is not a way to run a process in the background from ctest. To handle this for projects like paraview that use MPI, we write a c driver program that launches the processes and performs the test/tests. Basically each ctest test needs to be something that runs and returns a value. However, there is of course nothing keeping that test from starting and stopping as many processes as possible.