I've inherited a codebase which has had some bad check-ins -- some of the unit tests are completely hanging and I can't run the entire unit test suite because it will always get stuck on specific tests. -- I would like to take an inventory of those tests that are now hanging.
What's the right way to set a global timeout on all of my tests such that each one is timeboxed to a specific amount of time. (i.e. if I set it to 1 minute, and a test takes 61 seconds, that test is automatically aborted and marked as failed? -- The test runner should then move on to the next test immediately.)
I'm using Visual Studio 2015 Update 1, NUnit 2.6.4, and the NUnit 2.x Test Adapter for Visual Studio.
I believe it is timeout that you want to use here.
E.g.
[Test, Timeout(2000)]
public void PotentiallyLongRunningTest()
{
...
}
The NUnit documentation seems to indicate it can be set at an assembly level. In AssemblyInfo.cs:
First:
using NUnit.Framework;
Then:
[assembly: Timeout(1000)]
Related
I'm writing some unit tests for a UWP app that I'm working on. I'm using VS 2019. I've noticed that VS 2019 has got a real big, annoying bug. I cannot do a Save All, without it hanging with no recovery, so I have to kill VS 2019 in order to stop it's being hung.
What makes matters worse, for me, is I am so used to doing a Save All that often forget that I shouldn't do that with VS 2019. (BTW, I'm experiencing the same problem at work using VS 2019 Enterprise Edition as I do at home using VS 2019 Community Edition. But I'm not working with the same project types.)
So, I created a new MSTest unit test project, added a reference to the project/class I wanted to test, renamed the new unit test class and created a method. Then I made the fatal mistake of trying to do a Save All, which caused it to fail. It really shouldn't, this unit test is pretty basic:
[TestMethod]
public void WeeklyTest()
{
//assign
var o = new SalaryConvUS();
//act
o.Hourly = 20.0M;
var ret = o.Weekly;
//assert
Assert.AreEqual(800M, ret);
}
But, for some reason which I don't understand, VS 2019 doesn't seem to "see" this new unit test as a unit test. When I brought up the Test Explorer, it didn't recognize WeeklyTest as a unit test. I'm not sure what's going on. How can I fix it?
I have a suite of NUnit tests, some of which fail intermittently, probably because of timing problems. I'd like to find these flaky unit tests. Is there a way to repeat each test multiple times without having to put a Repeat() attribute on each test? We routinely use the resharper and ncrunch runners, but also have access to the nunit gui and console runners.
NUnit 3
In NUnit 3, you may use Retry attribute:
RetryAttribute is used on a test method to specify that it should be
rerun if it fails, up to a maximum number of times.
Notes:
It is not currently possible to use RetryAttribute on a TestFixture or any other type of test suite. Only single tests may be
repeated.
If a test has an unexpected exception, an error result is returned and it is not retried. Only assertion failures can trigger a retry. To
convert an unexpected exception into an assertion failure, see the
ThrowsConstraint.
NUnit 2
NUnit 2 doesn't support retries, but you may use NUnit-retry plug-in (NuGet, GitHub). An example of use:
private static int run = 0;
...
[Test]
[Retry(Times = 3, RequiredPassCount = 2)]
public void One_Failure_On_Three_Should_Pass()
{
run++;
if (run == 1)
{
Assert.Fail();
}
Assert.Pass();
}
See also
Feature - Add 'Retry Attribute' to repeat test upon failure. Discussion about the feature on Launchpad
I have always wondered how TeamCity recognizes that it is running xUnit.net tests and how it knows to put a separate "Test" tab in the build overview after a build step runs. Is the xUnit console runner somehow responsible for that?
Found finally what is actually going on. TeamCity has its own API. I dug this code snippet out of the xUnit source code and it becomes clear:
https://github.com/xunit/xunit/blob/v1/src/xunit.console/RunnerCallbacks/TeamCityRunnerCallback.cs
public override void AssemblyStart(TestAssembly testAssembly)
{
Console.WriteLine(
"##teamcity[testSuiteStarted name='{0}']",
Escape(Path.GetFileName(testAssembly.AssemblyFilename))
);
}
...code omitted for clarity
I have about 300 unit tests for an assembly which is part of a solution I originally started under VS2010. Numerous tests used the Moles framework provided by Micrsoft, but after upgrading to VS2012 (Update 2) I wanted to change the tests to use the officially supplied Fakes framework.
I updated the corresponding tests accordingly, which usually only involved creating a ShimsContext and some minor changes to the code:
Before
[TestMethod]
[HostType( "Moles" )]
public void MyUnitTest_CalledWithXyz_ThrowsException()
{
// Arrange
...
MGroupPrincipal.FindByIdentityPrincipalContextIdentityTypeString =
( t1, t2, t3 ) => null;
...
try
{
// Act
...
}
catch( Exception ex )
{
// Assert
...
}
}
After
[TestMethod]
public void MyUnitTest_CalledWithXyz_ThrowsException()
{
using( ShimsContext.Create() )
{
// Arrange
...
ShimGroupPrincipal.FindByIdentityPrincipalContextIdentityTypeString =
( t1, t2, t3 ) => null;
try
{
// Act
...
}
catch( Exception ex )
{
// Assert
...
}
}
}
I've got different test classes in my test project, and when I run the tests I get arbitrary erros which I cannot explain, e.g.:
Run tests for one class in Release mode => 21 tests fail / 15 pass
Run tests for same class in Debug mode => 2 tests fail / 34 pass
Run tests for same class again in Release mode => 2 tests fail / 34 pass
Run all tests in the project => 21 tests fail / 15 pass (for the class mentioned above)
Same behaviour for a colleague on his system. The error messages are always TypeLoadExceptions such as
Test method ... threw exception: System.TypeLoadException: Could not load type 'System.DirectoryServices.Fakes.ShimDirectorySearcher' in the assembly 'System.DirectoryServices.4.0.0.0.Fakes,Version=4.0.0.0, Culture=neutral, PublicKeyToken=..."
In VS2012 itself the source code editor doesn't show any errors, Intellisense works as expected, mouse tooltips over e.g. ShimDirectorySearcher show where it is located etc. Furthermore, when I open the Fakes assembly that's being generated (e.g. System.DirectoryServices.4.0.0.0.Fakes.dll) with .NET Reflector, the type shown in the error message exists.
All the tests worked fine (in Debug and Release mode as well) before we switched from VS2010 to VS2012, but now we don't have a clue what's wrong here. Why does the result change in the ways described above? Why do we get TypeLoadExceptions even though the types do exist?
Unfortunately there is hardly any help available from Micrsoft or on the internet.
I don't quite understand why having the old .testsettings file from VS2010 is such a problem, but deleting it and adding a .runsettings file as suggested by MSDN did the job for me.
All problems were solved:
All unit tests run (again) without problems
Arbitrary combinations of tests run (again) without problems
I can debug tests using Fakes (before I used to get test instrumentalisation errors)
Hope this helps others who run into problems, there doesn't seem to be too much information about Fakes yet.
One more thing regarding Code Coverage: This works (without having to configure any test settings) via menu Test => Analyze Code Coverage. For TFS build definitions you can enable code coverage for the build by choosing Process => Basic => Automated Tests => 1. Test Source. Now click into the corresponding text field and then on the ... button that is (only) shown when you click into the text field. Now choose Visual Studio Test Runner in the Test runner ComboBox. Now you can also choose Enable Code Coverage from the options.
I'm using Resharper 4.5 with Visual Studio 2008 and MBUnit testing, and there seems to be something odd with using ReSharpher to run the tests.
On the side there are the icons beside the class each test method with the options Run and Debug. When I select Run it just shows me the results of the single test. However I noticed that the test was taking a considerably long time to run.
When I ran Sql Server profiler and start stepping through the code, I realized that its not just running the selected test, but every single one in the class. Is there any reason it makes it look like its only running one unit test while actually running them all?
Its getting to be a pain waiting for all integration tests to run when I only care about the reuslt of one, is there any way to change this?
I just encountered this today and I think I might have realized what causes this bug, I had my methods named similarly
[TestMethod]
public void TestSomething()
[TestMethod]
public void TestSomethingPart2()
I saw that running TestSomething() would run both, however running TestSomethingPart2() would not. I concluded if you name methods that an exact match can occur for the method name it will run the test. After renaming my second test to TestPart2Something this issue went away.
I can confirm that this is a problem with ReSharper 5.1.
To reproduce run test A from my sample code below (all tests will execute); run test AB (all except A will execute); etc:
[TestMethod]
public void A()
{
Console.WriteLine("A");
}
[TestMethod]
public void AB()
{
Console.WriteLine("AB");
}
[TestMethod]
public void ABC()
{
Console.WriteLine("ABC");
}
[TestMethod]
public void ABCD()
{
Console.WriteLine("ABCD");
}
[TestMethod]
public void ABCDE()
{
Console.WriteLine("ABCDE");
}
It took me ages to work this out. I had the remote debugger attached to a development server, and it was breaking a bit more often than I was expecting it to...
It seems to be doing a StartsWith instead of a Contains as others have said.
The workaround is to not have test method names that start with the name of another test method name.
I hope this shows up under Chris post.
I had a similar situation that confirms the behavior he noticed.
[TestMethod()]
public void ArchiveAccountTest()
[TestMethod()]
public void ArchiveAccountTestRestore()
So running the first method would execute both and running the second would not. Renamed my second method to TestRestore and the problem went away.
Note: I'm using Resharper 5.1 so it's still a problem.
When you right-click in the editor, the context menu appears from which you can run and debug tests. Right-click inside a test method to run or debug that single test. Right-click outside of any test method to run or debug the entire test class contained in the current file.
The current release of Gallio includes a Unit Test runner with MbUnit (and NUnit) support built-in.
From the Resharper menu, you have the option of running a Single unit test or all Tests in your solution. What is cool, is that the Keyboard-shortcuts for this are:
Alt + R, U, R - Run test from current context (if you are at a [Test] level, it runs one test, if you are at a [TestFixture] level, it runs all in the fixture!)
Alt + R, U, N - Runs all Unit Tests in your Solution
I highly recommend that you uninstall your current Gallio and then check C:\Program Files\Jetbrains\Resharper\plugins\bin and clear out and files there. Then install Gallio afresh.
Once you've done this, you should startup VS2008 and goto at the Resharper | Plugins menu to check that the Gallio plugin is active. This will give you support for MbUnit.