I have a unit test written in MSTest with a deployment item decorator to copy a configuration file.
The test itself deletes a line from the configuration file and then verifies the delete occurred correctly.
Now here is the rub.
If I run the test again in ReSharper, it refuses to redeploy the item again, instead leaving the previous copy of the file there, now already missing a line, and the test fails.
If I run the same test again, but using MStest, it passes.
It appears MSTest will always honor [DeploymentItem] even when a build does not occur, but ReSharper will only deploy items if an actual build takes place.
Anyone else seen this?
Using Visual Studio 2012 and ReSharper 7.1.3
Related
A .NET Core MSTest unit test that passes when I run it in Visual Studio is failing when I run it from the command line using the command "dotnet test" (not passing any arguments). I'm getting the error message:
System.NullReferenceException: Object reference not set to an instance of an object
It's failing because a dependency that I've mocked using Moq is null. At first I thought that the issue might have something to do with the test in Visual Studio being run with loose mocks and then for some reason being run with strict mocks from the command line, this turned out to be false as even when I got the test passing in Visual Studio with strict mocks it was still failing from the command line.
Turns out I was being an idiot. The dll I was running against was an outdated file that preceded me changing the name of the test project. Thanks Peter Csala for making me look a bit closer at the unit test that was failing, it made me realise that the failing unit test no longer exists in my code, making me realise the dll was outdated.
Our TFS CI build uses dotCover for running xUnit tests. However some of the tests are failing there but locally they passes.
I downloaded dotCover.CommandLineTools and xunit.runner.console from nuget. unpacked the packages and tried to locally run dotCover cover command as is executed in TFS pipeline:
.\tools\dotcover.exe cover /TargetExecutable=".\..\xunit-runner\tools\net461\xunit.console.exe"
/TargetArguments="c:\Projects\MySolution\Sources\MyTestProject\bin\Debug\MyTestProject.dll -xml c:/temp/testResultFile.xml"
/Output="c:/temp/coverageResultFile.dcvr"
/Filters='-:module=xunit.*;-:module=*UnitTests*'
Get the same errors as in CI. From my investigation of errors that dotCover output showed in failing tests it seems that mock data are not the same when running tests locally from VS and running them with dotCover.
Does anybody faced such problem? Any solution for it?
EDIT: Just tried one more thing, ran command
.\xunit.console c:\Projects\MySolution\Sources\MyTestProject\bin\Debug\MyTestProject.dll and found that this gives me same errors as on CI so the cause of problems is not CI, nor dotCover but xunit.runner.console.
Any new points to this?
According to your description and analysis. This issue seems not related to TFS side(such as configuration of build task/pipeline. Since it's the same result when you directly run dotCover cover command locally as TFS build server.
Take a look at this similar issue check if it's helpful: Unit tests are failing when run from dotCover
It seems that the problem is related to the fact that my build uses an
XML file to hold certain data values. This XML file is found by VSTest
when run under TFS but not by dotCover.
When dotCover runs it creates a TestResults folder which it then
copies all the necessary files to required to run the unit tests. All
files are copied except the XML file. I have set the file to "Copy
always" so can't understand why this file isn't copied. I tried
copying the file manually as a batch file but the folder structure is
created by dotCover so it doesn't exist until I actually run the code
coverage
Another method if it is failed by the file copying: Shadow-copying in dotCover: if your NUnit tests fail during continuous testing .
I'm in the process of moving from on-premise TFS to Visual Studio Online (VSTS) and I'm trying to create a new CI build definition as our local one is xaml based.
For our server based tests I'm using a Visual Studio Test task it it's running fine.
I also need to run Silverlight tests (as we currently have in the local xaml build). Currently I can run my SL unit tests from the command line and output a .trx test results file using the following command:
"C:\Program Files (x86)\StatLight\statlight.exe" -x .\path-to-silverlight.xap --ReportOutputFileType:TRX -r=.\statlight-results.trx
I can also run this using a CommandLine task in vsts (I'm already using a local build agent instead of a hosted one), but I don't know how to:
Get it to fail the build if there's a failing test.
Get the test results into the Test output tab in vsts.
I'll also need to do something similar to fail a build based on js unit tests as we migrate away from Silverlight, so the solution for the SL tests may help with running js tests also.
See if there is a switch that causes the command line to return the number of failed tests. If so you can use the return code to fail your build. Any value other than zero will fail build.
As for test results just add a publish test results task and point to the generated trx file. That is it.
Hi fellow programmers!
I'm trying to do some test coverage using DotCover 2.7 combined with Resharper 7.1.3. I have approximately 1300 tests in four different test projects in my solution, all of which pass when run with resharper. However, if i try to cover these tests using DotCover, one of the projects fails 201 tests that normally pass.
The error is that it tries to load some files from the wrong path, when run through DotCover. I have disabled all sorts of deployment, but resharper still outputs some testfiles to the folder C:\project\Src\TestResults, but these are removed once the tests have completed. When run through dotcover the following path generates the error:
C:\project\Src\project.Test4\bin\Debug\Deploy_abc 2014-07-01 13_26_56\Out\Configuration\Configuration.xml
The file it trying to find is located this folder:
C:\project\Src\project.Test4\bin\Debug\Configuration\Configuration.xml
So it seems that DotCover does not adopt the settings from resharper, or that there is some setting in the project.Test4.csproj-file which specifies this path. I have been through the csproj-file but didn't find anything, which is why i now turn to you guys.
As a bonus i can inform you that DotCover pass the tests if i run them individually, which only confuses me further.
Please advice!
Have you tried to specify path to .testsettings file? On the 'MsTest' page, click 'Use this Test Run Configuration', and then choose a .testsettings file (dotCover | Options).
I'm trying to get my continuous integration setup to run my unit tests for me, so I've set
<RunTest>true</RunTest>
And I've setup the tests to auto-discover:
<ItemGroup>
<TestContainer Include="$(OutDir)%2a.Test.dll" />
</ItemGroup>
Running the build on my development machine runs my unit tests, and reports that the Build succeeded; When run as an automated build, the tests run (according to the BuildLog), but I'm seeing this in the log:
Publishing results of test run tfs_service#TFS-BUILD 2012-04-13 14:13:59_Any CPU_Release to http://mycompany-tfs:8080/Build/v1.0/PublishTestResultsBuildService2.asmx...
.The drop share directory "\tfs-build\Builds\Test\MyProduct Test_20120413.2" could not be accessed.
The drop share directory "\tfs-build\Builds\Test\MyProduct Test_20120413.2" could not be accessed.
MSBUILD : warning MSB6006: "MSTest.exe" exited with code 1.
The previous error was converted to a warning because the task was called with ContinueOnError=true.
This is odd, as the binaries publish quite merrily.
I tried to diagnose the issue, by logging on to the console of the build server, running up a Visual Studio 2008 Command Prompt, navigating to the source location, and invoking MSBuild, it doesn't appear to even attempt to invoke the unit tests?
Ultimately, I'm more interested in playing "guess why the drop of the results failed" - could it be the embedded space in my build definition name? What else could I check, given the binaries publish OK?
The unit tests are running, the problem is that the results are not being published correctly. When unit tests are run, the results are published to the tfs server and then copied to the drop location. Because of this, the drop share needs to allow the TFS build service account to write to this folder.
http://blogs.msdn.com/b/vstsqualitytools/archive/2010/10/24/test-agent-test-controller-and-mstest-faq.aspx