Our TFS CI build uses dotCover for running xUnit tests. However some of the tests are failing there but locally they passes.
I downloaded dotCover.CommandLineTools and xunit.runner.console from nuget. unpacked the packages and tried to locally run dotCover cover command as is executed in TFS pipeline:
.\tools\dotcover.exe cover /TargetExecutable=".\..\xunit-runner\tools\net461\xunit.console.exe"
/TargetArguments="c:\Projects\MySolution\Sources\MyTestProject\bin\Debug\MyTestProject.dll -xml c:/temp/testResultFile.xml"
/Output="c:/temp/coverageResultFile.dcvr"
/Filters='-:module=xunit.*;-:module=*UnitTests*'
Get the same errors as in CI. From my investigation of errors that dotCover output showed in failing tests it seems that mock data are not the same when running tests locally from VS and running them with dotCover.
Does anybody faced such problem? Any solution for it?
EDIT: Just tried one more thing, ran command
.\xunit.console c:\Projects\MySolution\Sources\MyTestProject\bin\Debug\MyTestProject.dll and found that this gives me same errors as on CI so the cause of problems is not CI, nor dotCover but xunit.runner.console.
Any new points to this?
According to your description and analysis. This issue seems not related to TFS side(such as configuration of build task/pipeline. Since it's the same result when you directly run dotCover cover command locally as TFS build server.
Take a look at this similar issue check if it's helpful: Unit tests are failing when run from dotCover
It seems that the problem is related to the fact that my build uses an
XML file to hold certain data values. This XML file is found by VSTest
when run under TFS but not by dotCover.
When dotCover runs it creates a TestResults folder which it then
copies all the necessary files to required to run the unit tests. All
files are copied except the XML file. I have set the file to "Copy
always" so can't understand why this file isn't copied. I tried
copying the file manually as a batch file but the folder structure is
created by dotCover so it doesn't exist until I actually run the code
coverage
Another method if it is failed by the file copying: Shadow-copying in dotCover: if your NUnit tests fail during continuous testing .
Related
I'm in the process of moving from on-premise TFS to Visual Studio Online (VSTS) and I'm trying to create a new CI build definition as our local one is xaml based.
For our server based tests I'm using a Visual Studio Test task it it's running fine.
I also need to run Silverlight tests (as we currently have in the local xaml build). Currently I can run my SL unit tests from the command line and output a .trx test results file using the following command:
"C:\Program Files (x86)\StatLight\statlight.exe" -x .\path-to-silverlight.xap --ReportOutputFileType:TRX -r=.\statlight-results.trx
I can also run this using a CommandLine task in vsts (I'm already using a local build agent instead of a hosted one), but I don't know how to:
Get it to fail the build if there's a failing test.
Get the test results into the Test output tab in vsts.
I'll also need to do something similar to fail a build based on js unit tests as we migrate away from Silverlight, so the solution for the SL tests may help with running js tests also.
See if there is a switch that causes the command line to return the number of failed tests. If so you can use the return code to fail your build. Any value other than zero will fail build.
As for test results just add a publish test results task and point to the generated trx file. That is it.
I have a simple application and Nunit unit test project with 2 tests to test that app.
I've managed to have unit tests discovered and run locally in Visual studio 2013 but when trying to build application and run these unit tests on TFS 2013 test runner does not find tests ( both projects are built successfully.
No test found. Make sure that installed test discoverers & executors, platform & framework version settings are appropriate and try again.
I've added "NUnit adapter with framework" package to Unit tests project according to answer in this question and committed changes to the TFS.
Unit tests projects is successfully built but still no tests are found. The Test dll fits the **\*Test*.dll naming pattern as configured in the Build Definition.
Why no tests are found? How I can make sure Test Runner is aware of my NUnit test adapter?
If you go to the build, and look in diagnostics (from the web portal), there is a Run VS Test Runner section.
In the same log, there is a Run MSBuild section, it should say "Successfully installed xxxx" where xxx is the name of your adapter package.
Something like:
Check to see that the test runner is successfully restored.
Found the reason - build agent folder in source settings of Build definition was not set correctly - instead of setting it somewhere under build agent working folder I set it outside of it that's why unit tests were not found..
I discovered this using Build process activity log (which was saved in drop folder in my case).this log contains very detailed information (much more than you can see in Visual studio) - there I found in which location test runner is looking for unit tests and then understood what i did wrong.
That's what I saw in log:
Run VS Test Runner00:00:00
Inputs
TestSpecs: BuildParameter[] Array
Enabled: True
ConfigurationsToTest: String[] Array
OutDir:
There were no matches for the search pattern C:\Builds\11\XXXX\Application -Gated\bin\**\*Test*.dll
Hope this will be helpful to someone else.
thanks to everyone who responded to my question!
Hi fellow programmers!
I'm trying to do some test coverage using DotCover 2.7 combined with Resharper 7.1.3. I have approximately 1300 tests in four different test projects in my solution, all of which pass when run with resharper. However, if i try to cover these tests using DotCover, one of the projects fails 201 tests that normally pass.
The error is that it tries to load some files from the wrong path, when run through DotCover. I have disabled all sorts of deployment, but resharper still outputs some testfiles to the folder C:\project\Src\TestResults, but these are removed once the tests have completed. When run through dotcover the following path generates the error:
C:\project\Src\project.Test4\bin\Debug\Deploy_abc 2014-07-01 13_26_56\Out\Configuration\Configuration.xml
The file it trying to find is located this folder:
C:\project\Src\project.Test4\bin\Debug\Configuration\Configuration.xml
So it seems that DotCover does not adopt the settings from resharper, or that there is some setting in the project.Test4.csproj-file which specifies this path. I have been through the csproj-file but didn't find anything, which is why i now turn to you guys.
As a bonus i can inform you that DotCover pass the tests if i run them individually, which only confuses me further.
Please advice!
Have you tried to specify path to .testsettings file? On the 'MsTest' page, click 'Use this Test Run Configuration', and then choose a .testsettings file (dotCover | Options).
I'm using tfs 2010 and I want to create a tfsbuild defintion that runs unit tests.
I created unit tests and it works perfectly on my local machine.
I created a build definition, enabled tests, opened automated tests and put the vsmdi file.
the build succeed but it does not run any tests. In the build log file I don't find any information about the number of tests ran, it just tell me no test results ,besides the TestResults Directory is empty.
Has any one an idea.
I finally found the Problem. The build platform is x86 but for the test project it's AnyCPU, that's why tfsbuild did not execute tests.
I'm trying to get my continuous integration setup to run my unit tests for me, so I've set
<RunTest>true</RunTest>
And I've setup the tests to auto-discover:
<ItemGroup>
<TestContainer Include="$(OutDir)%2a.Test.dll" />
</ItemGroup>
Running the build on my development machine runs my unit tests, and reports that the Build succeeded; When run as an automated build, the tests run (according to the BuildLog), but I'm seeing this in the log:
Publishing results of test run tfs_service#TFS-BUILD 2012-04-13 14:13:59_Any CPU_Release to http://mycompany-tfs:8080/Build/v1.0/PublishTestResultsBuildService2.asmx...
.The drop share directory "\tfs-build\Builds\Test\MyProduct Test_20120413.2" could not be accessed.
The drop share directory "\tfs-build\Builds\Test\MyProduct Test_20120413.2" could not be accessed.
MSBUILD : warning MSB6006: "MSTest.exe" exited with code 1.
The previous error was converted to a warning because the task was called with ContinueOnError=true.
This is odd, as the binaries publish quite merrily.
I tried to diagnose the issue, by logging on to the console of the build server, running up a Visual Studio 2008 Command Prompt, navigating to the source location, and invoking MSBuild, it doesn't appear to even attempt to invoke the unit tests?
Ultimately, I'm more interested in playing "guess why the drop of the results failed" - could it be the embedded space in my build definition name? What else could I check, given the binaries publish OK?
The unit tests are running, the problem is that the results are not being published correctly. When unit tests are run, the results are published to the tfs server and then copied to the drop location. Because of this, the drop share needs to allow the TFS build service account to write to this folder.
http://blogs.msdn.com/b/vstsqualitytools/archive/2010/10/24/test-agent-test-controller-and-mstest-faq.aspx