Hi fellow programmers!
I'm trying to do some test coverage using DotCover 2.7 combined with Resharper 7.1.3. I have approximately 1300 tests in four different test projects in my solution, all of which pass when run with resharper. However, if i try to cover these tests using DotCover, one of the projects fails 201 tests that normally pass.
The error is that it tries to load some files from the wrong path, when run through DotCover. I have disabled all sorts of deployment, but resharper still outputs some testfiles to the folder C:\project\Src\TestResults, but these are removed once the tests have completed. When run through dotcover the following path generates the error:
C:\project\Src\project.Test4\bin\Debug\Deploy_abc 2014-07-01 13_26_56\Out\Configuration\Configuration.xml
The file it trying to find is located this folder:
C:\project\Src\project.Test4\bin\Debug\Configuration\Configuration.xml
So it seems that DotCover does not adopt the settings from resharper, or that there is some setting in the project.Test4.csproj-file which specifies this path. I have been through the csproj-file but didn't find anything, which is why i now turn to you guys.
As a bonus i can inform you that DotCover pass the tests if i run them individually, which only confuses me further.
Please advice!
Have you tried to specify path to .testsettings file? On the 'MsTest' page, click 'Use this Test Run Configuration', and then choose a .testsettings file (dotCover | Options).
Related
Our TFS CI build uses dotCover for running xUnit tests. However some of the tests are failing there but locally they passes.
I downloaded dotCover.CommandLineTools and xunit.runner.console from nuget. unpacked the packages and tried to locally run dotCover cover command as is executed in TFS pipeline:
.\tools\dotcover.exe cover /TargetExecutable=".\..\xunit-runner\tools\net461\xunit.console.exe"
/TargetArguments="c:\Projects\MySolution\Sources\MyTestProject\bin\Debug\MyTestProject.dll -xml c:/temp/testResultFile.xml"
/Output="c:/temp/coverageResultFile.dcvr"
/Filters='-:module=xunit.*;-:module=*UnitTests*'
Get the same errors as in CI. From my investigation of errors that dotCover output showed in failing tests it seems that mock data are not the same when running tests locally from VS and running them with dotCover.
Does anybody faced such problem? Any solution for it?
EDIT: Just tried one more thing, ran command
.\xunit.console c:\Projects\MySolution\Sources\MyTestProject\bin\Debug\MyTestProject.dll and found that this gives me same errors as on CI so the cause of problems is not CI, nor dotCover but xunit.runner.console.
Any new points to this?
According to your description and analysis. This issue seems not related to TFS side(such as configuration of build task/pipeline. Since it's the same result when you directly run dotCover cover command locally as TFS build server.
Take a look at this similar issue check if it's helpful: Unit tests are failing when run from dotCover
It seems that the problem is related to the fact that my build uses an
XML file to hold certain data values. This XML file is found by VSTest
when run under TFS but not by dotCover.
When dotCover runs it creates a TestResults folder which it then
copies all the necessary files to required to run the unit tests. All
files are copied except the XML file. I have set the file to "Copy
always" so can't understand why this file isn't copied. I tried
copying the file manually as a batch file but the folder structure is
created by dotCover so it doesn't exist until I actually run the code
coverage
Another method if it is failed by the file copying: Shadow-copying in dotCover: if your NUnit tests fail during continuous testing .
I'm in the process of moving from on-premise TFS to Visual Studio Online (VSTS) and I'm trying to create a new CI build definition as our local one is xaml based.
For our server based tests I'm using a Visual Studio Test task it it's running fine.
I also need to run Silverlight tests (as we currently have in the local xaml build). Currently I can run my SL unit tests from the command line and output a .trx test results file using the following command:
"C:\Program Files (x86)\StatLight\statlight.exe" -x .\path-to-silverlight.xap --ReportOutputFileType:TRX -r=.\statlight-results.trx
I can also run this using a CommandLine task in vsts (I'm already using a local build agent instead of a hosted one), but I don't know how to:
Get it to fail the build if there's a failing test.
Get the test results into the Test output tab in vsts.
I'll also need to do something similar to fail a build based on js unit tests as we migrate away from Silverlight, so the solution for the SL tests may help with running js tests also.
See if there is a switch that causes the command line to return the number of failed tests. If so you can use the return code to fail your build. Any value other than zero will fail build.
As for test results just add a publish test results task and point to the generated trx file. That is it.
I've got a TeamCity build configuration with two steps currently:
MSBuild
Execute powershell sciprt
I want to add another one:
Run QUnit tests
Inside Visual Studio I use Chutzpah to run my QUnit tests. It seems that there is no "Runner type" appropriate though for QUnit tests, and because I do see one for NUnit I'm left wondering if I'm looking at this right.
Should I get a TeamCity extension to add a "Runner type" for QUnit tests? Should I create a powershell script that runs chutzpah-console? And if so how should I make sure the test output is picked up by TeamCity (do I need the /nunit2 or /junit flag?)?
I've done some research, allow me to share...
I've applied my Google-fu but that leads to a blog post with complicated QUnit tests with special TeamCity messages (or perhaps I misunderstood?) and a blog post on using a command-line build step to call PhantomJS that uses a rather contrived html file that seems like overkill to me if I'm already using Chutzpah in Visual Studio anyways (or perhaps I misunderstood)?
I've gone through the Chutzpah Full Documentation but as far as CI goes it only has a section on TFS Build, nothing on TeamCity or CI in general. I've carefully looked through the command line options documentation, and it does have one relevant option:
/teamcity :Forces TeamCity mode (normally auto-detected)
This suggests that you normally don't need to do anything to get things working, but that's certainly not the case: my sln (or better: csproj) contains several js files with QUnit tests, but TeamCity shows no tests.
So, how do I get TeamCity to run QUnit tests in the (c)leanest way possible / how do I use Chutzpah in a TeamCity build step?
No need for "service messges" or custom html files that go into a special PhantomJS call if you're already using Chutzpah. Here's one (other) way to do it in a (c)lean way.
There is no specific runner type for this type of build step. But no need for a custom Powershell script either: it'll be one line of script only. You can just use a Command Line step to invoke Chutzpah. To be more complete, assuming you've got TeamCity running already on a Windows machine, follow these steps:
Install chutzpah.console.exe. The easiest way is probably using chocolatey to install the Chutzpah package using this command in Powershell:
choco install chutzpah
This installs Chutzpah and makes sure the exe is in your path (you may need to reboot TeamCity parts to get this noticed!).
Note that alternatively you could also manually get chutzpah.console.exe on your CI server somehow (e.g. grab it from your dev machine) and reference it with a full path or place it in your path yourself.
Add a Command Line build step with a custom script like this:
chutzpah.console.exe "MyProject/UnitTestsSubDir"
E.g.:
And by some black magic this is all it takes: your tests will be run (see the build log) and TeamCity will pick up the test results.
I created a test framework to test a sample application. This basically required creating a Solution for my sample application and then adding a Native Test Project, which can test the application.
The test project generates an .exe, but if I were to execute the .exe on it's own, it does nothing.
Is there a way to get information from the test programme so it can be used for other purposes outside of the VS environment?
Yes, you can run unit tests without VS2013.
Use Run automated tests from the command line using MSTest source to do so.
In general it says:
Open a Visual Studio command prompt.
Either change directory to your solution folder or, when you run the MSTest.exe program in step 3, specify a full or relative path to the metadata file or to the test container.
Run the MSTest.exe program.
I hope it will solve your issue.
It took a couple of days, but I've found that DeJaVo's answer is incorrect. Because I am running a Native Test Project, MSTest will not work with it. To run a Native Test from the command line, use VSTest.Console.
Is it possible to integrate Hudson with MS Test?
I am setting up a smaller CI server on my development machine with Hudson right now, just so that I can have some statistics (ie. FxCop and compiler warnings). Of course, it would also be nice if it could just run my unit tests and present their output.
Up to now, I have added the following batch task to Hudson, which makes it run the tests properly.
"%PROGRAMFILES%\Microsoft Visual Studio 9.0\Common7\IDE\MSTest.exe" /runconfig:LocalTestRun.testrunconfig /testcontainer:Tests\bin\Debug\Tests.dll
However, as far as I know, Hudson does not support analysis of MS Test results, yet. Does anyone know whether the TRX files generated by MSTest.exe can be transformed to the JUnit or NUnit result format (because those are supported by Hudson), or whether there is any other way to integrate MS Test unit tests with Hudson?
I've been meaning to write this as a guide and develop a plugin but I havent gotten around to it. I know this question is old but I'm SURE someone else out there wants the same thing so here it is.
In the project configuration on Hudson:
Execute Windows batch command
SET MSTest="C:\Program Files\Microsoft Visual Studio 9.0\Common7\IDE\MSTest.exe"
SET XSLParser="C:\MsBuildNunit\msxsl.exe"
SET TestDLL=path-to-your-test-projects.dll
SET TestOutFILE=TestResults\some-unique-filename.trx
SET TransformedOutputFile=%TestOutFILE:.trx=%.xml
SET XSLFile=c:\MsBuildNunit\MSBuild-to-NUnit.xslt
MKDIR TestResults
%MSTest% "/testcontainer:%TestDLL%" /nologo /resultsfile:%TestOutFILE%
%XSLParser% %TestOutFILE% %XSLFile% -o %TransformedOutputFile%
SET ERRORLEVEL=0
Then check the box "Publish NUnit test result report" and for "Test report XMLs" enter
TestResults/*.xml
There is an XSLT in C:\MsBuildNunit as well as msxsl.exe which comes from Microsoft.
You can download the MSBuild-to-NUnit.xslt from here and get msxsl.exe from microsoft here or you can just get the zipped copy of my MsBuildNunit folder that contains the xslt and exe here
When run, it calls MSTest.exe which runs the tests and outputs the format in microsofts trx (xml) format. Then it calls msxsl.exe with the xslt and the trx and translates it to nunits xml format. At the end of the build, Hudson picks it up as any other Nunit test result and you are good to go.
Edited to add:
I forgot to mention, with this xslt we get full test results. We have multiple test projects and multiple dll's and we get great feedback with the ability to trend graph, view the tests by name, view the statuses of the tests, and if it errors we get the error message along with the stack trace. Basically almost everything that you would get with Nunit.
Edit (again): I just now added the test duration in the transform so it will show up in Hudson now! Seems to work great for our tests.
Edit: I tried the new MSTest plugin and it currently does not support parsing multiple TRX files, just 1, so currently this is your only solution if you are like us and have multiple test assemblies that you have to run through MSTest.
Hudson has a new plugin for MSTest. Just specify the location of the .trx file and the work is done for you. It wouldn't surprise me if the plugin used Allen's solution.
I've been able to use a variation of "hangy's" command line, and the MSTest plugin to successfully run and analyze/publish the test cases. The biggest change I made was to specify the output file for mstest.exe and fore the MSTest plugin to consume that file (no wildcards allowed... must be actual filename). For example, the following is my custom build step:
"%PROGRAMFILES%\Microsoft Visual Studio 9.0\Common7\IDE\MSTest.exe" /runconfig:LocalTestRun.testrunconfig /testcontainer:MyProject1.Test/bin/Debug/MyProject1.Test.dll /testcontainer: MyProject2.Test/bin/Debug/MyProject2.Test.dll /resultsfile:TestResults\HudsonJobTestResults.trx
exit 0
Notice that the "results file" is relative to the Job's workspace. Thus, the MSTest plugin's result file to parse is:
TestResults\HudsonJobTestResults.trx
And that's it!
Hudson has a Plot Plugin which can be used to plot generic data. It's not the easiest plugin to configure and use if you have multiple data points per graph, but if you can parse your MS Test output and generate input files for the plugin, you can at the very least plot the trends of failed, successful, and total tests.
I've not been able to use Hudson to perform analysis of MS Test results for historical purposes, but I've at least been able to figure out that if you use MSBuild and the Exec task, the Hudson build will properly be marked as "failed" if any of the tests fail.
<Exec Command=""C:\Program Files (x86)\Microsoft Visual Studio 9.0\Common7\IDE\mstest.exe" /testcontainer:"MyAssembly.dll"" />