Does MSpec produce a TRX result file after running tests? If not, can it? The tests are being run with ReSharper.
The feature of ReSharper that you're talking about is just a test runner. It can run tests from MSTest, NUnit, and other frameworks by using custom plugins. MSpec is a completely separate test engine/framework.
MSpec can output a number of reports: TeamCity, HTML, and XML. I can't run MSpec to verify the XML output right now, but I did review the XML report generator code. It appears to write the "concern | context | specification" report that you can see in the HTML example. And not an MSTest TestResults file (.trx).
I don't see any way to output a TRX right now. You can submit an issue or write some kind of generator, but I don't know if it's worthwhile.
Related
VS2017 Community Edition comes with test project templates for Xunit pre-installed. I have a Dotnet Standard 1.4 class library and a Dotnet Core 1.1 XUnit test project. This works well, everything builds, I can run all tests from the Visual Studio Test Explorer, and if my tests write to an ITestOutputHelper the Test Explorer shows an 'Output' link where I can view the output of the tests.
The problem is that if I have a load of tests I don't want to have to click on every test, and then on every 'Output' link in order to view the results. instead I want to see the results in some sort of output file.
The Xunit documentation indicates that it does have the ability to output test results in xml.
http://xunit.github.io/docs/format-xml-v2.html
But it doesn't explain how to get it to do it.
How can I get Xunit to save test output to a file?
xUnit logger can generate xml reports in the xUnit v2 format (https://xunit.github.io/docs/format-xml-v2.html).
Add a reference to the xUnit Logger nuget package in test project.
Use the following command line in tests:
dotnet test --test-adapter-path:. --logger:xunit
Test results are generated in the TestResults directory relative to the test.csproj. A path for the report file can be specified as follows:
dotnet test --test-adapter-path:. --logger:xunit;LogFilePath=loggerFile.xml
loggerFile.xml will be generated in the same directory as test.csproj.
Planning to switch over to the Catch unit test framework for c++.
I need the test results to be in junit report format and I just want to confirm if it has that capability.
Yes, it does.
You can generate a JUnit compatible XML report via adding
-r junit
to the command line of your compiled Catch test suite (cf. Integration with build systems).
A small amount of reading reveals:
Other core features
JUnit xml output is supported for integration with third-party tools, such as CI servers.
I am somewhat new to TeamCity. I have set it up for a Visual Studio solution.
I am using NUnit step to run unit tests and dotCover for code coverage. Problem is, there are particular parts of my code that I know are covered by unit tests but in the code coverage report, it shows them with zero coverage.
I know the DLLs are created just fine because I run a dir /s command and can see all the test DLLs.
The NUnit step uses MSIL 4.0 platform and the test path looks like this:
**/bin/Debug/*Tests.dll
There are no filters under code coverage.
Many tests are run. But there are some specific tests that are not present in the search report when I search.
Why is this happening and how do I fix it?
Can you please provide information about your NUnit step, especially about .Net Coverage (Filters).
Can you also confirm that tests are actually run?
Full build log (pasted in pastebin.com or such provider) would be also appreciated.
Stupid me. The DLL that was being ignored was spelled *Test.dll instead of *Tests.dll. I use *Tests.dll in my test path.
The issue got fixed once I changed it to *Test*.dll
Hi fellow programmers!
I'm trying to do some test coverage using DotCover 2.7 combined with Resharper 7.1.3. I have approximately 1300 tests in four different test projects in my solution, all of which pass when run with resharper. However, if i try to cover these tests using DotCover, one of the projects fails 201 tests that normally pass.
The error is that it tries to load some files from the wrong path, when run through DotCover. I have disabled all sorts of deployment, but resharper still outputs some testfiles to the folder C:\project\Src\TestResults, but these are removed once the tests have completed. When run through dotcover the following path generates the error:
C:\project\Src\project.Test4\bin\Debug\Deploy_abc 2014-07-01 13_26_56\Out\Configuration\Configuration.xml
The file it trying to find is located this folder:
C:\project\Src\project.Test4\bin\Debug\Configuration\Configuration.xml
So it seems that DotCover does not adopt the settings from resharper, or that there is some setting in the project.Test4.csproj-file which specifies this path. I have been through the csproj-file but didn't find anything, which is why i now turn to you guys.
As a bonus i can inform you that DotCover pass the tests if i run them individually, which only confuses me further.
Please advice!
Have you tried to specify path to .testsettings file? On the 'MsTest' page, click 'Use this Test Run Configuration', and then choose a .testsettings file (dotCover | Options).
Is it possible to integrate Hudson with MS Test?
I am setting up a smaller CI server on my development machine with Hudson right now, just so that I can have some statistics (ie. FxCop and compiler warnings). Of course, it would also be nice if it could just run my unit tests and present their output.
Up to now, I have added the following batch task to Hudson, which makes it run the tests properly.
"%PROGRAMFILES%\Microsoft Visual Studio 9.0\Common7\IDE\MSTest.exe" /runconfig:LocalTestRun.testrunconfig /testcontainer:Tests\bin\Debug\Tests.dll
However, as far as I know, Hudson does not support analysis of MS Test results, yet. Does anyone know whether the TRX files generated by MSTest.exe can be transformed to the JUnit or NUnit result format (because those are supported by Hudson), or whether there is any other way to integrate MS Test unit tests with Hudson?
I've been meaning to write this as a guide and develop a plugin but I havent gotten around to it. I know this question is old but I'm SURE someone else out there wants the same thing so here it is.
In the project configuration on Hudson:
Execute Windows batch command
SET MSTest="C:\Program Files\Microsoft Visual Studio 9.0\Common7\IDE\MSTest.exe"
SET XSLParser="C:\MsBuildNunit\msxsl.exe"
SET TestDLL=path-to-your-test-projects.dll
SET TestOutFILE=TestResults\some-unique-filename.trx
SET TransformedOutputFile=%TestOutFILE:.trx=%.xml
SET XSLFile=c:\MsBuildNunit\MSBuild-to-NUnit.xslt
MKDIR TestResults
%MSTest% "/testcontainer:%TestDLL%" /nologo /resultsfile:%TestOutFILE%
%XSLParser% %TestOutFILE% %XSLFile% -o %TransformedOutputFile%
SET ERRORLEVEL=0
Then check the box "Publish NUnit test result report" and for "Test report XMLs" enter
TestResults/*.xml
There is an XSLT in C:\MsBuildNunit as well as msxsl.exe which comes from Microsoft.
You can download the MSBuild-to-NUnit.xslt from here and get msxsl.exe from microsoft here or you can just get the zipped copy of my MsBuildNunit folder that contains the xslt and exe here
When run, it calls MSTest.exe which runs the tests and outputs the format in microsofts trx (xml) format. Then it calls msxsl.exe with the xslt and the trx and translates it to nunits xml format. At the end of the build, Hudson picks it up as any other Nunit test result and you are good to go.
Edited to add:
I forgot to mention, with this xslt we get full test results. We have multiple test projects and multiple dll's and we get great feedback with the ability to trend graph, view the tests by name, view the statuses of the tests, and if it errors we get the error message along with the stack trace. Basically almost everything that you would get with Nunit.
Edit (again): I just now added the test duration in the transform so it will show up in Hudson now! Seems to work great for our tests.
Edit: I tried the new MSTest plugin and it currently does not support parsing multiple TRX files, just 1, so currently this is your only solution if you are like us and have multiple test assemblies that you have to run through MSTest.
Hudson has a new plugin for MSTest. Just specify the location of the .trx file and the work is done for you. It wouldn't surprise me if the plugin used Allen's solution.
I've been able to use a variation of "hangy's" command line, and the MSTest plugin to successfully run and analyze/publish the test cases. The biggest change I made was to specify the output file for mstest.exe and fore the MSTest plugin to consume that file (no wildcards allowed... must be actual filename). For example, the following is my custom build step:
"%PROGRAMFILES%\Microsoft Visual Studio 9.0\Common7\IDE\MSTest.exe" /runconfig:LocalTestRun.testrunconfig /testcontainer:MyProject1.Test/bin/Debug/MyProject1.Test.dll /testcontainer: MyProject2.Test/bin/Debug/MyProject2.Test.dll /resultsfile:TestResults\HudsonJobTestResults.trx
exit 0
Notice that the "results file" is relative to the Job's workspace. Thus, the MSTest plugin's result file to parse is:
TestResults\HudsonJobTestResults.trx
And that's it!
Hudson has a Plot Plugin which can be used to plot generic data. It's not the easiest plugin to configure and use if you have multiple data points per graph, but if you can parse your MS Test output and generate input files for the plugin, you can at the very least plot the trends of failed, successful, and total tests.
I've not been able to use Hudson to perform analysis of MS Test results for historical purposes, but I've at least been able to figure out that if you use MSBuild and the Exec task, the Hudson build will properly be marked as "failed" if any of the tests fail.
<Exec Command=""C:\Program Files (x86)\Microsoft Visual Studio 9.0\Common7\IDE\mstest.exe" /testcontainer:"MyAssembly.dll"" />