We have been using Visual Studio Team Services for about 2 years now with great success. It seems that in recent months something happened to the display of unit test metrics/results. I used to see drill down information for the tests that were published but now that has disappeared. In addition I now see this strange JSON text where unit test information used to be. I've searched high and low for an answer to this and came up empty. Any help would be appreciated.
I can get the same behavior as you when publish the test result via MSTest or VSTest.Console. I have help you submit a feedback on MS Connect Page, you can track it here: Published test result does not show in Build Summary Report.
And for now, you can use the "Publish Test Result" task in vNext build to publish the test result, it can get the test result displayed in build summary.
After switching away from XAML builds to the new build system this problem was resolved. Now a simple Unit test step was added with the appropriate properties setup and it worked like a champ.
Related
Will allure provide a historical report of all the tests that have been run over a long period of time? I am open to other options as well. I am looking for a solution to having a bunch of individual test reports. Previously, Extent Reports has been used but would rather to have all the results in a centralized area that is available to all.
Update: we use Selenium, junit, cucumber, and java
Take a look at Klov.
Docs
Demo
There you have a "Builds" menu where you can see all the reports.
I like it because:
The testcases are updated in real time after execution
I can track progress on different versions
What I don't like:
Screenshots issue
Description and Author are not displayed
Hope it helps!
We have a TFS gated check-in which uses MSTest workflow activity for running our unit tests. We came across some issues recently due to the result folder that MSTest activity creates being too long so some of our unit tests are failing now because of that. It looks it uses a patter like <user>_<machine_name> <date> <time>_<platform>_<config> so we see very lengthy directory names like "tfsbuild_machine123 2015-09-10 10_00_00_Any CPU_Debug". I did some digging into the workflow and its options but couldn't identify where this pattern is coming from. I appreciate if someone can please point me to where this is coming from and how I can change it so we get more room for our unit testing.
I assume that you're referring to test part in the Build Summary page. Like:
As far as I know that the Summary part in the Build Summary page actually is a SummaryFactory type which drives from IBuildDetailFactory, it is not defined in the TFS build process template. The SummaryFactory class contains some functions like CreateSections and CreateNodes which are used to create nodes with on the Summary page, for example, a hyperlink with the format<user>_<machine_name> <date> <time>_<platform>_<config> . However, the SummaryFactory.cs is an internal class so you can't use it in your own program, nor to customize the test hyperlink format.
For your issue, I still would like to check the detailed error message to see what's wrong with it.
I have searched for a while and can't find an answer on here for the first time ever.
I have a solution which contains multiple C# projects. Two of these projects are unit test projects. I am building this solution with Jenkins, attempting to run all the unit tests, then packaging it for an internal NuGet server.
The issue I am having is that I am running the tests for one unit test project after the other, I think this is causing the second set of tests to save their results over the first set because I can see in the console output that both sets of tests ran and passed, however the Test Results in Jenkins only shows the second set of tests.
I am running these with a windows batch command using nunit-console like this:
nunit-console.exe MyFirstProject.UnitTests/bin/Debug/MyFirstProject.UnitTests.dll
nunit-console.exe MySecondProject.UnitTests/bin/Debug/MySecondProject.UnitTests.dll
Is there a better way I can run these so that all test results are recorded?
Thanks!
Figured it out. For anyone who runs into this it's quite simple, you just have to put both (or however many) assemblies in the same command like this:
nunit-console.exe MyFirstProject.UnitTests/bin/Debug/MyFirstProject.UnitTests.dll MySecondProject.UnitTests/bin/Debug/MySecondProject.UnitTests.dll
Now all my test results show up!
As an addition to anyone else who has this type of issue:
To get one report from two groups of tests run from the same assembly, define two Windows Batch Command build steps, each with its own category and result file :
"%nunit%\nunit-console.exe" "%WORKSPACE%\MyTests.sln" /include:TestCategory1 /xml=nunit-result1.xml
and
"%nunit%\nunit-console.exe" "%WORKSPACE%\MyTests.sln" /include:TestCategory2 /xml=nunit-result2.xml
Add a Post Build action to Publish the NUnit Test result report and define the XML filename as *.xml.
The Nunit plugin will concatenate the different test results into one result that can be viewed and emailed.
This way will also solve the OP's question.
We're developing alot of enhancements to salesforce using Visualforce and Apex as part of a larger system, as part of our quality metrics we have to provide a report to management on our Code Coverage.
I'd like to get a report similar to the one produced by Run All Tests in the Force.com IDE but in HTML so I can display it easily via a web interface.
For the rest of our system we use Sonar http://www.sonarsource.org/ to produce the reports.
Does anybody know the best approach to this?
I've explored the API documentation but am unable to find out if the Coverage Percentages is stored against the classes so querying that isn't an option.
Any help or pointers would be greatly appreciated.
If you run the Apex tests yourself via the API there are objects returned indicating which lines hasn't been covered by the tests in that run. You can run the tests via either the synchronous or asynchronous methods.
Then you can use the data to create a report in a format that you require. For example, I've used it to create a basic report in the FuseIT SFDC Explorer (Windows based and free). I'm just dumping out line ranges that aren't covered.
You will probably want to run all the tests in one run to get the complete code coverage of all the tests. For example, in the screenshot above I only ran one out of a much greater number of test classes. As a result it looks like the code coverage was much lower than the cumulative tests give. It does however show which lines an individual test class reaches.
I've also heard good things about MavensMate for the Sublime Text Editor. Being open source you should be able to find how it integrates with the testing api and then generates the reports.
Is there any tool that analyzes test reports of particular unit test runs and shows differences between them? Basically, I'm interested in a "graph of progress":
12 Aug 2012 10:00: 48/50 tests passed. Failed tests: "MyTest13", "MyTest43".
12 Aug 2012 10:02: 47/50 tests passed. "MyTest13" now passed, but "MyTest2" and "MyTest22" started failing.
NUnit is preferrable, however, unit testing framework is not that important.
I'm looking for a completely automated tool, so that I can set it to run it after each build and instantly look at the results and compare them with previous results. The closest thing I've found is nunit-results and a hand-written batch file to call NUnit (with specified xml report path) and nunit-results as a post-build action. However, html file that it produces is not that informative.
I'm really surprised that noone of the popular unit testing software is capable of storing test run information and analyzing series of runs in bulk. I've tried Resharper, NUnit GUI, Gallio and haven't found anything useful.
I would be glad for a solution that does not require a setup of a complicated CI server. My projects are typically small, but I need a tool like this for every one of them.
I don't know what your threshold is for "complicated CI server", but Jenkins is pretty easy to setup, and with the NUnit Plugin ought to give you what you're after:
This plugin makes it possible to import NUnit reports from each build into Jenkins so they are displayed with a trend graph and details about which tests that failed.
If you are interested in a "Graph of progress", I'd go for a way more simple (IMHO) approach and use NCrunch. It shows you your tests status as you code, without stopping for test runs. See my answer here for more details.