How to generate custom build errors in Visual Studio 10 - c++

I'm trying to integrate a custom c++ unit test framework into my build process, and would like the unit tests to run as a final step in the build. Presently I'm executing the unit test in a post-build execution of a batch file, which outputs the test results to the console. If any test fails, the post build step exits with status 1.
This produces a build error in the Error List pane, as desired, but I'm looking for a way to customize the error message content to show that some unit tests failed.
Is that possible?


Yes, in fact this is the method used by the Boost.Test library.
I'm not sure of the exact format requirements but output like:
c:\path\to\file.cpp(line_number) : message
should get added to the error pane in the IDE.

Related

TFS CI build based on code coverage or test pass rate

I'm trying to setup A CI build in TFS. I would like to have this build determine its success based on the code coverage and/test pass rate but I'm having trouble finding instructions on how to do this.
I have a simple node.js project set up with unit tests but currently if any one test fails, the whole build is marked as failed. How can I configure this?

You could use powershell script to do this, do a comparison of the control value(fail or pass build) and the code coverage of the build. If it is less than the control value you could exit with -1 and this would fail the build.
While below blog covers Code coverage percentage as the benchmark to determine the status of build, success of fail.
Controlling build result with code coverage percentage (using build vNext)
And for test pass rate you could use the same method, if any one test fails, the whole build is marked as failed. You could check the option "continue on error" in related test task under control. This will continue run even the task failed.

How to get Debug output from unit tests in TFS 2015

When running unit tests locally in Visual Studio 2015, I can click on the Output hyperlink in the test results and gain access to all the Debug Trace output (as Standard Output) on the test output page.
However when using a build agent to build and test, I can't find any way to gain access to this output information. I've dug through every screen I can think of and nothing. All it shows is the Assert exception message and the stack trace.
Even if I download the .trx file it doesn't include the Console Output section.
Is there any way to get this output from a test run performed by an agent?
Also, some of my tests write additional information to the TestResults folder. The contents of this folder also appears to be excluded from the stored test information. Is there any way to get that as well?
The only other thing I can think of would be to have my tests write all their debug information to files then copy those to another folder as a build step. Seems kind of kludgy. If I remember correctly, the "old" TFS build process would save all this information automatically and it was available looking at test runs in Visual Studio.

Using System.Diagnostics.Trace.WriteLine() instead of System.Diagnostics.Debug.WriteLine(), you will get the information when run the test from TFS:

TFS 2013 after build does not run the unit tests, no errors

I am working on a web app, solution in VS2013 with several project in it, one of them is used for unit tests - Ms Unit test framework. We have TFS 2013 set up to build the solution after each check in and trying to run the unit tests after each build.
From some reason unit tests are not executed after build, but also there is no error indicating that something broken or not found. I tried lot of things regarding editing the build definition - change the 'Test assembly file specification' to ***test*.dll, to use the exact dll name, changed the Target platform ...etc.
If I look into the log file I can see this:
Run VS Test Runner 00:00:15 Inputs TestSpecs: BuildParameter[] Array
Enabled: True ConfigurationsToTest: OutDir: The RunTestsActivity was
invoked without a value for Platform or Flavor. The values Any CPU and
Debug were used.
Test Run Completed. 0 tests executed.
What should I take a look, or what could be the problem ?

"Exception occurred while test discoverer was loading tests"

I'm writing a desktop Windows 8 application, and always get this error when building the project:
An exception occurred while test discoverer 'MSAppContainerTestDiscoverer' was loading tests. Exception: Object reference not set to an instance of an object.
I'm using the C++ unit test framework that's built into Visual Studio 2012, and the error doesn't seem to impact my builds or test runs. I haven't found anything about this online; any ideas what causes this error?

Post a build, it seems VS triggers all installed test-adapters (MSTest, et.all) into a discovery phase where they attempt to refresh the list of tests. If during this process, an exception occurs, the above generic error message is shown.
I faced this with NUnit - but you seem to be having the same with MSAppContainer. I'm not sure what that is. It won't affect your builds but may cause tests to be not found.
The NUnit adapter has been modified to log detailed information to the Output window (Tests) in case of a failure. For MSTest, I think this link should help uncover more details

Integrating Hudson with MS Test?

Is it possible to integrate Hudson with MS Test?
I am setting up a smaller CI server on my development machine with Hudson right now, just so that I can have some statistics (ie. FxCop and compiler warnings). Of course, it would also be nice if it could just run my unit tests and present their output.
Up to now, I have added the following batch task to Hudson, which makes it run the tests properly.
"%PROGRAMFILES%\Microsoft Visual Studio 9.0\Common7\IDE\MSTest.exe" /runconfig:LocalTestRun.testrunconfig /testcontainer:Tests\bin\Debug\Tests.dll
However, as far as I know, Hudson does not support analysis of MS Test results, yet. Does anyone know whether the TRX files generated by MSTest.exe can be transformed to the JUnit or NUnit result format (because those are supported by Hudson), or whether there is any other way to integrate MS Test unit tests with Hudson?

I've been meaning to write this as a guide and develop a plugin but I havent gotten around to it. I know this question is old but I'm SURE someone else out there wants the same thing so here it is.
In the project configuration on Hudson:
Execute Windows batch command
SET MSTest="C:\Program Files\Microsoft Visual Studio 9.0\Common7\IDE\MSTest.exe"
SET XSLParser="C:\MsBuildNunit\msxsl.exe"
SET TestDLL=path-to-your-test-projects.dll
SET TestOutFILE=TestResults\some-unique-filename.trx
SET TransformedOutputFile=%TestOutFILE:.trx=%.xml
SET XSLFile=c:\MsBuildNunit\MSBuild-to-NUnit.xslt
MKDIR TestResults
%MSTest% "/testcontainer:%TestDLL%" /nologo /resultsfile:%TestOutFILE%
%XSLParser% %TestOutFILE% %XSLFile% -o %TransformedOutputFile%
SET ERRORLEVEL=0
Then check the box "Publish NUnit test result report" and for "Test report XMLs" enter
TestResults/*.xml
There is an XSLT in C:\MsBuildNunit as well as msxsl.exe which comes from Microsoft.
You can download the MSBuild-to-NUnit.xslt from here and get msxsl.exe from microsoft here or you can just get the zipped copy of my MsBuildNunit folder that contains the xslt and exe here
When run, it calls MSTest.exe which runs the tests and outputs the format in microsofts trx (xml) format. Then it calls msxsl.exe with the xslt and the trx and translates it to nunits xml format. At the end of the build, Hudson picks it up as any other Nunit test result and you are good to go.
Edited to add:
I forgot to mention, with this xslt we get full test results. We have multiple test projects and multiple dll's and we get great feedback with the ability to trend graph, view the tests by name, view the statuses of the tests, and if it errors we get the error message along with the stack trace. Basically almost everything that you would get with Nunit.
Edit (again): I just now added the test duration in the transform so it will show up in Hudson now! Seems to work great for our tests.
Edit: I tried the new MSTest plugin and it currently does not support parsing multiple TRX files, just 1, so currently this is your only solution if you are like us and have multiple test assemblies that you have to run through MSTest.

Hudson has a new plugin for MSTest. Just specify the location of the .trx file and the work is done for you. It wouldn't surprise me if the plugin used Allen's solution.

I've been able to use a variation of "hangy's" command line, and the MSTest plugin to successfully run and analyze/publish the test cases. The biggest change I made was to specify the output file for mstest.exe and fore the MSTest plugin to consume that file (no wildcards allowed... must be actual filename). For example, the following is my custom build step:
"%PROGRAMFILES%\Microsoft Visual Studio 9.0\Common7\IDE\MSTest.exe" /runconfig:LocalTestRun.testrunconfig /testcontainer:MyProject1.Test/bin/Debug/MyProject1.Test.dll /testcontainer: MyProject2.Test/bin/Debug/MyProject2.Test.dll /resultsfile:TestResults\HudsonJobTestResults.trx
exit 0
Notice that the "results file" is relative to the Job's workspace. Thus, the MSTest plugin's result file to parse is:
TestResults\HudsonJobTestResults.trx
And that's it!

Hudson has a Plot Plugin which can be used to plot generic data. It's not the easiest plugin to configure and use if you have multiple data points per graph, but if you can parse your MS Test output and generate input files for the plugin, you can at the very least plot the trends of failed, successful, and total tests.

I've not been able to use Hudson to perform analysis of MS Test results for historical purposes, but I've at least been able to figure out that if you use MSBuild and the Exec task, the Hudson build will properly be marked as "failed" if any of the tests fail.
<Exec Command=""C:\Program Files (x86)\Microsoft Visual Studio 9.0\Common7\IDE\mstest.exe" /testcontainer:"MyAssembly.dll"" />