I'm trying to setup A CI build in TFS. I would like to have this build determine its success based on the code coverage and/test pass rate but I'm having trouble finding instructions on how to do this.
I have a simple node.js project set up with unit tests but currently if any one test fails, the whole build is marked as failed. How can I configure this?
You could use powershell script to do this, do a comparison of the control value(fail or pass build) and the code coverage of the build. If it is less than the control value you could exit with -1 and this would fail the build.
While below blog covers Code coverage percentage as the benchmark to determine the status of build, success of fail.
Controlling build result with code coverage percentage (using build vNext)
And for test pass rate you could use the same method, if any one test fails, the whole build is marked as failed. You could check the option "continue on error" in related test task under control. This will continue run even the task failed.
Related
I am working on a web app, solution in VS2013 with several project in it, one of them is used for unit tests - Ms Unit test framework. We have TFS 2013 set up to build the solution after each check in and trying to run the unit tests after each build.
From some reason unit tests are not executed after build, but also there is no error indicating that something broken or not found. I tried lot of things regarding editing the build definition - change the 'Test assembly file specification' to ***test*.dll, to use the exact dll name, changed the Target platform ...etc.
If I look into the log file I can see this:
Run VS Test Runner 00:00:15 Inputs TestSpecs: BuildParameter[] Array
Enabled: True ConfigurationsToTest: OutDir: The RunTestsActivity was
invoked without a value for Platform or Flavor. The values Any CPU and
Debug were used.
Test Run Completed. 0 tests executed.
What should I take a look, or what could be the problem ?
I want to get rid of the dialog saying
There were build errors. Would you like to continue and run tests from
the last successful build
when I am running unit tests (I use test -> run -> all tests).
How do I do that? I already know how to disable it when running a normal project.
I also want to know how this can ever be a useful feature?
How do I do that?
You can't.
I also want to know how this can ever be a useful feature?
I find it useful when I am working closely with someone who is configuring test data. I can re-run tests to ensure new test data is valid, without having to worry about compiling any changes that I have done in the meantime.
For example if someone has changed some data in a database, I want to be able to run my tests to ensure that this new data is valid, and I want to be able to run the tests whether the current state of my code compiles or not.
For Visual Studio 2017 / MSTestV2 you can do:
Tools > Options > Project and Solutions > Build and Run
"On Run, when build or deployment errors occur:" "Do not launch"
I'm trying to get my continuous integration setup to run my unit tests for me, so I've set
<RunTest>true</RunTest>
And I've setup the tests to auto-discover:
<ItemGroup>
<TestContainer Include="$(OutDir)%2a.Test.dll" />
</ItemGroup>
Running the build on my development machine runs my unit tests, and reports that the Build succeeded; When run as an automated build, the tests run (according to the BuildLog), but I'm seeing this in the log:
Publishing results of test run tfs_service#TFS-BUILD 2012-04-13 14:13:59_Any CPU_Release to http://mycompany-tfs:8080/Build/v1.0/PublishTestResultsBuildService2.asmx...
.The drop share directory "\tfs-build\Builds\Test\MyProduct Test_20120413.2" could not be accessed.
The drop share directory "\tfs-build\Builds\Test\MyProduct Test_20120413.2" could not be accessed.
MSBUILD : warning MSB6006: "MSTest.exe" exited with code 1.
The previous error was converted to a warning because the task was called with ContinueOnError=true.
This is odd, as the binaries publish quite merrily.
I tried to diagnose the issue, by logging on to the console of the build server, running up a Visual Studio 2008 Command Prompt, navigating to the source location, and invoking MSBuild, it doesn't appear to even attempt to invoke the unit tests?
Ultimately, I'm more interested in playing "guess why the drop of the results failed" - could it be the embedded space in my build definition name? What else could I check, given the binaries publish OK?
The unit tests are running, the problem is that the results are not being published correctly. When unit tests are run, the results are published to the tfs server and then copied to the drop location. Because of this, the drop share needs to allow the TFS build service account to write to this folder.
http://blogs.msdn.com/b/vstsqualitytools/archive/2010/10/24/test-agent-test-controller-and-mstest-faq.aspx
I'm trying to integrate a custom c++ unit test framework into my build process, and would like the unit tests to run as a final step in the build. Presently I'm executing the unit test in a post-build execution of a batch file, which outputs the test results to the console. If any test fails, the post build step exits with status 1.
This produces a build error in the Error List pane, as desired, but I'm looking for a way to customize the error message content to show that some unit tests failed.
Is that possible?
Yes, in fact this is the method used by the Boost.Test library.
I'm not sure of the exact format requirements but output like:
c:\path\to\file.cpp(line_number) : message
should get added to the error pane in the IDE.
Is it possible to integrate Hudson with MS Test?
I am setting up a smaller CI server on my development machine with Hudson right now, just so that I can have some statistics (ie. FxCop and compiler warnings). Of course, it would also be nice if it could just run my unit tests and present their output.
Up to now, I have added the following batch task to Hudson, which makes it run the tests properly.
"%PROGRAMFILES%\Microsoft Visual Studio 9.0\Common7\IDE\MSTest.exe" /runconfig:LocalTestRun.testrunconfig /testcontainer:Tests\bin\Debug\Tests.dll
However, as far as I know, Hudson does not support analysis of MS Test results, yet. Does anyone know whether the TRX files generated by MSTest.exe can be transformed to the JUnit or NUnit result format (because those are supported by Hudson), or whether there is any other way to integrate MS Test unit tests with Hudson?
I've been meaning to write this as a guide and develop a plugin but I havent gotten around to it. I know this question is old but I'm SURE someone else out there wants the same thing so here it is.
In the project configuration on Hudson:
Execute Windows batch command
SET MSTest="C:\Program Files\Microsoft Visual Studio 9.0\Common7\IDE\MSTest.exe"
SET XSLParser="C:\MsBuildNunit\msxsl.exe"
SET TestDLL=path-to-your-test-projects.dll
SET TestOutFILE=TestResults\some-unique-filename.trx
SET TransformedOutputFile=%TestOutFILE:.trx=%.xml
SET XSLFile=c:\MsBuildNunit\MSBuild-to-NUnit.xslt
MKDIR TestResults
%MSTest% "/testcontainer:%TestDLL%" /nologo /resultsfile:%TestOutFILE%
%XSLParser% %TestOutFILE% %XSLFile% -o %TransformedOutputFile%
SET ERRORLEVEL=0
Then check the box "Publish NUnit test result report" and for "Test report XMLs" enter
TestResults/*.xml
There is an XSLT in C:\MsBuildNunit as well as msxsl.exe which comes from Microsoft.
You can download the MSBuild-to-NUnit.xslt from here and get msxsl.exe from microsoft here or you can just get the zipped copy of my MsBuildNunit folder that contains the xslt and exe here
When run, it calls MSTest.exe which runs the tests and outputs the format in microsofts trx (xml) format. Then it calls msxsl.exe with the xslt and the trx and translates it to nunits xml format. At the end of the build, Hudson picks it up as any other Nunit test result and you are good to go.
Edited to add:
I forgot to mention, with this xslt we get full test results. We have multiple test projects and multiple dll's and we get great feedback with the ability to trend graph, view the tests by name, view the statuses of the tests, and if it errors we get the error message along with the stack trace. Basically almost everything that you would get with Nunit.
Edit (again): I just now added the test duration in the transform so it will show up in Hudson now! Seems to work great for our tests.
Edit: I tried the new MSTest plugin and it currently does not support parsing multiple TRX files, just 1, so currently this is your only solution if you are like us and have multiple test assemblies that you have to run through MSTest.
Hudson has a new plugin for MSTest. Just specify the location of the .trx file and the work is done for you. It wouldn't surprise me if the plugin used Allen's solution.
I've been able to use a variation of "hangy's" command line, and the MSTest plugin to successfully run and analyze/publish the test cases. The biggest change I made was to specify the output file for mstest.exe and fore the MSTest plugin to consume that file (no wildcards allowed... must be actual filename). For example, the following is my custom build step:
"%PROGRAMFILES%\Microsoft Visual Studio 9.0\Common7\IDE\MSTest.exe" /runconfig:LocalTestRun.testrunconfig /testcontainer:MyProject1.Test/bin/Debug/MyProject1.Test.dll /testcontainer: MyProject2.Test/bin/Debug/MyProject2.Test.dll /resultsfile:TestResults\HudsonJobTestResults.trx
exit 0
Notice that the "results file" is relative to the Job's workspace. Thus, the MSTest plugin's result file to parse is:
TestResults\HudsonJobTestResults.trx
And that's it!
Hudson has a Plot Plugin which can be used to plot generic data. It's not the easiest plugin to configure and use if you have multiple data points per graph, but if you can parse your MS Test output and generate input files for the plugin, you can at the very least plot the trends of failed, successful, and total tests.
I've not been able to use Hudson to perform analysis of MS Test results for historical purposes, but I've at least been able to figure out that if you use MSBuild and the Exec task, the Hudson build will properly be marked as "failed" if any of the tests fail.
<Exec Command=""C:\Program Files (x86)\Microsoft Visual Studio 9.0\Common7\IDE\mstest.exe" /testcontainer:"MyAssembly.dll"" />