I'm in the process of moving from on-premise TFS to Visual Studio Online (VSTS) and I'm trying to create a new CI build definition as our local one is xaml based.
For our server based tests I'm using a Visual Studio Test task it it's running fine.
I also need to run Silverlight tests (as we currently have in the local xaml build). Currently I can run my SL unit tests from the command line and output a .trx test results file using the following command:
"C:\Program Files (x86)\StatLight\statlight.exe" -x .\path-to-silverlight.xap --ReportOutputFileType:TRX -r=.\statlight-results.trx
I can also run this using a CommandLine task in vsts (I'm already using a local build agent instead of a hosted one), but I don't know how to:
Get it to fail the build if there's a failing test.
Get the test results into the Test output tab in vsts.
I'll also need to do something similar to fail a build based on js unit tests as we migrate away from Silverlight, so the solution for the SL tests may help with running js tests also.
See if there is a switch that causes the command line to return the number of failed tests. If so you can use the return code to fail your build. Any value other than zero will fail build.
As for test results just add a publish test results task and point to the generated trx file. That is it.
Related
I created a test framework to test a sample application. This basically required creating a Solution for my sample application and then adding a Native Test Project, which can test the application.
The test project generates an .exe, but if I were to execute the .exe on it's own, it does nothing.
Is there a way to get information from the test programme so it can be used for other purposes outside of the VS environment?
Yes, you can run unit tests without VS2013.
Use Run automated tests from the command line using MSTest source to do so.
In general it says:
Open a Visual Studio command prompt.
Either change directory to your solution folder or, when you run the MSTest.exe program in step 3, specify a full or relative path to the metadata file or to the test container.
Run the MSTest.exe program.
I hope it will solve your issue.
It took a couple of days, but I've found that DeJaVo's answer is incorrect. Because I am running a Native Test Project, MSTest will not work with it. To run a Native Test from the command line, use VSTest.Console.
I have a simple application and Nunit unit test project with 2 tests to test that app.
I've managed to have unit tests discovered and run locally in Visual studio 2013 but when trying to build application and run these unit tests on TFS 2013 test runner does not find tests ( both projects are built successfully.
No test found. Make sure that installed test discoverers & executors, platform & framework version settings are appropriate and try again.
I've added "NUnit adapter with framework" package to Unit tests project according to answer in this question and committed changes to the TFS.
Unit tests projects is successfully built but still no tests are found. The Test dll fits the **\*Test*.dll naming pattern as configured in the Build Definition.
Why no tests are found? How I can make sure Test Runner is aware of my NUnit test adapter?
If you go to the build, and look in diagnostics (from the web portal), there is a Run VS Test Runner section.
In the same log, there is a Run MSBuild section, it should say "Successfully installed xxxx" where xxx is the name of your adapter package.
Something like:
Check to see that the test runner is successfully restored.
Found the reason - build agent folder in source settings of Build definition was not set correctly - instead of setting it somewhere under build agent working folder I set it outside of it that's why unit tests were not found..
I discovered this using Build process activity log (which was saved in drop folder in my case).this log contains very detailed information (much more than you can see in Visual studio) - there I found in which location test runner is looking for unit tests and then understood what i did wrong.
That's what I saw in log:
Run VS Test Runner00:00:00
Inputs
TestSpecs: BuildParameter[] Array
Enabled: True
ConfigurationsToTest: String[] Array
OutDir:
There were no matches for the search pattern C:\Builds\11\XXXX\Application -Gated\bin\**\*Test*.dll
Hope this will be helpful to someone else.
thanks to everyone who responded to my question!
I'm trying to get my continuous integration setup to run my unit tests for me, so I've set
<RunTest>true</RunTest>
And I've setup the tests to auto-discover:
<ItemGroup>
<TestContainer Include="$(OutDir)%2a.Test.dll" />
</ItemGroup>
Running the build on my development machine runs my unit tests, and reports that the Build succeeded; When run as an automated build, the tests run (according to the BuildLog), but I'm seeing this in the log:
Publishing results of test run tfs_service#TFS-BUILD 2012-04-13 14:13:59_Any CPU_Release to http://mycompany-tfs:8080/Build/v1.0/PublishTestResultsBuildService2.asmx...
.The drop share directory "\tfs-build\Builds\Test\MyProduct Test_20120413.2" could not be accessed.
The drop share directory "\tfs-build\Builds\Test\MyProduct Test_20120413.2" could not be accessed.
MSBUILD : warning MSB6006: "MSTest.exe" exited with code 1.
The previous error was converted to a warning because the task was called with ContinueOnError=true.
This is odd, as the binaries publish quite merrily.
I tried to diagnose the issue, by logging on to the console of the build server, running up a Visual Studio 2008 Command Prompt, navigating to the source location, and invoking MSBuild, it doesn't appear to even attempt to invoke the unit tests?
Ultimately, I'm more interested in playing "guess why the drop of the results failed" - could it be the embedded space in my build definition name? What else could I check, given the binaries publish OK?
The unit tests are running, the problem is that the results are not being published correctly. When unit tests are run, the results are published to the tfs server and then copied to the drop location. Because of this, the drop share needs to allow the TFS build service account to write to this folder.
http://blogs.msdn.com/b/vstsqualitytools/archive/2010/10/24/test-agent-test-controller-and-mstest-faq.aspx
I'm trying to work out how to run xUnit tests from TFS 2010.
I found some articles on how to achieve that with the older version (http://jonnekats.wordpress.com/2009/05/07/integrate-xunit-tests-into-your-daily-team-build/, http://weblogs.asp.net/mehfuzh/archive/2009/08/25/configuring-team-build-using-xunit.aspx).
These approaches don't work anymore with TFS 2010 because builds are no longer MSBuild files but instead are Workflows with different tasks.
What I would like to achieve is something similar to what I've got on my dev machine:
- build everything
- run the .xunit project file
- check the results
All that I can see on the web are custom build steps which I can't/won't use because I'll have to configure them for each single unit test assembly and they'll probably get completely messed up in TFS.
Any input would be appreciated.
Why dont use "InvokeProcess" in TFS Build workflow and invoke xUnit command line interface with the necessary xUnit project files as parameters?
You can capture the output of the command line by setting InvokeProcess->stdout to write a build message.
Take a look at the xUnit work item Support VSTS Test Runner on codeplex. You can find there a VS2010 extension in Beta version.
Is it possible to integrate Hudson with MS Test?
I am setting up a smaller CI server on my development machine with Hudson right now, just so that I can have some statistics (ie. FxCop and compiler warnings). Of course, it would also be nice if it could just run my unit tests and present their output.
Up to now, I have added the following batch task to Hudson, which makes it run the tests properly.
"%PROGRAMFILES%\Microsoft Visual Studio 9.0\Common7\IDE\MSTest.exe" /runconfig:LocalTestRun.testrunconfig /testcontainer:Tests\bin\Debug\Tests.dll
However, as far as I know, Hudson does not support analysis of MS Test results, yet. Does anyone know whether the TRX files generated by MSTest.exe can be transformed to the JUnit or NUnit result format (because those are supported by Hudson), or whether there is any other way to integrate MS Test unit tests with Hudson?
I've been meaning to write this as a guide and develop a plugin but I havent gotten around to it. I know this question is old but I'm SURE someone else out there wants the same thing so here it is.
In the project configuration on Hudson:
Execute Windows batch command
SET MSTest="C:\Program Files\Microsoft Visual Studio 9.0\Common7\IDE\MSTest.exe"
SET XSLParser="C:\MsBuildNunit\msxsl.exe"
SET TestDLL=path-to-your-test-projects.dll
SET TestOutFILE=TestResults\some-unique-filename.trx
SET TransformedOutputFile=%TestOutFILE:.trx=%.xml
SET XSLFile=c:\MsBuildNunit\MSBuild-to-NUnit.xslt
MKDIR TestResults
%MSTest% "/testcontainer:%TestDLL%" /nologo /resultsfile:%TestOutFILE%
%XSLParser% %TestOutFILE% %XSLFile% -o %TransformedOutputFile%
SET ERRORLEVEL=0
Then check the box "Publish NUnit test result report" and for "Test report XMLs" enter
TestResults/*.xml
There is an XSLT in C:\MsBuildNunit as well as msxsl.exe which comes from Microsoft.
You can download the MSBuild-to-NUnit.xslt from here and get msxsl.exe from microsoft here or you can just get the zipped copy of my MsBuildNunit folder that contains the xslt and exe here
When run, it calls MSTest.exe which runs the tests and outputs the format in microsofts trx (xml) format. Then it calls msxsl.exe with the xslt and the trx and translates it to nunits xml format. At the end of the build, Hudson picks it up as any other Nunit test result and you are good to go.
Edited to add:
I forgot to mention, with this xslt we get full test results. We have multiple test projects and multiple dll's and we get great feedback with the ability to trend graph, view the tests by name, view the statuses of the tests, and if it errors we get the error message along with the stack trace. Basically almost everything that you would get with Nunit.
Edit (again): I just now added the test duration in the transform so it will show up in Hudson now! Seems to work great for our tests.
Edit: I tried the new MSTest plugin and it currently does not support parsing multiple TRX files, just 1, so currently this is your only solution if you are like us and have multiple test assemblies that you have to run through MSTest.
Hudson has a new plugin for MSTest. Just specify the location of the .trx file and the work is done for you. It wouldn't surprise me if the plugin used Allen's solution.
I've been able to use a variation of "hangy's" command line, and the MSTest plugin to successfully run and analyze/publish the test cases. The biggest change I made was to specify the output file for mstest.exe and fore the MSTest plugin to consume that file (no wildcards allowed... must be actual filename). For example, the following is my custom build step:
"%PROGRAMFILES%\Microsoft Visual Studio 9.0\Common7\IDE\MSTest.exe" /runconfig:LocalTestRun.testrunconfig /testcontainer:MyProject1.Test/bin/Debug/MyProject1.Test.dll /testcontainer: MyProject2.Test/bin/Debug/MyProject2.Test.dll /resultsfile:TestResults\HudsonJobTestResults.trx
exit 0
Notice that the "results file" is relative to the Job's workspace. Thus, the MSTest plugin's result file to parse is:
TestResults\HudsonJobTestResults.trx
And that's it!
Hudson has a Plot Plugin which can be used to plot generic data. It's not the easiest plugin to configure and use if you have multiple data points per graph, but if you can parse your MS Test output and generate input files for the plugin, you can at the very least plot the trends of failed, successful, and total tests.
I've not been able to use Hudson to perform analysis of MS Test results for historical purposes, but I've at least been able to figure out that if you use MSBuild and the Exec task, the Hudson build will properly be marked as "failed" if any of the tests fail.
<Exec Command=""C:\Program Files (x86)\Microsoft Visual Studio 9.0\Common7\IDE\mstest.exe" /testcontainer:"MyAssembly.dll"" />