When running unit tests locally in Visual Studio 2015, I can click on the Output hyperlink in the test results and gain access to all the Debug Trace output (as Standard Output) on the test output page.
However when using a build agent to build and test, I can't find any way to gain access to this output information. I've dug through every screen I can think of and nothing. All it shows is the Assert exception message and the stack trace.
Even if I download the .trx file it doesn't include the Console Output section.
Is there any way to get this output from a test run performed by an agent?
Also, some of my tests write additional information to the TestResults folder. The contents of this folder also appears to be excluded from the stored test information. Is there any way to get that as well?
The only other thing I can think of would be to have my tests write all their debug information to files then copy those to another folder as a build step. Seems kind of kludgy. If I remember correctly, the "old" TFS build process would save all this information automatically and it was available looking at test runs in Visual Studio.
Using System.Diagnostics.Trace.WriteLine() instead of System.Diagnostics.Debug.WriteLine(), you will get the information when run the test from TFS:
Related
We've just switched to using google test for unit testing due to deficiencies in our previous framework.
In Visual Studio, I have a solution with multiple projects. Each project contains unit tests using google test for a specific class (or collection of linked classes) within our application. We only use VS for managing and running google tests and reference the test classes found elsewhere.
When I want to run all tests, I set startup projects to multiple, then press ctrl-f5. This pops ups 1 console window per project with the standard google test output. This is great! I have to click a key to close each of the output windows, though. It sure would be nice to integrate into "Test Explorer" within VS2013 express.
I've found this script runner:
https://visualstudiogallery.msdn.microsoft.com/9dd47c21-97a6-4369-b326-c562678066f0.
It appears to be exactly what I want, but apparently express editions don't allow extensions? (I'm new the land of visual studio.) Anyway, we're using express instead of a different version because we're commercial. My fallback solution is spending the money on a full version of VS. However, I'd much rather find a free solution so I don't have to justify purchase, then deal with licensing visual studio for every new person that joins the group.
Has anyone figured out a solution to this?
Our team also tried using "Google Test Runner" extension. The
integration with VS Test Explorer was great but we did not like
the fact that we could not see gmock warnings. This means that
some tests can pass with warnings and you will simply let that
slip by and create problems in the future. The same problem
exists when trying to use XML output generated by gmock.
Since viewing output in console is a poor option, we've decided to
parse the output of executable that runs the tests and display them
in a browser. To this end, we've modified the project so that it directs
its output to a file ( "> test_output.txt" in Debugging -> Command
Arguments) and modified the main function to parse the output,
insert it into a html file template, save this html file and then open
it in default browser using:
ShellExecute(NULL, NULL, "test_output.html", "", "", SW_SHOW);
This also enabled us to make some customizations to the way results
are displayed using javascript.
You can use this approach in you case by creating a simple project which
will run last in the list of multiple startup projects, parse the output
of other executables and display it in a browser. I know this solution has
its drawbacks, but it is (in my opinion) better than ignoring warnings.
I created a test framework to test a sample application. This basically required creating a Solution for my sample application and then adding a Native Test Project, which can test the application.
The test project generates an .exe, but if I were to execute the .exe on it's own, it does nothing.
Is there a way to get information from the test programme so it can be used for other purposes outside of the VS environment?
Yes, you can run unit tests without VS2013.
Use Run automated tests from the command line using MSTest source to do so.
In general it says:
Open a Visual Studio command prompt.
Either change directory to your solution folder or, when you run the MSTest.exe program in step 3, specify a full or relative path to the metadata file or to the test container.
Run the MSTest.exe program.
I hope it will solve your issue.
It took a couple of days, but I've found that DeJaVo's answer is incorrect. Because I am running a Native Test Project, MSTest will not work with it. To run a Native Test from the command line, use VSTest.Console.
I have test cases that are executable (*.exe files). There is no user interface involved.
How do I use Team center /visual studio online to run these test cases on server.
For now, either on demand running or scheduled running will work for me.
(Currently I have no test case that runs on server. So you may mention the basic setup. )
I have written some test cases (they are exe files). I can run them locally line any other exe file.
My code is in C++.
My test cases are in C++.
You could run them as part of your build. Just configure a build in VSO for your solution, and then modify the msbuild project file to call your tests and send the output to the build folder so it gets uploaded as part of the drop. If you are using VS, you would get a better experience using the VS unit testing support (i.e., get results in VS): http://www.visualstudio.com/get-started/run-tests-with-builds-vs.
I'm trying to get my continuous integration setup to run my unit tests for me, so I've set
<RunTest>true</RunTest>
And I've setup the tests to auto-discover:
<ItemGroup>
<TestContainer Include="$(OutDir)%2a.Test.dll" />
</ItemGroup>
Running the build on my development machine runs my unit tests, and reports that the Build succeeded; When run as an automated build, the tests run (according to the BuildLog), but I'm seeing this in the log:
Publishing results of test run tfs_service#TFS-BUILD 2012-04-13 14:13:59_Any CPU_Release to http://mycompany-tfs:8080/Build/v1.0/PublishTestResultsBuildService2.asmx...
.The drop share directory "\tfs-build\Builds\Test\MyProduct Test_20120413.2" could not be accessed.
The drop share directory "\tfs-build\Builds\Test\MyProduct Test_20120413.2" could not be accessed.
MSBUILD : warning MSB6006: "MSTest.exe" exited with code 1.
The previous error was converted to a warning because the task was called with ContinueOnError=true.
This is odd, as the binaries publish quite merrily.
I tried to diagnose the issue, by logging on to the console of the build server, running up a Visual Studio 2008 Command Prompt, navigating to the source location, and invoking MSBuild, it doesn't appear to even attempt to invoke the unit tests?
Ultimately, I'm more interested in playing "guess why the drop of the results failed" - could it be the embedded space in my build definition name? What else could I check, given the binaries publish OK?
The unit tests are running, the problem is that the results are not being published correctly. When unit tests are run, the results are published to the tfs server and then copied to the drop location. Because of this, the drop share needs to allow the TFS build service account to write to this folder.
http://blogs.msdn.com/b/vstsqualitytools/archive/2010/10/24/test-agent-test-controller-and-mstest-faq.aspx
Is it possible to integrate Hudson with MS Test?
I am setting up a smaller CI server on my development machine with Hudson right now, just so that I can have some statistics (ie. FxCop and compiler warnings). Of course, it would also be nice if it could just run my unit tests and present their output.
Up to now, I have added the following batch task to Hudson, which makes it run the tests properly.
"%PROGRAMFILES%\Microsoft Visual Studio 9.0\Common7\IDE\MSTest.exe" /runconfig:LocalTestRun.testrunconfig /testcontainer:Tests\bin\Debug\Tests.dll
However, as far as I know, Hudson does not support analysis of MS Test results, yet. Does anyone know whether the TRX files generated by MSTest.exe can be transformed to the JUnit or NUnit result format (because those are supported by Hudson), or whether there is any other way to integrate MS Test unit tests with Hudson?
I've been meaning to write this as a guide and develop a plugin but I havent gotten around to it. I know this question is old but I'm SURE someone else out there wants the same thing so here it is.
In the project configuration on Hudson:
Execute Windows batch command
SET MSTest="C:\Program Files\Microsoft Visual Studio 9.0\Common7\IDE\MSTest.exe"
SET XSLParser="C:\MsBuildNunit\msxsl.exe"
SET TestDLL=path-to-your-test-projects.dll
SET TestOutFILE=TestResults\some-unique-filename.trx
SET TransformedOutputFile=%TestOutFILE:.trx=%.xml
SET XSLFile=c:\MsBuildNunit\MSBuild-to-NUnit.xslt
MKDIR TestResults
%MSTest% "/testcontainer:%TestDLL%" /nologo /resultsfile:%TestOutFILE%
%XSLParser% %TestOutFILE% %XSLFile% -o %TransformedOutputFile%
SET ERRORLEVEL=0
Then check the box "Publish NUnit test result report" and for "Test report XMLs" enter
TestResults/*.xml
There is an XSLT in C:\MsBuildNunit as well as msxsl.exe which comes from Microsoft.
You can download the MSBuild-to-NUnit.xslt from here and get msxsl.exe from microsoft here or you can just get the zipped copy of my MsBuildNunit folder that contains the xslt and exe here
When run, it calls MSTest.exe which runs the tests and outputs the format in microsofts trx (xml) format. Then it calls msxsl.exe with the xslt and the trx and translates it to nunits xml format. At the end of the build, Hudson picks it up as any other Nunit test result and you are good to go.
Edited to add:
I forgot to mention, with this xslt we get full test results. We have multiple test projects and multiple dll's and we get great feedback with the ability to trend graph, view the tests by name, view the statuses of the tests, and if it errors we get the error message along with the stack trace. Basically almost everything that you would get with Nunit.
Edit (again): I just now added the test duration in the transform so it will show up in Hudson now! Seems to work great for our tests.
Edit: I tried the new MSTest plugin and it currently does not support parsing multiple TRX files, just 1, so currently this is your only solution if you are like us and have multiple test assemblies that you have to run through MSTest.
Hudson has a new plugin for MSTest. Just specify the location of the .trx file and the work is done for you. It wouldn't surprise me if the plugin used Allen's solution.
I've been able to use a variation of "hangy's" command line, and the MSTest plugin to successfully run and analyze/publish the test cases. The biggest change I made was to specify the output file for mstest.exe and fore the MSTest plugin to consume that file (no wildcards allowed... must be actual filename). For example, the following is my custom build step:
"%PROGRAMFILES%\Microsoft Visual Studio 9.0\Common7\IDE\MSTest.exe" /runconfig:LocalTestRun.testrunconfig /testcontainer:MyProject1.Test/bin/Debug/MyProject1.Test.dll /testcontainer: MyProject2.Test/bin/Debug/MyProject2.Test.dll /resultsfile:TestResults\HudsonJobTestResults.trx
exit 0
Notice that the "results file" is relative to the Job's workspace. Thus, the MSTest plugin's result file to parse is:
TestResults\HudsonJobTestResults.trx
And that's it!
Hudson has a Plot Plugin which can be used to plot generic data. It's not the easiest plugin to configure and use if you have multiple data points per graph, but if you can parse your MS Test output and generate input files for the plugin, you can at the very least plot the trends of failed, successful, and total tests.
I've not been able to use Hudson to perform analysis of MS Test results for historical purposes, but I've at least been able to figure out that if you use MSBuild and the Exec task, the Hudson build will properly be marked as "failed" if any of the tests fail.
<Exec Command=""C:\Program Files (x86)\Microsoft Visual Studio 9.0\Common7\IDE\mstest.exe" /testcontainer:"MyAssembly.dll"" />