Trac won't parse google test xml output files - c++

I have run the my unit tests written with the google test 1.6.0 framework with the --gtest_output="xml:test-results.xml" flag and get a test result file like this:
<?xml version="1.0" encoding="UTF-8"?>
<testsuites tests="8" failures="0" disabled="0" errors="0" time="81.396" name="AllTests">
<testsuite name="TestSuite1" tests="8" failures="0" disabled="0" errors="0" time="81.396">
<testcase name="Test1" status="run" time="6.391" classname="Class1" />
<testcase name="Test2" status="run" time="6.1" classname="Class1" />
<testcase name="Test3" status="run" time="7.146" classname="Class1" />
<testcase name="Test4" status="run" time="16.164" classname="Class1" />
<testcase name="Test5" status="run" time="25.145" classname="Class1" />
<testcase name="Test6" status="run" time="6.099" classname="Class1" />
<testcase name="Test7" status="run" time="6.162" classname="Class1" />
<testcase name="Test8" status="run" time="8.187" classname="Class1" />
</testsuite>
</testsuites>
Based on what I have read in other posts the gtest xml output is supposed to be compatible with junit parsers. Related Post: Unable to get hudson to parse JUnit test output XML
The other possibility of error is in my bitten script. Running on Trac 0.12.2. Here is my bitten recipe for parsing the XML file using the java:junit parser:
<build xmlns:java="http://bitten.edgewall.org/tools/java">
<step id="parse_results" description="Gather Results" onerror="fail">
<java:junit file="/home/user/temp/test-results.xml" />
</step>
</build>
In trac, it says that the build was successful, but the test results are blank. 0 run, 0 failed, 0 ignored, 0 errors
Thanks.

I was able to solve the problem. It turns out that Trac's JUnit parser has a bug. It doesn't like the testsuites tag and it doesn't like having multiple testsuite sections. PHP allows the testsuites tag, but will not do multiple files. I chose to create a parser in Python that creates multiple XML files from the Gtest output file.
def move_results(results, results_dir):
# Moves all results into a temp folder to be consumed by Bitten
# Files with multiple test suite sections, split into individual files
for files in results:
fin = open(files)
test_id = 0
split_line = files.split('/')
file_name = split_line[len(split_line)-1].split('.xml')
for line in fin:
if not 'testsuites' in line:
if '<testsuite ' in line:
output_file = results_dir + file_name[0] + '-' + str(test_id) + '.xml'
test_id = test_id + 1
fout = open(output_file, 'w')
fout.write('<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n')
fout.write(line)
elif '<testsuite\\>' in line:
fout.write(line)
fout.close()
elif not '<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n' in line:
fout.write(line)
fin.close()
os.remove(files)
Alternatively, you could use Bitten to do convert using an XSLT. Also you could combine the output files into a single output (with something like nose's XUnit output) and use php:phpunit to parse the file.

Related

NUnit is not running my test cases

I have a test case that looks like:
namespace UnitTests
[<TestFixture>]
module ValidatorTests =
[<Test>]
let VerifyBasicFunctionality () =
Assert.That(bool)
Assert.That(bool)
and when I try to run it in the Visual Stuido test explorer, nothing happens (even with the Test Adapter for NUnit 3) and just says "Successful Build" and no tests discovered. Then when I run from the commandline with the nunit-console runner (tried with v.1, v.2, v.4) I get something different:
$ nunit-console4 bin/Release/UnitTests.dll
NUnit version 2.4.8
Copyright (C) 2002-2007 Charlie Poole.
Copyright (C) 2002-2004 James W. Newkirk, Michael C. Two, Alexei A. Vorontsov.
Copyright (C) 2000-2002 Philip Craig.
All Rights Reserved.
Runtime Environment -
OS Version: Unix 15.6.0.0
CLR Version: 4.0.30319.42000 ( 4.4.2 (Stable 4.4.2.11/f72fe45 Thu Aug 11 06:03:25 BST 2016) )
.N.N.N
Tests run: 0, Failures: 0, Not run: 3, Time: 0.011 seconds
When I run this with the -xmlConsole flag, I get this:
<?xml version="1.0" encoding="utf-8" standalone="no"?>
<!--This file represents the results of running a test suite-->
<test-results name="bin/Release/UnitTests.dll" total="0" failures="0" not-run="3" date="2016-09-08" time="10:26:00">
<environment nunit-version="2.4.8.0" clr-version="4.0.30319.42000" os-version="Unix 15.6.0.0" platform="Unix" cwd="/SOMEUSER/SOMEPATH/UnitTests" machine-name="gmaini-m.jet.local" user="SOMEUSER" user-domain="SOMELOCALBOX" />
<culture-info current-culture="en-US" current-uiculture="en-US" />
<test-suite name="bin/Release/UnitTests.dll" success="True" time="0.010" asserts="0">
<results>
<test-suite name="UnitTests" success="True" time="0.008" asserts="0">
<results>
<test-suite name="ValidatorTests" success="True" time="0.001" asserts="0">
<results>
<test-case name="UnitTests.ValidatorTests.VerifyBasicFunctionality" executed="False">
<reason>
<message><![CDATA[UnitTests.ValidatorTests is an abstract class]]></message>
</reason>
</test-case>
<test-case name="UnitTests.ValidatorTests.VerifyBasicOtherFunctionality" executed="False">
<reason>
<message><![CDATA[UnitTests.ValidatorTests is an abstract class]]></message>
</reason>
</test-case>
<test-case name="UnitTests.ValidatorTests.VerifyBasicSomeFunctionality" executed="False">
<reason>
<message><![CDATA[UnitTests.ValidatorTests is an abstract class]]></message>
</reason>
</test-case>
</results>
</test-suite>
</results>
</test-suite>
</results>
Any idea why it seems to discover the tests, I can dot access them with -run="Namespace.Module.Function" and talk about the fixture with -fixture="Namespace" but it won't run them? Is the UnitTests.ValidatorTests.VerifyBasicFunctionality is an abstract class a hint that there is some C# interop problem?
Thanks for your time :)
This fix was:
namespace UnitTests
[<TestFixtureAttribute>]
type ValidatorTests () =
[<Test>]
member __.VerifyBasicFunctionality() =
Assert.That(bool)
Assert.That(bool)
Basically, making this object oriented. So my hypothesis was correct.

Slowcheetah app.config pass parameters to transform file

I have following transform file app.Release.config to transform my app.config using SlowCheetah. How do I pass the BUILD_NUMBER from TeamCity so that the transform file replaces a particular xml elements value.
<?xml version="1.0" encoding="utf-8" ?>
<configuration xmlns:xdt="http://schemas.microsoft.com/XML-Document-Transform">
<runtime>
<assemblyBinding xmlns="urn:schemas-microsoft-com:asm.v1">
<dependentAssembly>
<assemblyIdentity name="MyApp.Data.Model" publicKeyToken="866d4a0fa0599fe0" culture="neutral" />
<bindingRedirect name="MyApp.Data.Model.BR" oldVersion="0.0.0.0-$(BUILD_NUMBER)" newVersion="$(BUILD_NUMBER)" xdt:Transform="SetAttributes" xdt:Locator="Match(name)"/>
</dependentAssembly>
</assemblyBinding>
If I hard code the value then SlowCheetah does the transformation. But, I don't know how to pass in the build_number as a argument so that for each build I can replace the correct version.
I could not find a way to pass parameters to slowcheetah transform files. I'm using a XmlPoke Task to modify the contents of app.config.
My answer for this can be found at this link

Is there a way to get build status as a property?

I have a ugly Teamcity build configuration using MSBuild. It executes custom application (test runner), which is using custom messaging to report test results to teamcity.
##teamcity[testStarted name='test1']
##teamcity[testFailed name='test1' message='failure message' details='message and stack trace']
Which show in teamcity in build overview and tests tab.
Teamcity recognizes failed tests and if any test fails, it marks the build as failed:
http://i.stack.imgur.com/Qz9UT.png
Later in the MSBuild target I would like to label cvs based on the test results.
Is there a way to get the build status (if it is failed, hanging, warning) as a property? something like %build.status%? The format does not matter - if its a string or number.
PS: I know that best solution to my problem would be to modify the application to return non-zero exit code if test fail.
TeamCty does not seem to expose this directly, but the status can be acquired using the REST api. Here is an example using curl; but you could also uwe PowserShell's Invoke-RestMethod for instance.
Here's the msbuild script that casues test failure I used for testing:
<?xml version="1.0" encoding="utf-8"?>
<Project ToolsVersion="4.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
<Target Name="Test">
<Message Importance="high" Text="##teamcity[testStarted name='test1']" />
<Message Importance="high" Text="##teamcity[testFailed name='test1' message='failure message' details='message and stack trace']" />
</Target>
</Project>
Then the script that gets the current build's status, dumps it to a file, reads the file into an msbuild item and then uses regex to get the status out of it. You just have it to supply the tc_user and tc_password properties (or allow guest access) and change the url to match your server.
<?xml version="1.0" encoding="utf-8"?>
<Project ToolsVersion="4.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003" DefaultTargets="GetBuildStatus">
<Target Name="RunCurl">
<PropertyGroup>
<MyTempFile>curl_out</MyTempFile>
</PropertyGroup>
<Exec Command="curl http://localhost/httpAuth/app/rest/builds/id:$(teamcity_build_id) -basic -u $(tc_user):$(tc_password) > $(MyTempFile)"/>
<ReadLinesFromFile File="$(MyTempFile)">
<Output TaskParameter="Lines" ItemName="CurlOutput"/>
</ReadLinesFromFile>
<Delete Files="$(MyTempFile)"/>
</Target>
<Target Name="GetBuildStatus" DependsOnTargets="RunCurl">
<PropertyGroup>
<CurlOutputFull>#(CurlOutput)</CurlOutputFull>
<BuildStatus>$([System.Text.RegularExpressions.Regex]::Match($(CurlOutputFull), `status="(\w*)"`).Groups[ 1 ].Value)</BuildStatus>
</PropertyGroup>
<Message Text="BuildStatus = $(BuildStatus)"/>
</Target>
</Project>
This prints:
BuildStatus = FAILURE

MSBuild RegexMatch not matching

I have the following
<RegexMatch Input="$(Configuration)" Expression="^.*?(?=\.)">
<Output ItemName="Theme" TaskParameter="Output" />
</RegexMatch>
My configuration variable is as follows Theme.Environment
So "Default.Debug"
or "Yellow.Release"
I would like to get the first portion in to a varaible called theme.
I have tested this regex and it works in stand alone regex testers
^.*?(?=\.)
but not when used in my build file.
I am echoing the variable out so that i can see the output
<Exec Command="echo $(Theme)"/>
<Exec Command="echo $(Configuration)"/>
Ideas?
If you should use MSBuild Community tasks for that - check this line: <Output PropertyName="Theme" TaskParameter="Output" />
you should use PropertyName="Theme" if you want to refer it like $(Theme) later.
ItemName will create items set, not property.
But it's much simplier to use MSBuild 4.0 inline functions than Msbuild community tasks for that concrete task. Your code will looks like this (adopt for your script):
<?xml version="1.0" encoding="utf-8"?>
<Project xmlns="http://schemas.microsoft.com/developer/msbuild/2003" ToolsVersion="4.0" DefaultTarget="Play">
<PropertyGroup>
<Configuration>Yellow.Release</Configuration>
</PropertyGroup>
<Target Name="Play">
<PropertyGroup>
<Theme>$([System.Text.RegularExpressions.Regex]::Match($(Configuration), `^.*?(?=\.)`))</Theme>
</PropertyGroup>
<Message Text="$(Theme)" />
<Message Text="$(Configuration)" />
</Target>
</Project>
Just realised that RegexMatch doenst return the matched string but rather returns the entire string if matched.
basically it called IsMatch method not Match method
Have re written as a RegexReplace
<RegexReplace Input="$(Configuration)" Expression="\..*" Replacement="" Count="1">
<Output ItemName="Theme" TaskParameter="Output" />
</RegexReplace>
After that it still wasnt working and then i realised i was doing
$(Theme)
Should have been
#(Theme)

nant script doesn't display unit test details

Can someone please tell me why my build script (nant) doesn't display the unit test details in the command prompt window? I have verbose set to true, but it doesn't want to display any details about my unit tests. Here's the target:
<target name="run-unit-tests" depends="compile, move.assemblies.for.tests, rebuildDatabase">
<mkdir dir="${tests.output.dir}" />
<nunit2 haltonfailure="true" failonerror="true" verbose="true">
<formatter type="Xml" extension=".xml" outputdir="${tests.output.dir}" usefile="true" />
<test assemblyname="${test.assembly.file}" />
</nunit2>
<echo message="Unit Testing Done!" />
</target>
The command prompt window just displays this:
[mkdir] Creating directory 'C:\Projects\TestProject\build\artifacts\UnitTestOutput'.
[echo] Unit Testing Done!
build:
BUILD SUCCEEDED
Am I missing something here?
Thanks!
I found the answer. I looked at the source for CodeCampServer and saw a line
<formatter type="Plain" />
and added it to my build script so it looks like this:
<nunit2 haltonfailure="true" failonerror="true" verbose="true">
<formatter type="Xml" extension=".xml" outputdir="${tests.output.dir}" usefile="true" />
<formatter type="Plain" />
<test assemblyname="${test.assembly.file}" />
</nunit2>
and now it displays the details.
Sorry to ask the question prematurely on here, but at least it might help someone in the future if they have a similar problem.
Is there a log file in ${tests.output.dir} ? If so, what if you set usefile to false and type to "Plain"?