Running all unit tests in nant, even if some fail! - unit-testing

I want my unit tests to run regardless of there being a failure. I want to know how many are failing not just the first one, so I don't go through the build, fail, fix and build again cycle. Also, it is the responsibility of an other team to fix some tests so I want to know our are ok.
So in Nant, I've added the following in a target for the unit tests, as failonerror is false it runs all the tests, but does not fail the build.
<nunit2 failonerror="false" haltonfailure="false">
<test appconfig="tests.config">
<assemblies basedir="${test.dir}">
<include name="SomeTests.dll" />
</assemblies>
</test>
</nunit2>
At the end of the run, Nant reports
1 non-fatal error(s), 0 warning(s)
I want to check the non-fatal error count, if it is more than 0, I want to do something like this...
<fail message="Failures reported in unit tests."
unless="report.errors == 0" />
except I don't know how to get the error count......does anyone know how?

Apart from the fact that dropping <nunit2> in favor of <exec> might be a good idea anyway, in Your special case the <exec> task can solve the problem:
<exec
program="C:\dev\tools\NUnit\2.5.9\bin\net-2.0\nunit-console.exe"
resultproperty="exec.nunit.result"
failonerror="false">
<arg file="C:\foo\bar.dll" />
</exec>
<if test="${int::parse(exec.nunit.result) != 0}">
<!-- fail, print number of failures etc. -->
</if>

Related

Bespoke C++ Windows app crashing and producing an empty dmp file

We have an in-house c++ app that we are running via task scheduler. We are attempting to trace an issue with the app which causes a crash, a windows event and we would usually look for the .dmp file to enable us to track the issue in visual studio.
However, these windows dump files [appname. dmp] is zero bytes.
Is there anyone out there that knows the potential causes of the .dmp file being empty, please?
Sample event log below in case it helps.
Many thanks for any help and thoughts :)
Andy P
- <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
- <System>
<Provider Name="Application Error" />
<EventID Qualifiers="0">1000</EventID>
<Level>2</Level>
<Task>100</Task>
<Keywords>0x80000000000000</Keywords>
<TimeCreated SystemTime="2021-11-09T15:48:04.642807400Z" />
<EventRecordID>5214</EventRecordID>
<Channel>Application</Channel>
<Computer>redacted>
<Security />
</System>
- <EventData>
<Data>redacted.exe</Data>
<Data>1.0.0.1</Data>
<Data>60816c15</Data>
<Data>KERNELBASE.dll</Data>
<Data>10.0.17763.2183</Data>
<Data>12a65345</Data>
<Data>e06d7363</Data>
<Data>001235e2</Data>
<Data>fa0</Data>
<Data>01d7d57fa38520cb</Data>
<Data>K:\redacted\redacted.exe</Data>
<Data>C:\WINDOWS\System32\KERNELBASE.dll</Data>
<Data>03eae790-81b8-4040-aab4-6068440c4db0</Data>
<Data />
<Data />
</EventData>
</Event>
MSDN notes one very important restriction: writing a minidump from inside a just-crashed process is not reliable. It's not clear from your question whether you're using a separate process to write the dumps ("redacted.exe"?), so this is definitely a possible cause.

Output of additional files (detectors, induction loops) are not being generated

Output of additional files (detectors, induction loops) are not being generated. I used a simple scenario, just to see how things work. But I cannot see or maybe these files are not being generated. I added them at launch.xml and sumo.cfg as additional files. This is the code for detector.xml :
<additional>
<laneAreaDetector id="E2" lane="1to2_0" pos="15" endPos="20" length="20.0" friendlyPos="false" freq="5.890e9" file="ostie.xml" timeThreshold="1" speedThreshold="3.6" jamThreshold="10.0" />
</additional>
<additional>
<inductionLoop id="myLoop1" lane="1to2_0" pos="16.0" freq="900" file="out.xml" />
<inductionLoop id="myLoop2" lane="1to2_0" pos="100.0" freq="900" file="out.xml" />
</additional>
The file out is empty. Please help!
Thanks in advance.
You might be using sumo-launchd.py of Veins to run your simulation. If so, the SUMO simulation will execute in a temporary directory (that is, by default, deleted after the simulation concludes). You will either turn this off (see the output of --help) or create the output files in another directory to retain them.

How to make `cake test` return error exit code when some test fails

It seems cake test always returns 0 to the shell, even when some tests fail. This is probably fine for human, but causes problem in continuous integration. Is there anyway to make cake test return non-0 when some test fails?
You can always use grep:
cake test | grep -qve '0 tests failed'
Replace "0 tests failed" with a string that clearly determines that all tests passed fine.
I don't know how good is an answer after 2 years, but looks like the solution to this is explained here:
https://www.hiddentao.com/archives/2011/12/21/instructing-cakefile-to-exit-with-error-if-a-vows-test-fails/
What I understand is that cake by itself is very lightweight and doesn't do anything about returning a proper exit code. But by looking for a couple of clues like presence of "✗ Broken" in the output or checking the exit code of the spec runner itself, you can instruct cake to return "1", so that your CI understands that the tests failed.

CFExecute not performing command

<cfset LOCAL.cmd = expandPath('..\library\gm.exe') />
<cfset LOCAL.args = "convert image1.jpg image2.jpg" />
<cfexecute variable="gm" errorVariable="error"
name="#LOCAL.cmd#"
timeout="10"
arguments="#local.args#" />
<cfdump var="#gm#" />
This code always results in an empty string in gm. No matter how I execute gm with or without parameters. Other examples work fine like running cmd.exe or netstat.exe as is in the CFDocs example. I get no errors thrown or warnings in errorVariable, it simply does nothing.
I modified the code, this version does not work either:
<cfset LOCAL.cmd = expandPath('..\library\gm.exe') />
<cfset LOCAL.args = "convert ""#variables.uploadDirectory##LOCAL.file.source#"" ""#variables.uploadDirectory#optimal-#LOCAL.file.source#""" />
<cfexecute errorVariable="error"
name="c:\windows\system32\cmd.exe"
timeout="10"
outputFile="#expandPath('.\gm.log')#"
arguments="/C #local.cmd# #LOCAL.args#" />
Permissions problems are the most common cause. However, if you are running CF8, you might also try redirecting the error stream and adding an explicit terminate flag. Just to see if you get any output or see different behavior. Early versions did not capture the error stream, which caused some processes to hang. It was fixed in one of the CF8 updaters.
Update: I just noticed your image paths are relative. Perhaps the program is having difficulty locating them. Try using absolute paths for the images.
Update: I tested it with CF9. It does work when using absolute image paths. Though the "gm" variable is understandably empty, since the output is directed to an image file.
<cfexecute variable="gm"
errorVariable="errorOut"
name="C:\GraphicsMagick-1.3.12-Q16\gm.exe"
timeout="10"
arguments="convert c:\art.gif c:\artCopyFromCF9.gif" />
<cfdump var="#variables#">
Without seeing code or your server setup, I would guess you need to check permissions for the user account CF runs under.
If CF is running under the default user, you may need to create a user with access to whatever it is you are trying to do. Then change the service(s) to run under this user. Alternately, you could assign more liberal permissions to the resource you're trying to access.
I have also encountered this with cfexecute and GraphicsMagick. I think the deal is that GM is operating asynchronously and returns before it completes. Running some tests with outputFile/errorFile instead of their variable equivalents, followed by cffile reading the fileInfo on the output file (which is empty per the test script but observed to have contents when opened), I see that the modified time of the output file with contents is actually after the last modified timestamp yielded by FileInfo.
I think if you output to a session variable or something of the sort that could be picked up by another template you could observe the results of the execution having populated the session variable, provided the other template executes after the variable is actually set.

What XSLT do you use to format MsBuild XML output in CruiseControl.Net?

We currently don't format our msbuild output in CC.NET (CruiseControl.Net) and as a result, finding the cause of a broken build involves reading the XML to find the last 'success="false"' instance in the output.
What XSLT do you use to format your msbuild output, and are you happy with the resulting HTML? I.e. do you find it easy to identify the cause of a broken build?
Thanks
b
EDIT:
Here's a sanitised sample of one of our CC project XML elements. I'm now wondering whether the merge of logs is the issue.
<project name="StackOverflowSample">
<workingDirectory>D:\_300</workingDirectory>
<webURL>&viewFarmReportWebURL;</webURL>
<sourcecontrol type="multi">
<sourceControls>
<vsts>
<!-- We get latest from TSF -->
</vsts>
</sourceControls>
</sourcecontrol>
<triggers>
<intervalTrigger seconds="60" />
</triggers>
<tasks>
<msbuild>
<executable>&msbuildExecutable;</executable>
<workingDirectory>app\consoleApp1</workingDirectory>
<projectFile>consoleApp1.sln</projectFile>
<buildArgs>/noconlog /p:Configuration=Release /v:quiet</buildArgs>
<logger>ThoughtWorks.CruiseControl.MsBuild.XmlLogger,"D:\Program Files\CruiseControl.NET\server\ThoughtWorks.CruiseControl.MsBuild.dll"</logger>
</msbuild>
<nunit>
<path>&nunitConsoleExecutable;</path>
<assemblies>
<assembly> D:\_300\app\consoleApp1\bin\Release\consoleApp1.exe</assembly>
</assemblies>
</nunit>
<exec>
<executable>&ncoverExecutable;</executable>
<buildArgs>"&nunitConsoleExecutable;" "app\consoleApp1\bin\Release\consoleApp1.exe" /nologo</buildArgs>
</exec>
<exec>
<executable>&ndependExecutable;</executable>
<buildArgs>D:\_300\app\consoleApp1.xml /Silent</buildArgs>
</exec>
<merge>
<files>
<file>D:\_300\app\consoleApp1\unit-test.xml</file>
<file>D:\_300\app\consoleApp1\ApplicationMetrics.xml</file>
<file>D:\_300\app\consoleApp1\AssembliesBuildOrder.xml</file>
<file>D:\_300\app\consoleApp1\AssembliesDependencies.xml</file>
<file>D:\_300\app\consoleApp1\AssembliesMetrics.xml</file>
<file>D:\_300\app\consoleApp1\CQLResult.xml</file>
<file>D:\_300\app\consoleApp1\InfoWarnings.xml</file>
<file>D:\_300\app\consoleApp1\NDependMain.xml</file>
<file>D:\_300\app\consoleApp1\TypesDependencies.xml</file>
<file>D:\_300\app\consoleApp1\TypesMetrics.xml</file>
</files>
</merge>
</tasks>
<publishers>
<merge>
<files>
<file>D:\_300\app\consoleApp1\SymbolModule.Xml</file>
</files>
</merge>
<xmllogger logDir="." />
&emailconsoleApp1;
</publishers>
</project>
I've tried CruiseControl.Net 1.4.4.83 with Rodemeyer.MsBuildToCCnet.dll 1.0.0.5 as a logger coupled with msbuild2ccnet.xsl, and the output is nothing like the output samples from the article:
Build started
Project "" (Integration.Common.csproj target(s)):
error CS1002:
Build succeeded
error CS1002:
1 Error(s)
0 Warning(s)
Time elapsed
Using ThoughtWorks.CruiseControl.MSBuild.dll as a logger coupled with msbuild.xsl, the results are just fine:
Build started 07/16/2009 13:46:38
Person.cs (18,53): error CS1002: ; expected
Build FAILED
Person.cs (18,53): error CS1002: ; expected
1 Error(s)
0 Warning(s)
Time elapsed 00:00:00
After a conversation the developers (I have been told that the alternative logger you link to is quite old and it's usage is discouraged), I am using the default, standard logger and xslt. If it does cause problems for you, please report a bug on CruiseControl.Net Jira, the more people vote on this the sooner someone with commit access to the code might look into it.
"View Build Log" is a raw view, not usable except for debugging the build server setup - for some of my project this is almost 5MB large and does not cause any problems for the server (although opening it hangs the browser for a while). NCover can cause problems inflating buildlog.xml to over 100MB - you have to use NCoverExplorer to analyze the results before merging them if this happens(but don't bother before you start getting exceptions from the server).
To see the formatted MSBuild results in the WebDashboard, make sure that the dashboard configuration includes msbuild.xsl (this will give you a link to "MSBuild Report" on ViewBuildReport page and include some basic information on the page itself).
It feels a bit wierd answering my own question, but there is a chance that someone might have the same question and appreciate my answer so here it is.
I visited the following page http://confluence.public.thoughtworks.org/display/CCNETCOMM/Improved+MSBuild+Integration that describes the use of a different logger than the traditional logger (normal logger produces very large files that bog the server down when performing the XSLT transformations).
The page provides the logger and an XSLT file along with simple and clear instructions on how to incorporate this into your CC.Net project. I tried this logger and XSLT and found I was still getting the raw XML; in fact, ALL of the XML combined in one HUGE page.