we do deployment using Jenkins job and Phing for builds. Our app is made on CakePHP. When I run Cake unit tests I got some errors (as expected).
/app/Console/cake test app AllQaTests --stderr --log-junit
But Phing doesn't perceive them and keeps building instead of stopping the process and marking the build as failed. Is there any elegant way of handling output of tests?
Now we use a separate script which scans test log and seeks for "FAILURE" word.
If your cake command outputs anything other than 0, it would be considered FAILURE by Jenkins, and the job will be marked accordingly.
To answer your question for searching console log for keywords, there is Text-finder plugin that allows to search console log and/or any other file for a RegEx, and mark the build as UNSTABLE or FAILED if found.
I was about to do the same, but eventually tried to catch failed builds via Phing using returnProperty of ExecTask:
<target name="caketest-local" description="Run CakePHP unit tests with PHPUnit and print human readable output.">
<exec dir="${basedir}" executable="${basedir}/app/Console/cake" output="${logdir}/caketest.log" returnProperty="test_result">
<arg line="test" />
<arg line="--stderr"/>
<arg line="--configuration=${basedir}/phpunit-coverage-text.xml" />
<arg line="app" />
<arg line="AllTests" />
</exec>
<if>
<not>
<equals arg1="${test_result}" arg2="0"/>
</not>
<then>
<fail msg="Build FAILED! Check caketest.log for details"/>
</then>
</if>
</target>
Related
What's Not working
At some point, the lists of tests that used to show up in Test Explorer are now no longer appearing. When I try to refresh the list, I get an error that the system is encountering a null value somewhere.
I have the following setup:
VSCODE
Version: 1.70.2 (user setup)
Commit: e4503b30fc78200f846c62cf8091b76ff5547662
Date: 2022-08-16T05:35:13.448Z
Electron: 18.3.5
Chromium: 100.0.4896.160
Node.js: 16.13.2
V8: 10.0.139.17-electron.0
OS: Windows_NT x64 10.0.19044
Widget Solution Setup:
I have a main project, and a test project organized like this:
Widgets.Server Project file:
In part, this is what the project file contains - NB: the platform
<Project Sdk="Microsoft.NET.Sdk.Web">
<PropertyGroup>
<TargetFramework>net5.0-windows</TargetFramework>
<PlatformTarget>AnyCPU</PlatformTarget>
<IsPackable>true</IsPackable>
This is what the Test project file contains - matching platform:
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<TargetFramework>net5.0-windows</TargetFramework>
<PlatformTarget>AnyCPU</PlatformTarget>
<IsPackable>false</IsPackable>
</PropertyGroup>
<ItemGroup>
<PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.1.0" />
<PackageReference Include="TagLibSharp" Version="2.2.0" />
<PackageReference Include="xunit.runner.visualstudio" Version="2.4.3">
<IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
<PrivateAssets>all</PrivateAssets>
</PackageReference>
<PackageReference Include="coverlet.collector" Version="3.1.2">
<IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
<PrivateAssets>all</PrivateAssets>
</PackageReference>
<PackageReference Include="Moq" Version="4.17.2" />
<PackageReference Include="Shouldly" Version="4.0.3" />
<PackageReference Include="xunit" Version="2.4.1" />
</ItemGroup>
<ItemGroup>
<ProjectReference Include="..\Widgets.Server\Widgets.Server.csproj" />
</ItemGroup>
<ItemGroup>
<None Update="appsettings.json">
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
</None>
</ItemGroup>
</Project>
settings.json
In my .vscode folder which lives at the same level as the server folder, this is what the settings file contains: (which is correct as far as path to the test project)
{
"dotnet-test-explorer.testProjectPath": "Widgets.Server.Tests/Widgets.Server.Tests.csproj"
}
Cannot Read Properties Error
Manually Running Tests
When I open up a specific test class, and run it, it seems to work. Please see the picture below:
EDIT 1
The other behavior I'm noticing is that when I first open VSCOde, it finds all the tests and lists the total at the bottom status bar. But sometimes, when I try to debug just one test, I get this error:
Failed to start debugger: "System.InvalidOperationException: The debugger could not be started\r\n at OmniSharp.DotNetTest.Services.DebugTestService.Handle(DebugTestGetStartInfoRequest request) in D:\\a\\1\\s\\src\\OmniSharp.DotNetTest\\Services\\DebugTestService.cs:line 42\r\n at OmniSharp.Endpoint.EndpointHandler`2.GetFirstNotEmptyResponseFromHandlers(ExportHandler`2[] handlers, TRequest request) in D:\\a\\1\\s\\src\\OmniSharp.Host\\Endpoint\\EndpointHandler.cs:line 198\r\n at OmniSharp.Endpoint.EndpointHandler`2.HandleRequestForLanguage(String language, TRequest request, RequestPacket packet) in D:\\a\\1\\s\\src\\OmniSharp.Host\\Endpoint\\EndpointHandler.cs:line 234\r\n at OmniSharp.Endpoint.EndpointHandler`2.Process(RequestPacket packet, LanguageModel model, JToken requestObject) in D:\\a\\1\\s\\src\\OmniSharp.Host\\Endpoint\\EndpointHandler.cs:line 143\r\n at OmniSharp.Stdio.Host.HandleRequest(String json, ILogger logger) in D:\\a\\1\\s\\src\\OmniSharp.Stdio\\Host.cs:line 258"
And in the output window i see this:
----- Debugging test method Widgets.Server.Tests.FeatureToggleSampleControllerShould.Return_String_If_Toggle_Enabled -----
MSBuild version 17.3.0+92e077650 for .NET
C:\Program Files\dotnet\sdk\6.0.400\Microsoft.Common.CurrentVersion.targets(4809,5): error MSB3021: Unable to copy file "C:\Users\me\.nuget\packages\microsoft.testplatform.objectmodel\17.1.0\lib\netcoreapp2.1\Microsoft.TestPlatform.PlatformAbstractions.dll" to "bin\Debug\net5.0-windows\Microsoft.TestPlatform.PlatformAbstractions.dll". The process cannot access the file 'c:\Users\me\Documents\src\codeReview\widgets\server\Widgets.Server.Tests\bin\Debug\net5.0-windows\Microsoft.TestPlatform.PlatformAbstractions.dll' because it is being used by another process. [c:\Users\me\Documents\src\codeReview\widgets\server\Widgets.Server.Tests\Jw.O3M.Server.Tests.csproj]
10 Warning(s)
2 Error(s)
When this happens, I have to delete the bin folder in the Test project and also find and kill the testhost process.
And then I can run a single test by clicking on the "Run Test | Debug Test" options under the [Fact] decorator
But the test explorer remains empty.
No matter what I do i cannot get the list of tests to appear.
This is a know issue in the Test Explorer extension, see https://github.com/formulahendry/vscode-dotnet-test-explorer/issues/370
A workaround is to show the tests in a 'flat' Tree Mode view:
Screenshot of settings in VSCode
I've got some build content that may or may not generate TRX reports, depending on the configuration; this build content is running on TeamCity. I also have an XML report processing build feature enabled so the test results can be reported if they are generated. This build feature seems to be failing the build if no TRX files are found. Is there any way to get parse TRX results if they exist but avoid failing the build otherwise?
The short answer is, not yet.
There is currently an issue raised with JetBrains to add this functionality https://youtrack.jetbrains.com/issue/TW-17939 - you can go there and vote for it to make it a higher priority.
This question is also similar to:
How to run a build step conditionally in TeamCity
Conditionally execute a TeamCity build step
Here is a way to keep the XML/TRX report processor from erroring out: copy a dummy TRX file into a directory searched by the processor. I took a TRX file generated from a successful test run, stripped out some content and replaced various attributes with placeholder values, and marked the UnitTestResult outcome as "NotExecuted". TC will display the test results like this:
This is the content of the dummy TRX file that I used. It's a bit of a hack, but at least it keeps the build from failing.
<?xml version="1.0" encoding="UTF-8"?>
<TestRun id="0240f32b-a8c9-4ad5-ae23-c7b64fe32cd2" name="TrxPlaceholder" runUser="User_Placeholder" xmlns="http://microsoft.com/schemas/VisualStudio/TeamTest/2010">
<Times creation="2018-04-06T14:10:21.7011071-07:00" queuing="2018-04-06T14:10:21.7011176-07:00" start="2018-04-06T14:10:20.5775149-07:00" finish="2018-04-06T14:10:21.7220949-07:00" />
<TestSettings name="default" id="40f31577-b2c8-4f67-845f-58155fce4a2b">
<Deployment runDeploymentRoot="Placeholder" />
</TestSettings>
<Results>
<UnitTestResult executionId="9658777a-f6f7-40cb-a331-95cfec3d7b91" testId="7a75abda-f387-442f-bcb3-fca6aa0ce577" testName="TrxPlaceholder" computerName="Placeholder" duration="00:00:00.0073781" startTime="2018-04-06T14:10:21.3332054-07:00" endTime="2018-04-06T14:10:21.5447253-07:00" testType="13cdc9d9-ddb5-4fa4-a97d-d965ccfc6d4b" outcome="NotExecuted" testListId="8c84fa94-04c1-424b-9868-57a2d4851a1d" relativeResultsDirectory="9658777a-f6f7-40cb-a331-95cfec3d7b91" />
</Results>
<TestDefinitions>
<UnitTest name="TrxPlaceholder" storage="c:\temp\TrxPlaceholder.dll" id="7a75abda-f387-442f-bcb3-fca6aa0ce577">
<Execution id="9658777a-f6f7-40cb-a331-95cfec3d7b91" />
<TestMethod codeBase="c:\temp\TrxPlaceholder.dll" executorUriOfAdapter="executor://mstestadapter/v2" className="TrxPlaceholder" name="TrxPlaceholder" />
</UnitTest>
</TestDefinitions>
<TestLists>
<TestList name="Results Not in a List" id="8c84fa94-04c1-424b-9868-57a2d4851a1d" />
<TestList name="All Loaded Results" id="19431567-8539-422a-85d7-44ee4e166bda" />
</TestLists>
<ResultSummary outcome="Completed">
<Counters total="0" executed="0" passed="0" failed="0" error="0" timeout="0" aborted="0" inconclusive="0" passedButRunAborted="0" notRunnable="0" notExecuted="0" disconnected="0" warning="0" completed="0" inProgress="0" pending="0" />
</ResultSummary>
</TestRun>
For some reason about a quarter of our tests is not being run on TFS, build ends up green but reports only 3/4 saying all tests have passed.
I guess the problem is in some tested async code that perhaps crashes the test runner or keeps running on its own. Locally in VS all tests are passing fine.
Is there any way to enable crash dumps on TFS or how should I address this problem?
EDIT: In TFS web interface the Summary says "All 2217 tests passed" but in Diagnotics section, under the passed tests it says "Test run completed. 3228 tests executed" ...still not all (cca 3450 passing locally in VS2015)
I checked last builds on the server but as far as the history goes, they were already failing then (always different amonut have passed). So I can't trace that.
I also ran tests with vstest.console.exe and all is fine except 5 failed on long filepath (I disabled those with Ignore attribbute to no avail)
Edit by the bounty starter:
Sorry for the text in bounty box. Didn't think that line breaks would be swallowed. Providing the same text here.
I have exact same situation. Say I have 100 tests, but only say 60 are finally in the .trx file (say 50 passed and 10 failed). It would be great to solve this issue. Thanks!
<Target Name="CoreTestConfiguration">
<Exec Command=""C:\Program Files\dotnet\dotnet.exe" vstest /Blame /Diag:"$(SolutionRoot)##########################\bin\Release\diag.txt" "$(SolutionRoot)###########################\#################s.dll" /logger:trx;LogFileName="$(SolutionRoot)\#################################\Tests\bin\Release\TestOutput.trx" /Settings:"$(SolutionRoot)\############################################ests\bin\Release\#################sts.runsettings""
ContinueOnError="true"/>
</Target>
<RunSettings>
<RunConfiguration>
<TestSessionTimeout>4400000</TestSessionTimeout>
<TestTimeout>342000</TestTimeout>
<TargetPlatform>X64</TargetPlatform>
<TargetFrameworkVersion>
.NETCoreApp,Version=v2.0
</TargetFrameworkVersion>
<!-- same with net461: but different count of tests in final trx -->
<!-- <TargetFrameworkVersion>
.NETFramework,Version=v4.6.1
</TargetFrameworkVersion> -->
<DesignMode>False</DesignMode>
<CollectSourceInformation>False</CollectSourceInformation>
</RunConfiguration>
<LoggerRunSettings>
<Loggers>
<Logger friendlyName="blame" enabled="True" />
<Logger friendlyName="Console" uri="logger://microsoft/TestPlatform/ConsoleLogger/v1" assemblyQualifiedName="Microsoft.VisualStudio.TestPlatform.CommandLine.Internal.ConsoleLogger, vstest.console, Version=15.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a" codeBase="C:\Program Files\dotnet\sdk\2.1.503\vstest.console.dll" enabled="True" />
</Loggers>
</LoggerRunSettings>
<DataCollectionRunSettings>
<DataCollectors>
<DataCollector friendlyName="blame" enabled="True">
<Configuration>
<ResultsDirectory>
############################\bin\Release
</ResultsDirectory>
</Configuration>
</DataCollector>
</DataCollectors>
</DataCollectionRunSettings>
</RunSettings>
which version Test Framework are you using?
It seems you want to build/run a dotnet core application.
In my opinion xunit works fine with dotnet core 2.x.
There is also a new version of MS Test framework available which has some interesting improvements. I read that in one of the latest dotnetpro article.
see also
Article, dotnetpro (GER)
Hopefully this will help you to solve this issue,
See ya
Leonhard
I am trying to execute a NAnt build script, which is perfectly working in one environment, but recently I have migrated my Build files to a new machine where I am facing the below issue.
NAnt Code Used :
exec program="cmd" commandline="/c ${build.dir}\XXX.vbs ${build.version}" failonerror="false"
NAne Error received
External Program Failed: cmd (return code was 1)
My Possible tries to overcome this
I have checked the environment variables and found that all are fine and also compared with the old machine to match the same.
Any solutions / comments ??
It seems "cmd" is running and you don't need to provide the full path. The error "External Program Failed: cmd (return code was 1)" means the 'cmd' ran but it produced an error. In other words your VB script (i.e. 'XXX.vbs') is failing. Try the following to find the actual error:
Validate that the expanded value of "${build.dir}\XXX.vbs" is a valid file/location. Double check your working directory if you're working with relative paths.
<echo message="${build.dir}\XXX.vbs" />
or
<echo message="${file::exists(build.dir + '\XXX.vbs')}" />
Run your NAnt script from the command line. This will give you a better error message, like a popup error from Windows Script Host.
C:\YourTools\NAnt.exe -buildfile:MyBuildScript.build
Avoid the command window, redirect the program's output. Append " >> exec.log" to your commandline string in the 'exec' task, then check the contents of the output/log file (i.e. 'exec.log').
<exec program="cmd" commandline="/c ${build.dir}\XXX.vbs ${build.version} >> xxx_exec.log" failonerror="false" />
Alternatively, a better way to use the 'exec' task is to use nested 'arg' elements. This would help in the case that your arguments have spaces in them; and your script will read clearer:
<exec program="cmd" failonerror="false">
<arg value="/c" />
<arg value="${build.dir}\XXX.vbs" />
<arg value="${build.version}" />
</exec>
Lastly, in the case your VB script is failing because of invalid arguments (maybe because of spaces in '${build.version}'), use something like the following to debug the VB script:
WScript.echo "Argument count", wscript.arguments.count
For i = 0 to wscript.arguments.count - 1
Wscript.Echo wscript.arguments.item(i)
Next
Hope this helps.
Setting up CI within Microsoft Team Foundation Server, I have a build that will build the solution and execute all of the unit tests within the solution.
Currently the build will show as partially succeeded if the build is successful and an of the unit test fail. I would like to show the build as failed when a unit test fails.
Can anyone tell me if there is a way to accomplish this functionality?
If you have VS2008 SP1 installed on your build machine then you can simply add the following property to your TFSBuild.proj file:
<TreatTestFailureAsBuildFailure>true</TreatTestFailureAsBuildFailure>
If you don't have SP1 installed and you don't want to install it, then you can do it the old fashioned route as detailed here by the Dev Lead on the TFS Build team, Aaaron Hallberg:
<Target Name="AfterTest">
<!-- Refresh the build properties. -->
<GetBuildProperties TeamFoundationServerUrl="$(TeamFoundationServerUrl)"
BuildUri="$(BuildUri)"
Condition=" '$(IsDesktopBuild)' != 'true' ">
<Output TaskParameter="TestSuccess" PropertyName="TestSuccess" />
</GetBuildProperties>
<!-- Set CompilationStatus to Failed if TestSuccess is false. -->
<SetBuildProperties TeamFoundationServerUrl="$(TeamFoundationServerUrl)"
BuildUri="$(BuildUri)"
CompilationStatus="Failed"
Condition=" '$(IsDesktopBuild)' != 'true' and '$(TestSuccess)' != 'true' ">
</Target>