I've got some build content that may or may not generate TRX reports, depending on the configuration; this build content is running on TeamCity. I also have an XML report processing build feature enabled so the test results can be reported if they are generated. This build feature seems to be failing the build if no TRX files are found. Is there any way to get parse TRX results if they exist but avoid failing the build otherwise?
The short answer is, not yet.
There is currently an issue raised with JetBrains to add this functionality https://youtrack.jetbrains.com/issue/TW-17939 - you can go there and vote for it to make it a higher priority.
This question is also similar to:
How to run a build step conditionally in TeamCity
Conditionally execute a TeamCity build step
Here is a way to keep the XML/TRX report processor from erroring out: copy a dummy TRX file into a directory searched by the processor. I took a TRX file generated from a successful test run, stripped out some content and replaced various attributes with placeholder values, and marked the UnitTestResult outcome as "NotExecuted". TC will display the test results like this:
This is the content of the dummy TRX file that I used. It's a bit of a hack, but at least it keeps the build from failing.
<?xml version="1.0" encoding="UTF-8"?>
<TestRun id="0240f32b-a8c9-4ad5-ae23-c7b64fe32cd2" name="TrxPlaceholder" runUser="User_Placeholder" xmlns="http://microsoft.com/schemas/VisualStudio/TeamTest/2010">
<Times creation="2018-04-06T14:10:21.7011071-07:00" queuing="2018-04-06T14:10:21.7011176-07:00" start="2018-04-06T14:10:20.5775149-07:00" finish="2018-04-06T14:10:21.7220949-07:00" />
<TestSettings name="default" id="40f31577-b2c8-4f67-845f-58155fce4a2b">
<Deployment runDeploymentRoot="Placeholder" />
</TestSettings>
<Results>
<UnitTestResult executionId="9658777a-f6f7-40cb-a331-95cfec3d7b91" testId="7a75abda-f387-442f-bcb3-fca6aa0ce577" testName="TrxPlaceholder" computerName="Placeholder" duration="00:00:00.0073781" startTime="2018-04-06T14:10:21.3332054-07:00" endTime="2018-04-06T14:10:21.5447253-07:00" testType="13cdc9d9-ddb5-4fa4-a97d-d965ccfc6d4b" outcome="NotExecuted" testListId="8c84fa94-04c1-424b-9868-57a2d4851a1d" relativeResultsDirectory="9658777a-f6f7-40cb-a331-95cfec3d7b91" />
</Results>
<TestDefinitions>
<UnitTest name="TrxPlaceholder" storage="c:\temp\TrxPlaceholder.dll" id="7a75abda-f387-442f-bcb3-fca6aa0ce577">
<Execution id="9658777a-f6f7-40cb-a331-95cfec3d7b91" />
<TestMethod codeBase="c:\temp\TrxPlaceholder.dll" executorUriOfAdapter="executor://mstestadapter/v2" className="TrxPlaceholder" name="TrxPlaceholder" />
</UnitTest>
</TestDefinitions>
<TestLists>
<TestList name="Results Not in a List" id="8c84fa94-04c1-424b-9868-57a2d4851a1d" />
<TestList name="All Loaded Results" id="19431567-8539-422a-85d7-44ee4e166bda" />
</TestLists>
<ResultSummary outcome="Completed">
<Counters total="0" executed="0" passed="0" failed="0" error="0" timeout="0" aborted="0" inconclusive="0" passedButRunAborted="0" notRunnable="0" notExecuted="0" disconnected="0" warning="0" completed="0" inProgress="0" pending="0" />
</ResultSummary>
</TestRun>
Related
I am working on a java pluging which is built from a Jenkins project, the build is done using gradle.
From times to times, we may have to skip a test method, or a test overall. I do that by using the #Disable annotation.
However, in this case, the test method, or test, simply disappear from the xml report. I expect the test to be reported as "Skipped" in the xml report so Jenkins can know it was skipped and display it in yellow.
Any solution to that ?
If you are using the junitPlatform() support in Gradle, Gradle will generate XML output similar to the following showing that a test has been skipped.
<testcase name="enabledAndDisabled()" classname="org.junit.jupiter.extensions.EnabledIfTests" time="0.0">
<skipped/>
</testcase>
I copied the above from the TEST-org.junit.jupiter.extensions.EnabledIfTests.xml generated in the platform-tests module of the JUnit 5 build.
In short, using a .runsetting unit test file in a build step on TFS 2015 results in a 503 Service Unavailable exception.
After extensive searching and testing on a new TFS 2015 installation (as also described in TFS2015 new install, 503 Service Unavailable), I may have found the problem.
We are using .runsettings files as described on https://msdn.microsoft.com/en-us/library/jj159530.aspx, in a 'Visual Studio Test' build step on our new TFS2015 installation, as soon as the VSTest.console.exe is called, the next three application pools on the TFS server crash, resulting in a 503 Service Unavailable error:
DefaultAppPool
Microsoft Team Foundation Server Application Pool
Microsoft Team Foundation SErver Message Queueu Application Pool
First my .runsetting file was kind of large, but i did try the next one (I think as small as possible):
<?xml version="1.0" encoding="utf-8"?>
<RunSettings>
<!-- Configurations for data collectors -->
<DataCollectionRunSettings>
<DataCollectors>
<DataCollector friendlyName="Code Coverage" uri="datacollector://Microsoft/CodeCoverage/2.0" assemblyQualifiedName="Microsoft.VisualStudio.Coverage.DynamicCoverageDataCollector, Microsoft.VisualStudio.TraceCollector, Version=14.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a">
<Configuration>
<CodeCoverage>
<!-- Match assembly file paths: -->
<ModulePaths>
<Include>
<ModulePath>.*\.dll$</ModulePath>
<ModulePath>.*\.exe$</ModulePath>
</Include>
<Exclude>
<ModulePath>.*CPPUnitTestFramework.*</ModulePath>
<ModulePath>.*fluentassertions.*</ModulePath>
<ModulePath>.*\.test\.dll$</ModulePath>
</Exclude>
</ModulePaths>
</CodeCoverage>
</Configuration>
</DataCollector>
</DataCollectors>
</DataCollectionRunSettings>
</RunSettings>
When using this one, the app pools crashes. Even if i remove the part entirely, it will not work.
As soon as the next line is called (taken from the log lines of the build step), the app pools crashes.
Executing C:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe "D:\TfsAgents\Agent1\_work\2\s\[...]\Release\Microsoft.QualityTools.Testing.Fakes.dll" "D:\TfsAgents\Agent1\_work\2\s\[...]\Release\Microsoft.VisualStudio.QualityTools.UnitTestFramework.dll" "D:\TfsAgents\Agent1\_work\2\s\[...]\Release\MyClassLib.dll" /Settings:"D:\TfsAgents\Agent1\_work\2\s\[...]\test.runsettings" /EnableCodeCoverage /logger:trx
If you look at the DataCollector tag in the .runsettings file, you see the version is set to 14.0.0.0. If you look at the page of Miscrosoft, they state that it should be 11.0.0.0, but also this version does not work. I thought it might be that version 11.0.0.0 was wrong, so I opened the dll with Telerik JustDecompile, and saw that the 'real' version was 14.0.0.0, so I put that version into the runsetting-file. But no luck.
So for now I'm just remove the runsetting file from the definition of the build, which unfortunately results in a wrong code coverate percentage.. So hopefully someone might have the answer for this.
This problem is limited to the specific runsettings file when used with build agent running on the same machine where IIS server is hosting TFS. There is a workaround available. Within DataCollectors->Configuration->CodeCoverage tag in runsettings file you can add following xml tags with given values :
<UseVerifiableInstrumentation>True</UseVerifiableInstrumentation>
<AllowLowIntegrityProcesses>True</AllowLowIntegrityProcesses>
<CollectFromChildProcesses>True</CollectFromChildProcesses>
<CollectAspDotNet>false</CollectAspDotNet>
This should solve the issue, and also its recommended that in future if you are using runsettings file with Code coverage in Data Collector then these tags should be added.
For some reason about a quarter of our tests is not being run on TFS, build ends up green but reports only 3/4 saying all tests have passed.
I guess the problem is in some tested async code that perhaps crashes the test runner or keeps running on its own. Locally in VS all tests are passing fine.
Is there any way to enable crash dumps on TFS or how should I address this problem?
EDIT: In TFS web interface the Summary says "All 2217 tests passed" but in Diagnotics section, under the passed tests it says "Test run completed. 3228 tests executed" ...still not all (cca 3450 passing locally in VS2015)
I checked last builds on the server but as far as the history goes, they were already failing then (always different amonut have passed). So I can't trace that.
I also ran tests with vstest.console.exe and all is fine except 5 failed on long filepath (I disabled those with Ignore attribbute to no avail)
Edit by the bounty starter:
Sorry for the text in bounty box. Didn't think that line breaks would be swallowed. Providing the same text here.
I have exact same situation. Say I have 100 tests, but only say 60 are finally in the .trx file (say 50 passed and 10 failed). It would be great to solve this issue. Thanks!
<Target Name="CoreTestConfiguration">
<Exec Command=""C:\Program Files\dotnet\dotnet.exe" vstest /Blame /Diag:"$(SolutionRoot)##########################\bin\Release\diag.txt" "$(SolutionRoot)###########################\#################s.dll" /logger:trx;LogFileName="$(SolutionRoot)\#################################\Tests\bin\Release\TestOutput.trx" /Settings:"$(SolutionRoot)\############################################ests\bin\Release\#################sts.runsettings""
ContinueOnError="true"/>
</Target>
<RunSettings>
<RunConfiguration>
<TestSessionTimeout>4400000</TestSessionTimeout>
<TestTimeout>342000</TestTimeout>
<TargetPlatform>X64</TargetPlatform>
<TargetFrameworkVersion>
.NETCoreApp,Version=v2.0
</TargetFrameworkVersion>
<!-- same with net461: but different count of tests in final trx -->
<!-- <TargetFrameworkVersion>
.NETFramework,Version=v4.6.1
</TargetFrameworkVersion> -->
<DesignMode>False</DesignMode>
<CollectSourceInformation>False</CollectSourceInformation>
</RunConfiguration>
<LoggerRunSettings>
<Loggers>
<Logger friendlyName="blame" enabled="True" />
<Logger friendlyName="Console" uri="logger://microsoft/TestPlatform/ConsoleLogger/v1" assemblyQualifiedName="Microsoft.VisualStudio.TestPlatform.CommandLine.Internal.ConsoleLogger, vstest.console, Version=15.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a" codeBase="C:\Program Files\dotnet\sdk\2.1.503\vstest.console.dll" enabled="True" />
</Loggers>
</LoggerRunSettings>
<DataCollectionRunSettings>
<DataCollectors>
<DataCollector friendlyName="blame" enabled="True">
<Configuration>
<ResultsDirectory>
############################\bin\Release
</ResultsDirectory>
</Configuration>
</DataCollector>
</DataCollectors>
</DataCollectionRunSettings>
</RunSettings>
which version Test Framework are you using?
It seems you want to build/run a dotnet core application.
In my opinion xunit works fine with dotnet core 2.x.
There is also a new version of MS Test framework available which has some interesting improvements. I read that in one of the latest dotnetpro article.
see also
Article, dotnetpro (GER)
Hopefully this will help you to solve this issue,
See ya
Leonhard
I don't have much of a programming background, but I have been using Wix to build very basic installations for several years, usually consisting of a browser shortcut with an icon file (The desktop shortcut simply opens 32-bit Internet Explorer to a specific URL). The clientele that use my installers don't usually know about their system and it appears the time has come where I can create a bootstrapper that will run 1 of the 2 .msi files, one for x86 & one for x64. The problem is that the Wix help documentation out there assumes a certain level of knowledge about programming and/or Wix and I need "for dummies" level of help. Using code snippets from Rob's answers to an earlier post here (similarly titled) and a post on another site, I have this in my .wxs:
<?xml version="1.0" encoding="UTF-8"?>
<Wix xmlns="http://schemas.microsoft.com/wix/2006/wi" xmlns:util="http://schemas.microsoft.com/wix/UtilExtension" xmlns:bal="http://schemas.microsoft.com/wix/BalExtension">
<Bundle Name="Intermountain SecureAccess Desktop Icon Installer" Version="1.0.0.0" Manufacturer="Intermountain Healthcare" UpgradeCode="61b75a8f-67f6-43a1-beb9-1a0be426b5a6">
<BootstrapperApplicationRef Id='WixStandardBootstrapperApplication.HyperlinkLicense' />
<Payload SourceFile="86IHCSAHCO.msi"/>
<Payload SourceFile="64IHCSAHCO.msi"/>
<Payload SourceFile="C:\Program Files (x86)\WiX Toolset v3.7\SDK\Microsoft.Deployment.WindowsInstaller.dll"/>
</BootstrapperApplicationRef>
<Chain>
<PackageGroupRef Id='Netfx4Full' />
<MsiPackage SourceFile="IHCSAHCO.msi" Id="InstallationPackageId" Cache="yes" Visible="no"/>
<MsiPackage InstallCondition='NOT VersionNT64' SourceFile='86IHCSAHCO.msi' />
<MsiPackage InstallCondition='VersionNT64' SourceFile='64IHCSAHCO.msi' />
</Chain>
</Bundle>
</Wix>
The goal is to create a single .msi (or .exe) called IHCSAHCO.msi which contains the 2 msi packages 86IHCSAHCO.msi & 64IHCSAHCO.msi and then simply runs one or the other depending on the environment. When I run the batch file (called Burn.wxs) with this in it...
set WIX_ROOT=%programfiles(x86)%\WiX Toolset v3.7\bin
del /q /f *.wixobj *.msi
call "%WIX_ROOT%\candle.exe" Burn.wxs
call "%WIX_ROOT%\light.exe" Burn.wixobj -sice:ICE38
pause
...here is the error:
Burn.wxs
C:\PATH\Burn.wxs(8) : error CNDL0104 : Not a valid source file; detail: The 'Bundle' start tag on line 3 position 4 does not match the end tag of 'BootstrapperApplicationRef'. Line 8, position 7.
I feel as though I am close and am hoping one of you could take a quick peek at the code and give suggestions. It would be much appreciated...
Thanks!
RHH
Your opening BootstrapperApplicationRef element on line 4 is an empty element - it shouldn't end in />.
What are you using to edit your files? Pasting your file into Notepad++ and turning on XML highlighting identified the problem pretty quickly.
Setting up CI within Microsoft Team Foundation Server, I have a build that will build the solution and execute all of the unit tests within the solution.
Currently the build will show as partially succeeded if the build is successful and an of the unit test fail. I would like to show the build as failed when a unit test fails.
Can anyone tell me if there is a way to accomplish this functionality?
If you have VS2008 SP1 installed on your build machine then you can simply add the following property to your TFSBuild.proj file:
<TreatTestFailureAsBuildFailure>true</TreatTestFailureAsBuildFailure>
If you don't have SP1 installed and you don't want to install it, then you can do it the old fashioned route as detailed here by the Dev Lead on the TFS Build team, Aaaron Hallberg:
<Target Name="AfterTest">
<!-- Refresh the build properties. -->
<GetBuildProperties TeamFoundationServerUrl="$(TeamFoundationServerUrl)"
BuildUri="$(BuildUri)"
Condition=" '$(IsDesktopBuild)' != 'true' ">
<Output TaskParameter="TestSuccess" PropertyName="TestSuccess" />
</GetBuildProperties>
<!-- Set CompilationStatus to Failed if TestSuccess is false. -->
<SetBuildProperties TeamFoundationServerUrl="$(TeamFoundationServerUrl)"
BuildUri="$(BuildUri)"
CompilationStatus="Failed"
Condition=" '$(IsDesktopBuild)' != 'true' and '$(TestSuccess)' != 'true' ">
</Target>