TFS Not all tests are run although the build is green - unit-testing

For some reason about a quarter of our tests is not being run on TFS, build ends up green but reports only 3/4 saying all tests have passed.
I guess the problem is in some tested async code that perhaps crashes the test runner or keeps running on its own. Locally in VS all tests are passing fine.
Is there any way to enable crash dumps on TFS or how should I address this problem?
EDIT: In TFS web interface the Summary says "All 2217 tests passed" but in Diagnotics section, under the passed tests it says "Test run completed. 3228 tests executed" ...still not all (cca 3450 passing locally in VS2015)
I checked last builds on the server but as far as the history goes, they were already failing then (always different amonut have passed). So I can't trace that.
I also ran tests with vstest.console.exe and all is fine except 5 failed on long filepath (I disabled those with Ignore attribbute to no avail)
Edit by the bounty starter:
Sorry for the text in bounty box. Didn't think that line breaks would be swallowed. Providing the same text here.
I have exact same situation. Say I have 100 tests, but only say 60 are finally in the .trx file (say 50 passed and 10 failed). It would be great to solve this issue. Thanks!
<Target Name="CoreTestConfiguration">
<Exec Command=""C:\Program Files\dotnet\dotnet.exe" vstest /Blame /Diag:"$(SolutionRoot)##########################\bin\Release\diag.txt" "$(SolutionRoot)###########################\#################s.dll" /logger:trx;LogFileName="$(SolutionRoot)\#################################\Tests\bin\Release\TestOutput.trx" /Settings:"$(SolutionRoot)\############################################ests\bin\Release\#################sts.runsettings""
ContinueOnError="true"/>
</Target>
<RunSettings>
<RunConfiguration>
<TestSessionTimeout>4400000</TestSessionTimeout>
<TestTimeout>342000</TestTimeout>
<TargetPlatform>X64</TargetPlatform>
<TargetFrameworkVersion>
.NETCoreApp,Version=v2.0
</TargetFrameworkVersion>
<!-- same with net461: but different count of tests in final trx -->
<!-- <TargetFrameworkVersion>
.NETFramework,Version=v4.6.1
</TargetFrameworkVersion> -->
<DesignMode>False</DesignMode>
<CollectSourceInformation>False</CollectSourceInformation>
</RunConfiguration>
<LoggerRunSettings>
<Loggers>
<Logger friendlyName="blame" enabled="True" />
<Logger friendlyName="Console" uri="logger://microsoft/TestPlatform/ConsoleLogger/v1" assemblyQualifiedName="Microsoft.VisualStudio.TestPlatform.CommandLine.Internal.ConsoleLogger, vstest.console, Version=15.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a" codeBase="C:\Program Files\dotnet\sdk\2.1.503\vstest.console.dll" enabled="True" />
</Loggers>
</LoggerRunSettings>
<DataCollectionRunSettings>
<DataCollectors>
<DataCollector friendlyName="blame" enabled="True">
<Configuration>
<ResultsDirectory>
############################\bin\Release
</ResultsDirectory>
</Configuration>
</DataCollector>
</DataCollectors>
</DataCollectionRunSettings>
</RunSettings>

which version Test Framework are you using?
It seems you want to build/run a dotnet core application.
In my opinion xunit works fine with dotnet core 2.x.
There is also a new version of MS Test framework available which has some interesting improvements. I read that in one of the latest dotnetpro article.
see also
Article, dotnetpro (GER)
Hopefully this will help you to solve this issue,
See ya
Leonhard

Related

Junit5 does not report skipped tests

I am working on a java pluging which is built from a Jenkins project, the build is done using gradle.
From times to times, we may have to skip a test method, or a test overall. I do that by using the #Disable annotation.
However, in this case, the test method, or test, simply disappear from the xml report. I expect the test to be reported as "Skipped" in the xml report so Jenkins can know it was skipped and display it in yellow.
Any solution to that ?
If you are using the junitPlatform() support in Gradle, Gradle will generate XML output similar to the following showing that a test has been skipped.
<testcase name="enabledAndDisabled()" classname="org.junit.jupiter.extensions.EnabledIfTests" time="0.0">
<skipped/>
</testcase>
I copied the above from the TEST-org.junit.jupiter.extensions.EnabledIfTests.xml generated in the platform-tests module of the JUnit 5 build.

How to prevent TeamCity XML report processing from failing build

I've got some build content that may or may not generate TRX reports, depending on the configuration; this build content is running on TeamCity. I also have an XML report processing build feature enabled so the test results can be reported if they are generated. This build feature seems to be failing the build if no TRX files are found. Is there any way to get parse TRX results if they exist but avoid failing the build otherwise?
The short answer is, not yet.
There is currently an issue raised with JetBrains to add this functionality https://youtrack.jetbrains.com/issue/TW-17939 - you can go there and vote for it to make it a higher priority.
This question is also similar to:
How to run a build step conditionally in TeamCity
Conditionally execute a TeamCity build step
Here is a way to keep the XML/TRX report processor from erroring out: copy a dummy TRX file into a directory searched by the processor. I took a TRX file generated from a successful test run, stripped out some content and replaced various attributes with placeholder values, and marked the UnitTestResult outcome as "NotExecuted". TC will display the test results like this:
This is the content of the dummy TRX file that I used. It's a bit of a hack, but at least it keeps the build from failing.
<?xml version="1.0" encoding="UTF-8"?>
<TestRun id="0240f32b-a8c9-4ad5-ae23-c7b64fe32cd2" name="TrxPlaceholder" runUser="User_Placeholder" xmlns="http://microsoft.com/schemas/VisualStudio/TeamTest/2010">
<Times creation="2018-04-06T14:10:21.7011071-07:00" queuing="2018-04-06T14:10:21.7011176-07:00" start="2018-04-06T14:10:20.5775149-07:00" finish="2018-04-06T14:10:21.7220949-07:00" />
<TestSettings name="default" id="40f31577-b2c8-4f67-845f-58155fce4a2b">
<Deployment runDeploymentRoot="Placeholder" />
</TestSettings>
<Results>
<UnitTestResult executionId="9658777a-f6f7-40cb-a331-95cfec3d7b91" testId="7a75abda-f387-442f-bcb3-fca6aa0ce577" testName="TrxPlaceholder" computerName="Placeholder" duration="00:00:00.0073781" startTime="2018-04-06T14:10:21.3332054-07:00" endTime="2018-04-06T14:10:21.5447253-07:00" testType="13cdc9d9-ddb5-4fa4-a97d-d965ccfc6d4b" outcome="NotExecuted" testListId="8c84fa94-04c1-424b-9868-57a2d4851a1d" relativeResultsDirectory="9658777a-f6f7-40cb-a331-95cfec3d7b91" />
</Results>
<TestDefinitions>
<UnitTest name="TrxPlaceholder" storage="c:\temp\TrxPlaceholder.dll" id="7a75abda-f387-442f-bcb3-fca6aa0ce577">
<Execution id="9658777a-f6f7-40cb-a331-95cfec3d7b91" />
<TestMethod codeBase="c:\temp\TrxPlaceholder.dll" executorUriOfAdapter="executor://mstestadapter/v2" className="TrxPlaceholder" name="TrxPlaceholder" />
</UnitTest>
</TestDefinitions>
<TestLists>
<TestList name="Results Not in a List" id="8c84fa94-04c1-424b-9868-57a2d4851a1d" />
<TestList name="All Loaded Results" id="19431567-8539-422a-85d7-44ee4e166bda" />
</TestLists>
<ResultSummary outcome="Completed">
<Counters total="0" executed="0" passed="0" failed="0" error="0" timeout="0" aborted="0" inconclusive="0" passedButRunAborted="0" notRunnable="0" notExecuted="0" disconnected="0" warning="0" completed="0" inProgress="0" pending="0" />
</ResultSummary>
</TestRun>

TFS2015 503 service unavailable using .runsetting unit test file

In short, using a .runsetting unit test file in a build step on TFS 2015 results in a 503 Service Unavailable exception.
After extensive searching and testing on a new TFS 2015 installation (as also described in TFS2015 new install, 503 Service Unavailable), I may have found the problem.
We are using .runsettings files as described on https://msdn.microsoft.com/en-us/library/jj159530.aspx, in a 'Visual Studio Test' build step on our new TFS2015 installation, as soon as the VSTest.console.exe is called, the next three application pools on the TFS server crash, resulting in a 503 Service Unavailable error:
DefaultAppPool
Microsoft Team Foundation Server Application Pool
Microsoft Team Foundation SErver Message Queueu Application Pool
First my .runsetting file was kind of large, but i did try the next one (I think as small as possible):
<?xml version="1.0" encoding="utf-8"?>
<RunSettings>
<!-- Configurations for data collectors -->
<DataCollectionRunSettings>
<DataCollectors>
<DataCollector friendlyName="Code Coverage" uri="datacollector://Microsoft/CodeCoverage/2.0" assemblyQualifiedName="Microsoft.VisualStudio.Coverage.DynamicCoverageDataCollector, Microsoft.VisualStudio.TraceCollector, Version=14.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a">
<Configuration>
<CodeCoverage>
<!-- Match assembly file paths: -->
<ModulePaths>
<Include>
<ModulePath>.*\.dll$</ModulePath>
<ModulePath>.*\.exe$</ModulePath>
</Include>
<Exclude>
<ModulePath>.*CPPUnitTestFramework.*</ModulePath>
<ModulePath>.*fluentassertions.*</ModulePath>
<ModulePath>.*\.test\.dll$</ModulePath>
</Exclude>
</ModulePaths>
</CodeCoverage>
</Configuration>
</DataCollector>
</DataCollectors>
</DataCollectionRunSettings>
</RunSettings>
When using this one, the app pools crashes. Even if i remove the part entirely, it will not work.
As soon as the next line is called (taken from the log lines of the build step), the app pools crashes.
Executing C:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe "D:\TfsAgents\Agent1\_work\2\s\[...]\Release\Microsoft.QualityTools.Testing.Fakes.dll" "D:\TfsAgents\Agent1\_work\2\s\[...]\Release\Microsoft.VisualStudio.QualityTools.UnitTestFramework.dll" "D:\TfsAgents\Agent1\_work\2\s\[...]\Release\MyClassLib.dll" /Settings:"D:\TfsAgents\Agent1\_work\2\s\[...]\test.runsettings" /EnableCodeCoverage /logger:trx
If you look at the DataCollector tag in the .runsettings file, you see the version is set to 14.0.0.0. If you look at the page of Miscrosoft, they state that it should be 11.0.0.0, but also this version does not work. I thought it might be that version 11.0.0.0 was wrong, so I opened the dll with Telerik JustDecompile, and saw that the 'real' version was 14.0.0.0, so I put that version into the runsetting-file. But no luck.
So for now I'm just remove the runsetting file from the definition of the build, which unfortunately results in a wrong code coverate percentage.. So hopefully someone might have the answer for this.
This problem is limited to the specific runsettings file when used with build agent running on the same machine where IIS server is hosting TFS. There is a workaround available. Within DataCollectors->Configuration->CodeCoverage tag in runsettings file you can add following xml tags with given values :
<UseVerifiableInstrumentation>True</UseVerifiableInstrumentation>
<AllowLowIntegrityProcesses>True</AllowLowIntegrityProcesses>
<CollectFromChildProcesses>True</CollectFromChildProcesses>
<CollectAspDotNet>false</CollectAspDotNet>
This should solve the issue, and also its recommended that in future if you are using runsettings file with Code coverage in Data Collector then these tags should be added.

Server Collection cannot be empty - Windows Server AppFabric 1.1

I get this error when i use multiple dataCacheClients. Ive seen this question and this msdn question but at the time of AppFabric 1.0, multiple dataCacheClients were not possible. Microsoft added this feature in AppFabric 1.1 (see changelog), which I'm using currently. Any ideas on why I get this error? Here's my config file:
<configuration>
<configSections>
<section name="dataCacheClients"
type="Microsoft.ApplicationServer.Caching.DataCacheClientsSection, Microsoft.ApplicationServer.Caching.Core"
allowLocation="true"
allowDefinition="Everywhere"/>
</configSections>
<dataCacheClients>
<!--client 1 for caching-->
<dataCacheClient name="dataCacheClient1">
<localCache isEnabled="false" sync="NotificationBased" objectCount="100000"/>
<clientNotification pollInterval="5"/>
<hosts>
<host name="!2345623ghf1.fg.com" cachePort="22233"/>
</hosts>
<securityProperties mode="None" protectionLevel="None" />
<transportProperties maxBufferPoolSize="2147483647" maxBufferSize="2147483647" channelInitializationTimeout="60000" receiveTimeout="900000"/>
</dataCacheClient>
<!-- client 2 for session -->
<dataCacheClient name="dataCacheClient2">
<localCache isEnabled="false" sync="NotificationBased" objectCount="100000"/>
<clientNotification pollInterval="5"/>
<hosts>
<host name="!2345623ghf2.fg.com" cachePort="22233"/>
</hosts>
<securityProperties mode="None" protectionLevel="None" />
<transportProperties maxBufferPoolSize="2147483647" maxBufferSize="2147483647" channelInitializationTimeout="60000" receiveTimeout="900000"/>
</dataCacheClient>
</dataCacheClients>
<system.web>
<compilation debug="true" targetFramework="4.0">
</compilation>
<sessionState
mode="Custom"
customProvider="AppFabricCacheSessionStoreProvider">
<providers>
<add name="AppFabricCacheSessionStoreProvider"
type="Microsoft.Web.DistributedCache.DistributedCacheSessionStateStoreProvider, Microsoft.Web.DistributedCache"
cacheName="default"
dataCacheClientName="dataCacheClient2" />
</providers>
</sessionState>
NOTE:
I'm using the DLL's found in .\Program Files\AppFabric 1.1 for Windows Server
More Error details :
I was finally able to get it working. There were some rudimentary but stupid mistakes I committed. Hopefully someone with the same issues would be able to get some guidance on this by my answer. The mistakes I committed/how it was rectified :
1) Check the DLL's CLR version
Always, I mean, always check the version of the DLLs you're using to refer in your consuming client. The DLLs to look out for are :
Microsoft.ApplicationServer.Caching.Core.dll
Microsoft.ApplicationServer.Caching.Client.dll
Microsoft.WindowsFabric.Common.dll
Microsoft.WindowsFabric.Data.Common.dll
Microsoft.Web.DistributedCache (this was what was causing me problems - I had an older version added in my solution; there's no need to refer this, just having this DLL in the same folder as the Caching.Core and Caching.Client is enough)
A good (or rather fail-proof) way to get the latest DLLs is to download & install Appfabric and get the DLLs from .\Program Files\AppFabric 1.1, add it in a folder inside your project and refer it from there.
2) DataCacheFactory must have a reference to what dataCacheClient its referring to
Just like how AppFabricCacheSessionStoreProvider must contain a dataCacheClientName attribute to refer to a specific cluster put in dataCacheClient, DataCacheFactory init in code must also contain a reference to the dataCacheClient which is gonna take care of caching :
DataCacheFactory _factory = new DataCacheFactory(new DataCacheFactoryConfiguration("dataCacheClient1"))
Thanks to everyone who helped in solving this issue!
I had this problem on one of our development machines and it was ultimately found to be installation of Windows Server AppFabric. My configuration was totally correct. The same configuration worked on all machines except one and once we un-installed Windows Server AppFabric from that machine and installed Microsoft AppFabric 1.1 version, the issue was resolved.
I'm also using AppFabric to store Asp.NET SessionState.

Fail TFS Build on Single Unit Test Failure

Setting up CI within Microsoft Team Foundation Server, I have a build that will build the solution and execute all of the unit tests within the solution.
Currently the build will show as partially succeeded if the build is successful and an of the unit test fail. I would like to show the build as failed when a unit test fails.
Can anyone tell me if there is a way to accomplish this functionality?
If you have VS2008 SP1 installed on your build machine then you can simply add the following property to your TFSBuild.proj file:
<TreatTestFailureAsBuildFailure>true</TreatTestFailureAsBuildFailure>
If you don't have SP1 installed and you don't want to install it, then you can do it the old fashioned route as detailed here by the Dev Lead on the TFS Build team, Aaaron Hallberg:
<Target Name="AfterTest">
<!-- Refresh the build properties. -->
<GetBuildProperties TeamFoundationServerUrl="$(TeamFoundationServerUrl)"
BuildUri="$(BuildUri)"
Condition=" '$(IsDesktopBuild)' != 'true' ">
<Output TaskParameter="TestSuccess" PropertyName="TestSuccess" />
</GetBuildProperties>
<!-- Set CompilationStatus to Failed if TestSuccess is false. -->
<SetBuildProperties TeamFoundationServerUrl="$(TeamFoundationServerUrl)"
BuildUri="$(BuildUri)"
CompilationStatus="Failed"
Condition=" '$(IsDesktopBuild)' != 'true' and '$(TestSuccess)' != 'true' ">
</Target>