Setting up CI within Microsoft Team Foundation Server, I have a build that will build the solution and execute all of the unit tests within the solution.
Currently the build will show as partially succeeded if the build is successful and an of the unit test fail. I would like to show the build as failed when a unit test fails.
Can anyone tell me if there is a way to accomplish this functionality?
If you have VS2008 SP1 installed on your build machine then you can simply add the following property to your TFSBuild.proj file:
<TreatTestFailureAsBuildFailure>true</TreatTestFailureAsBuildFailure>
If you don't have SP1 installed and you don't want to install it, then you can do it the old fashioned route as detailed here by the Dev Lead on the TFS Build team, Aaaron Hallberg:
<Target Name="AfterTest">
<!-- Refresh the build properties. -->
<GetBuildProperties TeamFoundationServerUrl="$(TeamFoundationServerUrl)"
BuildUri="$(BuildUri)"
Condition=" '$(IsDesktopBuild)' != 'true' ">
<Output TaskParameter="TestSuccess" PropertyName="TestSuccess" />
</GetBuildProperties>
<!-- Set CompilationStatus to Failed if TestSuccess is false. -->
<SetBuildProperties TeamFoundationServerUrl="$(TeamFoundationServerUrl)"
BuildUri="$(BuildUri)"
CompilationStatus="Failed"
Condition=" '$(IsDesktopBuild)' != 'true' and '$(TestSuccess)' != 'true' ">
</Target>
Related
I have a pretty straight forward WiX project. Nothing fancy. When trying to perform a MajorUpgrade over an existing installation, it is unable to start the service and inevitably rolls back to the previous version and starts the service just fine. I have removed the Start="install" and manually started the application successfully, so I know it's not a dependency issue.
I have searched endlessly and found no answers to my problem.
<MajorUpgrade DowngradeErrorMessage="A newer version of [ProductName] is already installed." Schedule="afterInstallFinalize" />
My service install:
<ServiceInstall
Id="ServiceInstaller"
Type="ownProcess"
Name="LsdnService"
DisplayName="Lsdn Service"
Description="Placeholder for now."
Start="auto"
Account="[SERVICEACCOUNT]"
Password="[SERVICEPASSWORD]"
ErrorControl="normal"/>
<ServiceControl Id="StartService" Start="install" Stop="both" Remove="uninstall" Name="LsdnService" Wait="yes" />
I dumped the MSI log to a file and got this error but it is quite vague.
MSI (s) (18:48) [22:41:27:349]: Note: 1: 2205 2: 3: Error
MSI (s) (18:48) [22:41:27:349]: Note: 1: 2228 2: 3: Error 4: SELECT `Message` FROM `Error` WHERE `Error` = 1920
There are some registry modifications during an installation. The installer attempts to read from the registry and inherit the already existing values.
<Property Id="LSDNADDRESS" Value="127.0.0.1">
<RegistrySearch Id="LsdnAddressProperty" Root="HKLM" Key="$(var.RegistryKey)" Name="LsdnAddress" Type="raw" />
</Property>
<Property Id="LSDNPORT" Value="9920">
<RegistrySearch Id="LsdnPortProperty" Root="HKLM" Key="$(var.RegistryKey)" Name="LsdnPort" Type="raw" />
</Property>
<Property Id="LSDNKEY" Value="6f380b07-0b54-4904-8303-95d1ec45d453">
<RegistrySearch Id="LsdnKeyProperty" Root="HKLM" Key="$(var.RegistryKey)" Name="LsdnKey" Type="raw" />
</Property>
Debugging Results: Following a lot of debugging (by original poster - OP) this turned out to be a known MSI issue described here:
https://wix-users.narkive.com/EMfQPDrM/a-bug-get-reg-sz-when-using-type-integer. Nice search work.
What is in a DWORD? (a REG_SZ apparently): Essentially MSI "converts" a DWORD value found via a RegistrySearch
operation to a formatted string - REG_SZ - during upgrade
installations (could be more involved too). This causes services that
expect a DWORD value to fall over on startup during major
upgrades. A very exotic error.
Workaround: One can try to "solve" this problem by making the service code capable of reading both DWORD and REG_SZ.
This yields a more robust solution than solving the problem in a
custom action since it is a "permanent" fix as long as the code is in
there (and the presence of the code alerts other developers about the
problem). Or maybe use only REG_SZ?
Quick Checks: Check the service password and login - obviously. Anything in the
Event Viewer? Windows Key + Tap R + eventvwr.msc + Enter. How to use the Event Viewer to troubleshoot problems with a Windows Service. Perhaps you can try to do a folder diff on the before and after folders and see if you see something unexpected in
the config files? Naturally there will be lots of binary
differences, but check the text files (also encoding). Check the MSI log file
again and search for "value 3" as described here: Tips For Checking MSI Log
Files. Manually copy the new files in place and attempt to start the service via the services.msc applet.
Service Experts: Windows Services Frequently Asked Questions (FAQ). Content seems to be up to date - at face value at least.
These guys claim to be experts on services. I have no idea who they
are.
Look in the "Errors" section in the link above. Here are some
extracts:
1053: The service did not respond to the start or control request in a timely fashion
Why doesn't my Windows Service start automatically after a reboot?
1069: The service did not start due to a logon failure
Generic Check Lists: If none of the above does anything, maybe try these "torpedoes full spread" check-lists (just ideas to start debugging):
Desktop applicaton not opening after installation in client system
Windows Application Startup Error Exception code: 0xe0434352
General Purpose Debugging: Throwing in some general-purpose debugging approaches.
Custom Action Debugging: WIxsharp debug custom action in console
Dependency Scanning: Which winform project files should be packed up into the installer
Some Further Links:
C# Debug folder when copied to another location does not run the exe
wix service install not enough permission
How exactly does the WiX 'Service Install' work internally?
WiX Toolset PermissionEx Problem - App Does Not Run After Installation
It certainly could be a dependency issue. For example, GAC / WinSXS files don't get installed into the GAC until the commit phase which is after StartServices.
I would leave the Start="Install" in and while it's sitting at the failed to start prompt inspect the state of the machine and debug the service start manually. I bet you'll find something missing.
I've got some build content that may or may not generate TRX reports, depending on the configuration; this build content is running on TeamCity. I also have an XML report processing build feature enabled so the test results can be reported if they are generated. This build feature seems to be failing the build if no TRX files are found. Is there any way to get parse TRX results if they exist but avoid failing the build otherwise?
The short answer is, not yet.
There is currently an issue raised with JetBrains to add this functionality https://youtrack.jetbrains.com/issue/TW-17939 - you can go there and vote for it to make it a higher priority.
This question is also similar to:
How to run a build step conditionally in TeamCity
Conditionally execute a TeamCity build step
Here is a way to keep the XML/TRX report processor from erroring out: copy a dummy TRX file into a directory searched by the processor. I took a TRX file generated from a successful test run, stripped out some content and replaced various attributes with placeholder values, and marked the UnitTestResult outcome as "NotExecuted". TC will display the test results like this:
This is the content of the dummy TRX file that I used. It's a bit of a hack, but at least it keeps the build from failing.
<?xml version="1.0" encoding="UTF-8"?>
<TestRun id="0240f32b-a8c9-4ad5-ae23-c7b64fe32cd2" name="TrxPlaceholder" runUser="User_Placeholder" xmlns="http://microsoft.com/schemas/VisualStudio/TeamTest/2010">
<Times creation="2018-04-06T14:10:21.7011071-07:00" queuing="2018-04-06T14:10:21.7011176-07:00" start="2018-04-06T14:10:20.5775149-07:00" finish="2018-04-06T14:10:21.7220949-07:00" />
<TestSettings name="default" id="40f31577-b2c8-4f67-845f-58155fce4a2b">
<Deployment runDeploymentRoot="Placeholder" />
</TestSettings>
<Results>
<UnitTestResult executionId="9658777a-f6f7-40cb-a331-95cfec3d7b91" testId="7a75abda-f387-442f-bcb3-fca6aa0ce577" testName="TrxPlaceholder" computerName="Placeholder" duration="00:00:00.0073781" startTime="2018-04-06T14:10:21.3332054-07:00" endTime="2018-04-06T14:10:21.5447253-07:00" testType="13cdc9d9-ddb5-4fa4-a97d-d965ccfc6d4b" outcome="NotExecuted" testListId="8c84fa94-04c1-424b-9868-57a2d4851a1d" relativeResultsDirectory="9658777a-f6f7-40cb-a331-95cfec3d7b91" />
</Results>
<TestDefinitions>
<UnitTest name="TrxPlaceholder" storage="c:\temp\TrxPlaceholder.dll" id="7a75abda-f387-442f-bcb3-fca6aa0ce577">
<Execution id="9658777a-f6f7-40cb-a331-95cfec3d7b91" />
<TestMethod codeBase="c:\temp\TrxPlaceholder.dll" executorUriOfAdapter="executor://mstestadapter/v2" className="TrxPlaceholder" name="TrxPlaceholder" />
</UnitTest>
</TestDefinitions>
<TestLists>
<TestList name="Results Not in a List" id="8c84fa94-04c1-424b-9868-57a2d4851a1d" />
<TestList name="All Loaded Results" id="19431567-8539-422a-85d7-44ee4e166bda" />
</TestLists>
<ResultSummary outcome="Completed">
<Counters total="0" executed="0" passed="0" failed="0" error="0" timeout="0" aborted="0" inconclusive="0" passedButRunAborted="0" notRunnable="0" notExecuted="0" disconnected="0" warning="0" completed="0" inProgress="0" pending="0" />
</ResultSummary>
</TestRun>
In short, using a .runsetting unit test file in a build step on TFS 2015 results in a 503 Service Unavailable exception.
After extensive searching and testing on a new TFS 2015 installation (as also described in TFS2015 new install, 503 Service Unavailable), I may have found the problem.
We are using .runsettings files as described on https://msdn.microsoft.com/en-us/library/jj159530.aspx, in a 'Visual Studio Test' build step on our new TFS2015 installation, as soon as the VSTest.console.exe is called, the next three application pools on the TFS server crash, resulting in a 503 Service Unavailable error:
DefaultAppPool
Microsoft Team Foundation Server Application Pool
Microsoft Team Foundation SErver Message Queueu Application Pool
First my .runsetting file was kind of large, but i did try the next one (I think as small as possible):
<?xml version="1.0" encoding="utf-8"?>
<RunSettings>
<!-- Configurations for data collectors -->
<DataCollectionRunSettings>
<DataCollectors>
<DataCollector friendlyName="Code Coverage" uri="datacollector://Microsoft/CodeCoverage/2.0" assemblyQualifiedName="Microsoft.VisualStudio.Coverage.DynamicCoverageDataCollector, Microsoft.VisualStudio.TraceCollector, Version=14.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a">
<Configuration>
<CodeCoverage>
<!-- Match assembly file paths: -->
<ModulePaths>
<Include>
<ModulePath>.*\.dll$</ModulePath>
<ModulePath>.*\.exe$</ModulePath>
</Include>
<Exclude>
<ModulePath>.*CPPUnitTestFramework.*</ModulePath>
<ModulePath>.*fluentassertions.*</ModulePath>
<ModulePath>.*\.test\.dll$</ModulePath>
</Exclude>
</ModulePaths>
</CodeCoverage>
</Configuration>
</DataCollector>
</DataCollectors>
</DataCollectionRunSettings>
</RunSettings>
When using this one, the app pools crashes. Even if i remove the part entirely, it will not work.
As soon as the next line is called (taken from the log lines of the build step), the app pools crashes.
Executing C:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe "D:\TfsAgents\Agent1\_work\2\s\[...]\Release\Microsoft.QualityTools.Testing.Fakes.dll" "D:\TfsAgents\Agent1\_work\2\s\[...]\Release\Microsoft.VisualStudio.QualityTools.UnitTestFramework.dll" "D:\TfsAgents\Agent1\_work\2\s\[...]\Release\MyClassLib.dll" /Settings:"D:\TfsAgents\Agent1\_work\2\s\[...]\test.runsettings" /EnableCodeCoverage /logger:trx
If you look at the DataCollector tag in the .runsettings file, you see the version is set to 14.0.0.0. If you look at the page of Miscrosoft, they state that it should be 11.0.0.0, but also this version does not work. I thought it might be that version 11.0.0.0 was wrong, so I opened the dll with Telerik JustDecompile, and saw that the 'real' version was 14.0.0.0, so I put that version into the runsetting-file. But no luck.
So for now I'm just remove the runsetting file from the definition of the build, which unfortunately results in a wrong code coverate percentage.. So hopefully someone might have the answer for this.
This problem is limited to the specific runsettings file when used with build agent running on the same machine where IIS server is hosting TFS. There is a workaround available. Within DataCollectors->Configuration->CodeCoverage tag in runsettings file you can add following xml tags with given values :
<UseVerifiableInstrumentation>True</UseVerifiableInstrumentation>
<AllowLowIntegrityProcesses>True</AllowLowIntegrityProcesses>
<CollectFromChildProcesses>True</CollectFromChildProcesses>
<CollectAspDotNet>false</CollectAspDotNet>
This should solve the issue, and also its recommended that in future if you are using runsettings file with Code coverage in Data Collector then these tags should be added.
For some reason about a quarter of our tests is not being run on TFS, build ends up green but reports only 3/4 saying all tests have passed.
I guess the problem is in some tested async code that perhaps crashes the test runner or keeps running on its own. Locally in VS all tests are passing fine.
Is there any way to enable crash dumps on TFS or how should I address this problem?
EDIT: In TFS web interface the Summary says "All 2217 tests passed" but in Diagnotics section, under the passed tests it says "Test run completed. 3228 tests executed" ...still not all (cca 3450 passing locally in VS2015)
I checked last builds on the server but as far as the history goes, they were already failing then (always different amonut have passed). So I can't trace that.
I also ran tests with vstest.console.exe and all is fine except 5 failed on long filepath (I disabled those with Ignore attribbute to no avail)
Edit by the bounty starter:
Sorry for the text in bounty box. Didn't think that line breaks would be swallowed. Providing the same text here.
I have exact same situation. Say I have 100 tests, but only say 60 are finally in the .trx file (say 50 passed and 10 failed). It would be great to solve this issue. Thanks!
<Target Name="CoreTestConfiguration">
<Exec Command=""C:\Program Files\dotnet\dotnet.exe" vstest /Blame /Diag:"$(SolutionRoot)##########################\bin\Release\diag.txt" "$(SolutionRoot)###########################\#################s.dll" /logger:trx;LogFileName="$(SolutionRoot)\#################################\Tests\bin\Release\TestOutput.trx" /Settings:"$(SolutionRoot)\############################################ests\bin\Release\#################sts.runsettings""
ContinueOnError="true"/>
</Target>
<RunSettings>
<RunConfiguration>
<TestSessionTimeout>4400000</TestSessionTimeout>
<TestTimeout>342000</TestTimeout>
<TargetPlatform>X64</TargetPlatform>
<TargetFrameworkVersion>
.NETCoreApp,Version=v2.0
</TargetFrameworkVersion>
<!-- same with net461: but different count of tests in final trx -->
<!-- <TargetFrameworkVersion>
.NETFramework,Version=v4.6.1
</TargetFrameworkVersion> -->
<DesignMode>False</DesignMode>
<CollectSourceInformation>False</CollectSourceInformation>
</RunConfiguration>
<LoggerRunSettings>
<Loggers>
<Logger friendlyName="blame" enabled="True" />
<Logger friendlyName="Console" uri="logger://microsoft/TestPlatform/ConsoleLogger/v1" assemblyQualifiedName="Microsoft.VisualStudio.TestPlatform.CommandLine.Internal.ConsoleLogger, vstest.console, Version=15.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a" codeBase="C:\Program Files\dotnet\sdk\2.1.503\vstest.console.dll" enabled="True" />
</Loggers>
</LoggerRunSettings>
<DataCollectionRunSettings>
<DataCollectors>
<DataCollector friendlyName="blame" enabled="True">
<Configuration>
<ResultsDirectory>
############################\bin\Release
</ResultsDirectory>
</Configuration>
</DataCollector>
</DataCollectors>
</DataCollectionRunSettings>
</RunSettings>
which version Test Framework are you using?
It seems you want to build/run a dotnet core application.
In my opinion xunit works fine with dotnet core 2.x.
There is also a new version of MS Test framework available which has some interesting improvements. I read that in one of the latest dotnetpro article.
see also
Article, dotnetpro (GER)
Hopefully this will help you to solve this issue,
See ya
Leonhard
I am manually running tests from msbuild/tfsbuild by manually invoking mstest.exe but it is failing unexpectedly with error MSB3073 and ExitCode 1 when I am expecting 0.
I have this target that searches for all DLLs with a postfix of *UnitTests.DLL in the $(OutDir) folder. It builds up a commandline statement that is then executed:
<Target Name="RunUnitTests">
<CreateItem Include="$(OutDir)\*.UnitTests.dll"
AdditionalMetadata="TestContainerPrefix=/testcontainer:">
<Output TaskParameter="Include"
ItemName="UnitTestAssemblies" />
</CreateItem>
<Exec Timeout="120000"
Command=""$(VS110COMNTOOLS)..\IDE\mstest.exe" #(UnitTestAssemblies->'%(TestContainerPrefix)"%(FullPath)"',' ') /testsettings:"$(OutDir)..\..\Sources\mysettings.testsettings"" >
<Output TaskParameter="ExitCode" PropertyName="ExitCode"/>
</Exec>
<Error Condition=" '$(ExitCode)' != '0' And '$(ExitCode)' != '2'" Text="An error [$(ExitCode)] occurred running unit tests." />
<OnError ExecuteTargets="MarkBuildStepAsFailed" />
</Target>
I've added a Timeout property above because some googling suggested this but it didnt make a difference.
This gets equated in the buildlog file as below (quotes included) (the folders names I have changed but left spaces where relevant but they don't look too long):
"C:\Program Files (x86)\Microsoft Visual Studio 11.0\Common7\..\IDE\mstest.exe" /testcontainer:"C:\b\someprojectfolder\anotherfolder\Binaries\..\..\debug\some.unittests.dll" /testsettings:"C:\b\someprojectfolder\anotherfolder\Sources\..\..\mysettings.testsettings"
The tests DO run on the build server as part of the build process (i.e. calling the target above) as I can see the test results folder get created on disk. All unit tests pass as expected. I can see the MSTest.exe console output in my build log as well (e.g. Starting execution, list of tests and results, the results file is listed etc)
Additionally I can RDC onto the build server as the build service account and manually run the commandline above using a CMD and it works. (the test results (*.trx) and folder are there).
They also work when I manually invoke the commandline above using CMD them on my local developer machine as myself. It creates the test results file and folders.
FYI We are using Visual Studio 2012 Ultimate on my local machine and installed on the build server as well.
FYI We are using TFS 2012 with an upgrade process definition
I've got a feeling its to do with the "parsing"/escaping of quotes and/or apostrophes or could it be the use of using ..\ in the paths?
I have checked the event log on the build server and it displays no errors/information. Is there any other logs I can check? or properties I can define to "see" the actual error code?
NOTE: I know I could use the <RunTest> style msbuild/tfs build syntaxbut I manual trigger these tests at a more convenient time in the build process
this wasnt to with do quotes or apostrophes in paths. My .testsettings file connects to a remote test controller (on another server running Windows Server 2008 R2). I was collecting all data and diagnostics (video recorder etc, network emulation, event log, system diagnostics etc)
When I checked the event log on the test controller/agent server it was full of errors saying MSTest had to be run as administrative permissions. Thats what i am now investigating.