I am manually running tests from msbuild/tfsbuild by manually invoking mstest.exe but it is failing unexpectedly with error MSB3073 and ExitCode 1 when I am expecting 0.
I have this target that searches for all DLLs with a postfix of *UnitTests.DLL in the $(OutDir) folder. It builds up a commandline statement that is then executed:
<Target Name="RunUnitTests">
<CreateItem Include="$(OutDir)\*.UnitTests.dll"
AdditionalMetadata="TestContainerPrefix=/testcontainer:">
<Output TaskParameter="Include"
ItemName="UnitTestAssemblies" />
</CreateItem>
<Exec Timeout="120000"
Command=""$(VS110COMNTOOLS)..\IDE\mstest.exe" #(UnitTestAssemblies->'%(TestContainerPrefix)"%(FullPath)"',' ') /testsettings:"$(OutDir)..\..\Sources\mysettings.testsettings"" >
<Output TaskParameter="ExitCode" PropertyName="ExitCode"/>
</Exec>
<Error Condition=" '$(ExitCode)' != '0' And '$(ExitCode)' != '2'" Text="An error [$(ExitCode)] occurred running unit tests." />
<OnError ExecuteTargets="MarkBuildStepAsFailed" />
</Target>
I've added a Timeout property above because some googling suggested this but it didnt make a difference.
This gets equated in the buildlog file as below (quotes included) (the folders names I have changed but left spaces where relevant but they don't look too long):
"C:\Program Files (x86)\Microsoft Visual Studio 11.0\Common7\..\IDE\mstest.exe" /testcontainer:"C:\b\someprojectfolder\anotherfolder\Binaries\..\..\debug\some.unittests.dll" /testsettings:"C:\b\someprojectfolder\anotherfolder\Sources\..\..\mysettings.testsettings"
The tests DO run on the build server as part of the build process (i.e. calling the target above) as I can see the test results folder get created on disk. All unit tests pass as expected. I can see the MSTest.exe console output in my build log as well (e.g. Starting execution, list of tests and results, the results file is listed etc)
Additionally I can RDC onto the build server as the build service account and manually run the commandline above using a CMD and it works. (the test results (*.trx) and folder are there).
They also work when I manually invoke the commandline above using CMD them on my local developer machine as myself. It creates the test results file and folders.
FYI We are using Visual Studio 2012 Ultimate on my local machine and installed on the build server as well.
FYI We are using TFS 2012 with an upgrade process definition
I've got a feeling its to do with the "parsing"/escaping of quotes and/or apostrophes or could it be the use of using ..\ in the paths?
I have checked the event log on the build server and it displays no errors/information. Is there any other logs I can check? or properties I can define to "see" the actual error code?
NOTE: I know I could use the <RunTest> style msbuild/tfs build syntaxbut I manual trigger these tests at a more convenient time in the build process
this wasnt to with do quotes or apostrophes in paths. My .testsettings file connects to a remote test controller (on another server running Windows Server 2008 R2). I was collecting all data and diagnostics (video recorder etc, network emulation, event log, system diagnostics etc)
When I checked the event log on the test controller/agent server it was full of errors saying MSTest had to be run as administrative permissions. Thats what i am now investigating.
Related
I have a pretty straight forward WiX project. Nothing fancy. When trying to perform a MajorUpgrade over an existing installation, it is unable to start the service and inevitably rolls back to the previous version and starts the service just fine. I have removed the Start="install" and manually started the application successfully, so I know it's not a dependency issue.
I have searched endlessly and found no answers to my problem.
<MajorUpgrade DowngradeErrorMessage="A newer version of [ProductName] is already installed." Schedule="afterInstallFinalize" />
My service install:
<ServiceInstall
Id="ServiceInstaller"
Type="ownProcess"
Name="LsdnService"
DisplayName="Lsdn Service"
Description="Placeholder for now."
Start="auto"
Account="[SERVICEACCOUNT]"
Password="[SERVICEPASSWORD]"
ErrorControl="normal"/>
<ServiceControl Id="StartService" Start="install" Stop="both" Remove="uninstall" Name="LsdnService" Wait="yes" />
I dumped the MSI log to a file and got this error but it is quite vague.
MSI (s) (18:48) [22:41:27:349]: Note: 1: 2205 2: 3: Error
MSI (s) (18:48) [22:41:27:349]: Note: 1: 2228 2: 3: Error 4: SELECT `Message` FROM `Error` WHERE `Error` = 1920
There are some registry modifications during an installation. The installer attempts to read from the registry and inherit the already existing values.
<Property Id="LSDNADDRESS" Value="127.0.0.1">
<RegistrySearch Id="LsdnAddressProperty" Root="HKLM" Key="$(var.RegistryKey)" Name="LsdnAddress" Type="raw" />
</Property>
<Property Id="LSDNPORT" Value="9920">
<RegistrySearch Id="LsdnPortProperty" Root="HKLM" Key="$(var.RegistryKey)" Name="LsdnPort" Type="raw" />
</Property>
<Property Id="LSDNKEY" Value="6f380b07-0b54-4904-8303-95d1ec45d453">
<RegistrySearch Id="LsdnKeyProperty" Root="HKLM" Key="$(var.RegistryKey)" Name="LsdnKey" Type="raw" />
</Property>
Debugging Results: Following a lot of debugging (by original poster - OP) this turned out to be a known MSI issue described here:
https://wix-users.narkive.com/EMfQPDrM/a-bug-get-reg-sz-when-using-type-integer. Nice search work.
What is in a DWORD? (a REG_SZ apparently): Essentially MSI "converts" a DWORD value found via a RegistrySearch
operation to a formatted string - REG_SZ - during upgrade
installations (could be more involved too). This causes services that
expect a DWORD value to fall over on startup during major
upgrades. A very exotic error.
Workaround: One can try to "solve" this problem by making the service code capable of reading both DWORD and REG_SZ.
This yields a more robust solution than solving the problem in a
custom action since it is a "permanent" fix as long as the code is in
there (and the presence of the code alerts other developers about the
problem). Or maybe use only REG_SZ?
Quick Checks: Check the service password and login - obviously. Anything in the
Event Viewer? Windows Key + Tap R + eventvwr.msc + Enter. How to use the Event Viewer to troubleshoot problems with a Windows Service. Perhaps you can try to do a folder diff on the before and after folders and see if you see something unexpected in
the config files? Naturally there will be lots of binary
differences, but check the text files (also encoding). Check the MSI log file
again and search for "value 3" as described here: Tips For Checking MSI Log
Files. Manually copy the new files in place and attempt to start the service via the services.msc applet.
Service Experts: Windows Services Frequently Asked Questions (FAQ). Content seems to be up to date - at face value at least.
These guys claim to be experts on services. I have no idea who they
are.
Look in the "Errors" section in the link above. Here are some
extracts:
1053: The service did not respond to the start or control request in a timely fashion
Why doesn't my Windows Service start automatically after a reboot?
1069: The service did not start due to a logon failure
Generic Check Lists: If none of the above does anything, maybe try these "torpedoes full spread" check-lists (just ideas to start debugging):
Desktop applicaton not opening after installation in client system
Windows Application Startup Error Exception code: 0xe0434352
General Purpose Debugging: Throwing in some general-purpose debugging approaches.
Custom Action Debugging: WIxsharp debug custom action in console
Dependency Scanning: Which winform project files should be packed up into the installer
Some Further Links:
C# Debug folder when copied to another location does not run the exe
wix service install not enough permission
How exactly does the WiX 'Service Install' work internally?
WiX Toolset PermissionEx Problem - App Does Not Run After Installation
It certainly could be a dependency issue. For example, GAC / WinSXS files don't get installed into the GAC until the commit phase which is after StartServices.
I would leave the Start="Install" in and while it's sitting at the failed to start prompt inspect the state of the machine and debug the service start manually. I bet you'll find something missing.
I added some prebuild script to a Visual Studio 2017 project.
powershell.exe -ExecutionPolicy ByPass -NoProfile -NonInteractive -File "./myscript.ps1"
It's just a script outputting some information in console.
The script executes correctly, whether it is launched directly in my Powershell console or from the build event (I can see the correct output in the Build Output panel in VS).
After the execution, the build fails with one error :
The "Exec" task was not given a value for the required parameter "Command"
I tried to reduce the problem to a minimal "./myscript.ps1" to show you my problem, and the problem occurs with any script, even an empty one!
And again, whatever the PS script is, it gives its output correctly.
Why does my build fail, and what can I do to fix it, while still running the script before build ?
The issue was caused by this entry in the .csproj project file :
<Target Name="PostBuild" AfterTargets="PostBuildEvent">
<Exec Command="" />
</Target>
Apparently, I (or VS ;) ) mistakenly added an empty script for PostBuild at the same time.
Removing the quoted entry solved the problem.
Thanks to Hans Passant's comment for pointing to the right direction.
In short, using a .runsetting unit test file in a build step on TFS 2015 results in a 503 Service Unavailable exception.
After extensive searching and testing on a new TFS 2015 installation (as also described in TFS2015 new install, 503 Service Unavailable), I may have found the problem.
We are using .runsettings files as described on https://msdn.microsoft.com/en-us/library/jj159530.aspx, in a 'Visual Studio Test' build step on our new TFS2015 installation, as soon as the VSTest.console.exe is called, the next three application pools on the TFS server crash, resulting in a 503 Service Unavailable error:
DefaultAppPool
Microsoft Team Foundation Server Application Pool
Microsoft Team Foundation SErver Message Queueu Application Pool
First my .runsetting file was kind of large, but i did try the next one (I think as small as possible):
<?xml version="1.0" encoding="utf-8"?>
<RunSettings>
<!-- Configurations for data collectors -->
<DataCollectionRunSettings>
<DataCollectors>
<DataCollector friendlyName="Code Coverage" uri="datacollector://Microsoft/CodeCoverage/2.0" assemblyQualifiedName="Microsoft.VisualStudio.Coverage.DynamicCoverageDataCollector, Microsoft.VisualStudio.TraceCollector, Version=14.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a">
<Configuration>
<CodeCoverage>
<!-- Match assembly file paths: -->
<ModulePaths>
<Include>
<ModulePath>.*\.dll$</ModulePath>
<ModulePath>.*\.exe$</ModulePath>
</Include>
<Exclude>
<ModulePath>.*CPPUnitTestFramework.*</ModulePath>
<ModulePath>.*fluentassertions.*</ModulePath>
<ModulePath>.*\.test\.dll$</ModulePath>
</Exclude>
</ModulePaths>
</CodeCoverage>
</Configuration>
</DataCollector>
</DataCollectors>
</DataCollectionRunSettings>
</RunSettings>
When using this one, the app pools crashes. Even if i remove the part entirely, it will not work.
As soon as the next line is called (taken from the log lines of the build step), the app pools crashes.
Executing C:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe "D:\TfsAgents\Agent1\_work\2\s\[...]\Release\Microsoft.QualityTools.Testing.Fakes.dll" "D:\TfsAgents\Agent1\_work\2\s\[...]\Release\Microsoft.VisualStudio.QualityTools.UnitTestFramework.dll" "D:\TfsAgents\Agent1\_work\2\s\[...]\Release\MyClassLib.dll" /Settings:"D:\TfsAgents\Agent1\_work\2\s\[...]\test.runsettings" /EnableCodeCoverage /logger:trx
If you look at the DataCollector tag in the .runsettings file, you see the version is set to 14.0.0.0. If you look at the page of Miscrosoft, they state that it should be 11.0.0.0, but also this version does not work. I thought it might be that version 11.0.0.0 was wrong, so I opened the dll with Telerik JustDecompile, and saw that the 'real' version was 14.0.0.0, so I put that version into the runsetting-file. But no luck.
So for now I'm just remove the runsetting file from the definition of the build, which unfortunately results in a wrong code coverate percentage.. So hopefully someone might have the answer for this.
This problem is limited to the specific runsettings file when used with build agent running on the same machine where IIS server is hosting TFS. There is a workaround available. Within DataCollectors->Configuration->CodeCoverage tag in runsettings file you can add following xml tags with given values :
<UseVerifiableInstrumentation>True</UseVerifiableInstrumentation>
<AllowLowIntegrityProcesses>True</AllowLowIntegrityProcesses>
<CollectFromChildProcesses>True</CollectFromChildProcesses>
<CollectAspDotNet>false</CollectAspDotNet>
This should solve the issue, and also its recommended that in future if you are using runsettings file with Code coverage in Data Collector then these tags should be added.
I am running tests through Jenkins on a windows box. In my "Execute Windows Batch command" portion of the project configuration I have the following command:
nosetests --nocapture --with-xunitmp --eval-attr "%APPLICATION% and priority<=%PRIORITY% and smoketest and not dev" --processes=4 --process-timeout=2000
The post build actions have "Publish JUnit test result report" with the Test report XMLs path being:
trunk\automation\selenium\src\nosetests.xml
When I do a test run, the nosetests.xml file is created, however it is empty, and I am not getting any Test Results for the build.
I am not really sure what is wrong here.
EDIT 1
I ran the tests with just --with-xunit and REM'd out the --processes and got test results. Does anyone of problems with xunitmp not working with a Windows environment?
EDIT 2
I unstalled an reinstalled nose and nose_xunitmp to no avail.
The nosetest plugin for parallelizing tests and plugin for producing xml output are incompatible. Enabling them at the same time will produce the exact result you got.
If you want to keep using nosetest, you need to execute tests sequentially or find other means of parallelizing them (e.g. by executing multiple parallel nosetest commands (which is what I do at work.))
Alternatively you can use another test runner like nose2 or py.test which do not have this limitation.
Apparently the problem is indeed Windows and how it handles threads. We attempted several tests outside of our Windows Jenkins server and they do not work either. Stupid Windows.
i set up mstest on target machine as standalone program without installing visual studio. So i copied the files(*.exe's,dll's) mstest needed to the target machine and register all dlls in gac which mstest needs to run unit tests.
Works fine so far for normal unit tests.
Acually i try to setup mstest to run coded ui test. I copied some new dll's to the target machine:
- Microsoft.VisualStudio.QualityTools.CodedUITestFramework.dll
- Microsoft.VisualStudio.TestTools.UITest.Common.dll,TestTools.UITest.Extensions.dll
- Microsoft.VisualStudio.TestTools.UITesting.dll
- Microsoft.VisualStudio.HostingProcess.Utilities.Sync
But when i try to execute a test on cmd.exe via mstest.exe on my target machine, the test will run but always fail (works fine on my local machine, Visual Studio 2012). Its a pretty simple test:
- just click on "Start",
- type in "calc press enter"
Both machines use the same operating system.
I guess MSTest.exe still needs some dll's to perform codedUiTests. The bad thing is the missing dll's arent show up in cmd.exe, no errors, no hints just nothing :/
The test runs like normal but will fail all tests.
cmd-output:
Microsoft (R) Test Execution Command Line Tool Version 11.0.50727.1
Copyright (c) Microsoft Corporation. All rights reserved.
Loading D:\Users\Jenkins\Desktop\CodedUITestProject1\CodedUITestProject1\bin\Debug\CodedUITestProject1.dll...
Starting execution...
Results Top Level Tests
------- ---------------
Failed CodedUITestProject1.CodedUITest1.CodedUITestMethod1
0/1 test(s) Passed, 1 Failed
Summary
-------
Test Run Failed.
Failed 1
---------
Total 1
Results file: C:\VS2011Stub\Common7\IDE\TestResults\Jenkins_MSGP166C 2013-05-24 12_36_28.trx
Test Settings: Default Test Settings
Does anyone know what files mstest.exe needs to run codeduitests in a correct way?
Edit: I checked the *.trx file and copied the missing dll's to my target machine.
I tried to run the test again, it failed again. I was checking the trx file again and got an awesome message :/
<Results>
<UnitTestResult executionId="4652eeb1-e1b4-4782-a288-dbd4bb0bda5a" testId="484ddbfe-fdc6-0f5d-9e7b-bab4da5b5905" testName="CodedUITestMethod1" computerName="MSGP166C" duration="00:00:00.0887388" startTime="2013-05-24T16:30:24.1716290+02:00" endTime="2013-05-24T16:30:24.6266745+02:00" testType="13cdc9d9-ddb5-4fa4-a97d-d965ccfc6d4b" outcome="Failed" testListId="8c84fa94-04c1-424b-9868-57a2d4851a1d" relativeResultsDirectory="4652eeb1-e1b4-4782-a288-dbd4bb0bda5a">
<Output>
<ErrorInfo>
<Message>Error calling Initialization method for test class CodedUITestProject1.CodedUITest1: System.IO.FileNotFoundException: Das System kann die angegebene Datei nicht finden. (Ausnahme von HRESULT: 0x80070002)</Message>
<StackTrace> bei System.Reflection.RuntimeAssembly.nLoadFile(String path, Evidence evidence)
bei System.Reflection.Assembly.LoadFile(String path)
bei Microsoft.VisualStudio.TestTools.UITest.Framework.UITestExtensionPackageManager.LoadAssembly(String assemblyFile)
</StackTrace>
</ErrorInfo>
</Output>
</UnitTestResult>
</Results>
So what Assembly is missing now?? It was not mentioned.
Thanks in Advance
Instead of trying to copy over the dlls I would install VS Test Agents. It's much lighter than visual studio and will get you mstest. This is the approach I use when running CodedUI tests on a test machine.
http://search.microsoft.com/en-us/DownloadResults.aspx?q=test+agents
Download the version of test agents that you built the project in.
(doesn't really matter)
There will be 3 options when you open the installer. You want test agents.
You should now have everything you need for mstest and codedui
Another that works well is putting all of your codedui tests into an ordered test and passing that into mstest.
MSTest /TestContainer:OrderedTest1.orderedtest
should do the trick