I have just recently started using Continuous Integration with VSTS and I have set up a pipeline which includes the "Visual Studio Test" Task.
Within the task, there's an option to execute a code coverage scan as part of the test.
To ensure that my code coverage only covers MY code and have a) created a .RunSettings file to include only the assemblies I generate and b) there are some parts of the code with the [ExcludeFromCodeCoverage] attribute.
Now, when execute Analyze Code Coverage from Visual Studio (2017 Enterprise, 15.7.4), everything works as I would expect, only my assembly gets analyzed and the code I have excluded is, errr, excluded.
However, when the VSTS pipeline is run, no such limitations are applied and ALL assemblies are tested and ALL code, including the specifically excluded code, which then results in a dramatic drop on the code-coverage %age.
Not sure where to go from here so hoping the S/O community can help.
Update - RunSettings file in use
<?xml version="1.0" encoding="utf-8"?>
<RunSettings>
<!-- Configurations for data collectors -->
<DataCollectionRunSettings>
<DataCollectors>
<DataCollector friendlyName="Code Coverage" uri="datacollector://Microsoft/CodeCoverage/2.0" assemblyQualifiedName="Microsoft.VisualStudio.Coverage.DynamicCoverageDataCollector, Microsoft.VisualStudio.TraceCollector, Version=11.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a">
<Configuration>
<CodeCoverage>
<ModulePaths>
<Include>
<ModulePath>.*\.dll$</ModulePath>
</Include>
<Exclude>
<ModulePath>.*Tests.dll$</ModulePath>
<ModulePath>.*moq.dll$</ModulePath>
</Exclude>
</ModulePaths>
<UseVerifiableInstrumentation>False</UseVerifiableInstrumentation>
</CodeCoverage>
</Configuration>
</DataCollector>
</DataCollectors>
</DataCollectionRunSettings>
There is an issue here: RunSettings file not used in TFS VsTest task, you can go and check the discussion about this issue.
As a workaround you can use below .RunSettings formats (add the UseVerifiableInstrumentation = False)
<?xml version="1.0" encoding="utf-8"?>
<RunSettings>
<DataCollectionRunSettings>
<DataCollectors>
<DataCollector friendlyName="Code Coverage" uri="datacollector://Microsoft/CodeCoverage/2.0" assemblyQualifiedName="Microsoft.VisualStudio.Coverage.DynamicCoverageDataCollector, Microsoft.VisualStudio.TraceCollector, Version=11.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a">
<Configuration>
<CodeCoverage>
<UseVerifiableInstrumentation>False</UseVerifiableInstrumentation>
<ModulePaths>
<Include>
<ModulePath>.*\\MyProjectName.*\.dll$</ModulePath>
</Include>
<Exclude>
<ModulePath>.*Tests.dll$</ModulePath>
</Exclude>
</ModulePaths>
</CodeCoverage>
</Configuration>
</DataCollector>
</DataCollectors>
</DataCollectionRunSettings>
</RunSettings>
Just check acesiddhu's explanation:
you are using visual studio test tools installer task in your
definition. in case that task is used useverifiable property needs to
be set to false because in xcopy mode we don't GAC the above dll
'Microsoft.VisualStudio.CodeCoverage.Shim (this is a dependency which
needs to be loaded when useverifiable is set to true)
marking this property false ensures it doesn't use this particular
dll.
Related
I am working on a a test solution through the CodedUITest framework on visual studio 2017 enterprise.
I am using a runsettings file as detailed here : https://learn.microsoft.com/fr-fr/visualstudio/test/configure-unit-tests-by-using-a-dot-runsettings-file?view=vs-2019#example-runsettings-file
I set the mediarecorder in order to get the video even when the test passed:
<DataCollector uri="datacollector://microsoft/VideoRecorder/1.0" assemblyQualifiedName="Microsoft.VisualStudio.TestTools.DataCollection.VideoRecorder.VideoRecorderDataCollector, Microsoft.VisualStudio.TestTools.DataCollection.VideoRecorder, Version=15.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a" friendlyName="Screen and Voice Recorder">
<!--Video data collector was introduced in Visual Studio 2017 version 15.5 -->
<Configuration>
<!-- Set "sendRecordedMediaForPassedTestCase" to "false" to add video attachments to failed tests only -->
<MediaRecorder sendRecordedMediaForPassedTestCase="true" xmlns="">
<ScreenCaptureVideo bitRate="512" frameRate="2" quality="20" />
</MediaRecorder>
</Configuration>
</DataCollector>
But I did not find where the video file is stored.
Any idea where I can find it? Is it possible to specify a folder path where we want to store the record?
Thx,
Check further up in the runsettings file: Recordings are stored underneath the ResultsDirectory
<RunSettings>
<RunConfiguration>
...
<!-- Path relative to directory that contains .runsettings file-->
<ResultsDirectory>.\TestResults</ResultsDirectory>
After some research, to get VisualStudio to build a project's object files in parallel, one must ensure that project properties MinimalRebuild is off and MultiProcessorCompilation is on. I can configure this in the project file, but is there a way to override these project settings on the command line using msbuild?
In other words, using msbuild from the command line, how would I override MinimalRebuild (/Gm-) to be false and MultiProcessorCompilation (/MP) to be true? The following doesn't seem to work
msbuild /m:7 /p:CL_MPCount=7 /p:MinimalRebuild=false /p:MultiProcessorCompilation=true
As an aside, is /p:CL_MPCOUNT redundant with the MultiProcessorCompilation property? Overall, I'm having trouble finding documentation on what fits the /p option.
In other words, using msbuild from the command line, how would I
override MinimalRebuild (/Gm-) to be false and
MultiProcessorCompilation (/MP) to be true?
CL_MPCount can be overrided by /p:xxx=xxx but MultiProcessorCompilation and MinimalRebuild are not. Just to be clear, It can only be overridden by /p:xxx=xxx(p means property) if it is a property of MSBuild. And the Property usually can be called by $. So I do some tests:
Edit
Test
1)CL_MPCount
This value means Maximum Number of concurrent C++ compilations in VS IDE.You can refer to this.
So I have assigned a value by VS IDE by(Debug->Options->Projects and Solutions->VC++ Project Solutions and set the Maximum Number of concurrent C++ compilations to 2.)
write a custom target like
<Target Name ="Test" AfterTargets="Build">
<Message Importance="high" Text="CL_MPCount is= $(CL_MPCount)">
</Message>
</Target>
Then Build and it shows;
From this we can see that this is a property that belongs to MSBuild.
2)MinimalRebuild
Usually, it means /GM and it usually set in the Properties in a c++ project. l have set it to True by (Right-click on the project-->Properties-->C/C++-->Code Generation-->set Enable Minimal Rebuild to True).
Custom target like
<Target Name ="Test" AfterTargets="Build">
<Message Importance="high" Text="MinimalRebuild is= $(MinimalRebuild)">
</Message>
</Target>
The result is like
From it, the value which set in the VS IDE cannot be available by $(MinimalRebuild) which means it is not a property of MSBuild.
3) MultiProcessorCompilation
It also means /MP and from the document we will find that it is an option of the the compiler not a property of MSBuild. Besides, l did the same as the MinimalRebuild and it is not being output. So it is also not a property of MSBuild.
In addition, l thought MinimalRebuild and MultiProcessorCompilation are the options of MSBuild before. So I type MSBuild -help in Command Line and found that it did not list these parameters at all. So I think these two parameters have nothing to do with MSBuild at all, but are just some of the compiler's options.
Hope it could help you.
I found a way to override both properties at User level. Anyway you may find them helpful (for example for Build host where incremental build is not meaningful at all).
Create these two properties files:
%USERPROFILE%\AppData\Local\Microsoft\MSBuild\v4.0\Microsoft.Cpp.Win32.user.props
%USERPROFILE%\AppData\Local\Microsoft\MSBuild\v4.0\Microsoft.Cpp.x64.user.props
with following content:
<?xml version="1.0" encoding="utf-8"?>
<Project DefaultTargets="Build" ToolsVersion="4.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
<ImportGroup Label="PropertySheets" />
<PropertyGroup Label="UserMacros" />
<PropertyGroup />
<ItemDefinitionGroup>
<ClCompile>
<MinimalRebuild>false</MinimalRebuild>
<MultiProcessorCompilation>true</MultiProcessorCompilation>
<ProcessorNumber>2</ProcessorNumber>
</ClCompile>
</ItemDefinitionGroup>
<ItemGroup />
</Project>
Feel free to customize ProcessorNumber to suit your hardware.
Now MSBUild will use above values for all projects.
In my C++ project Visual Studio (2017) fails to detect changes in a property value when I trigger a "build" (e.g. via F5 requesting a debugger start).
I want to be able to use the property pages dialog in visual studio to specify a path variable to another library.
The simplified versions are here. First I use a .targets file:
<?xml version="1.0" encoding="utf-8"?>
<Project ToolsVersion="4.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
<ItemGroup>
<PropertyPageSchema Include="$(MSBuildThisFileDirectory)\overrideSettings.xml" />
</ItemGroup>
<PropertyGroup Condition="'$(myDir)' == ''">
<__my_IncludeDir>somewhere\include\</__my_IncludeDir>
</PropertyGroup>
<PropertyGroup Condition="'$(myDir)' != ''">
<__my_IncludeDir>somewhere_else\include\</__my_IncludeDir>
</PropertyGroup>
<ItemDefinitionGroup>
<ClCompile>
<AdditionalIncludeDirectories>$(__my_IncludeDir);%(AdditionalIncludeDirectories)</AdditionalIncludeDirectories>
</ClCompile>
<ResourceCompile>
<AdditionalIncludeDirectories>$(__my_IncludeDir);%(AdditionalIncludeDirectories)</AdditionalIncludeDirectories>
</ResourceCompile>
</ItemDefinitionGroup>
</Project>
And an .xml file for the settings:
<?xml version="1.0" encoding="utf-8"?>
<ProjectSchemaDefinitions xmlns="clr-namespace:Microsoft.Build.Framework.XamlTypes;assembly=Microsoft.Build.Framework">
<Rule Name="ProjectSettings_NugetDevOverride" PageTemplate="tool" DisplayName="Nuget Development Override" SwitchPrefix="/" Order="1">
<Rule.Categories>
<Category Name="myTest" DisplayName="myTest" />
</Rule.Categories>
<Rule.DataSource>
<DataSource Persistence="UserFile" ItemType="" />
</Rule.DataSource>
<StringProperty Name="myDir" Category="myTest" />
</Rule>
</ProjectSchemaDefinitions>
Now, this does work to some extent.
However, Visual Studio fails to correctly detect changes in the property variable defined on the property page.
I build the project -> 1 succeeded
I build the project again (F5) -> 1 up-to-date
I change the variable myDir using the property page; making it empty or setting a value, so that the other property group in the .targets file is triggered.
I build the project again (F5) -> 1 up-to-date This is wrong!
I re-build the project -> 1 succeeded with correctly used new property value.
Where is the problem in my setup? Can I explicitly add myDir as property to be checked before the build is marked as up-to-date?
I found one workaround: DisableFastUpToDateCheck
https://stackoverflow.com/a/36004494/552373
But this is horrible, since my real project is very large and the fast-up-to-date-check really helps.
Update 20171128:
I now also tried the following:
I build the project -> 1 succeeded
I build the project again (F5) -> 1 up-to-date
I change the variable myDir using the property page; making it empty or setting a value, so that the other property group in the .targets file is triggered.
I close Visual Studio and open it again. No file-save dialog popped up.
I build the project again (F5) -> 1 up-to-date This is wrong!
The fact, that even this does not work, points at a real problem with the FastUpToDateCheck in Visual Studio.
Does anyone have a further idea?
Goal
I want to be able to run my unit tests remotely on another machine since they interact with the UI of another application. For clarity these tests are not Coded UI Tests, they are tests methods that use FlaUI to interact with the desktop.
Problem
I can't get the Visual Studio Test Controller and Test Agent to work with MSTest V2. When I set the .runsettings file to use the .testsettings file and to ForcedLegacyMode like the documentation says here I get the following warnings and no tests are loaded into the test explorer.
[11/22/2017 9:54:12 AM Warning] Index was outside the bounds of the array.
[11/22/2017 9:54:13 AM Warning] Index was outside the bounds of the array.
[11/22/2017 9:54:13 AM Warning] Index was outside the bounds of the array.
[11/22/2017 9:54:14 AM Warning] Warning : A testsettings file or a runsettings with a ForcedLegacyMode set to true is not supported with the MSTest V2 Adapter.
I am hoping I am just missing some setting I can put into my .runsettings file that will allow me to specify the url for my Test Controller.
Settings
Here are my .runsettings and .testsettings files for reference. These settings successfully connect to the machine but when I build my test runner no longer finds and tests to run.
.runSettings
<?xml version="1.0" encoding="utf-8"?>
<RunSettings>
<!-- Configurations that affect the Test Framework -->
<RunConfiguration>
<MaxCpuCount>1</MaxCpuCount>
<!-- Path relative to solution directory -->
<ResultsDirectory>.\TestResults</ResultsDirectory>
<!-- [x86] | x64
- You can also change it from menu Test, Test Settings, Default Processor Architecture -->
<TargetPlatform>x86</TargetPlatform>
<!-- Framework35 | [Framework40] | Framework45 -->
<TargetFrameworkVersion>Framework40</TargetFrameworkVersion>
</RunConfiguration>
<!-- Configurations for data collectors -->
<DataCollectionRunSettings>
<DataCollectors>
<DataCollector friendlyName="Code Coverage" uri="datacollector://Microsoft/CodeCoverage/2.0" assemblyQualifiedName="Microsoft.VisualStudio.Coverage.DynamicCoverageDataCollector, Microsoft.VisualStudio.TraceCollector, Version=11.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a">
<Configuration>
<CodeCoverage>
<ModulePaths>
<Exclude>
<ModulePath>.*CPPUnitTestFramework.*</ModulePath>
</Exclude>
</ModulePaths>
<!-- We recommend you do not change the following values: -->
<UseVerifiableInstrumentation>True</UseVerifiableInstrumentation>
<AllowLowIntegrityProcesses>True</AllowLowIntegrityProcesses>
<CollectFromChildProcesses>True</CollectFromChildProcesses>
<CollectAspDotNet>False</CollectAspDotNet>
</CodeCoverage>
</Configuration>
</DataCollector>
</DataCollectors>
</DataCollectionRunSettings>
<!-- Parameters used by tests at runtime -->
<TestRunParameters>
</TestRunParameters>
<!-- Adapter Specific sections -->
<!-- MSTest adapter -->
<MSTest>
<MapInconclusiveToFailed>True</MapInconclusiveToFailed>
<CaptureTraceOutput>false</CaptureTraceOutput>
<DeleteDeploymentDirectoryAfterTestRunIsComplete>False</DeleteDeploymentDirectoryAfterTestRunIsComplete>
<DeploymentEnabled>False</DeploymentEnabled>
</MSTest>
</RunSettings>
.testsettings
<?xml version="1.0" encoding="UTF-8"?>
<TestSettings name="rmoqa01" id="076be28c-d18b-46bf-ad20-4d43ec821ea4" xmlns="http://microsoft.com/schemas/VisualStudio/TeamTest/2010">
<Description>These are default test settings for a local test run.</Description>
<RemoteController name="10.2.0.101" />
<Execution location="Remote">
<Hosts skipUnhostableTests="false">
<VSSDKTestHostRunConfig name="VS IDE" HiveKind="DevEnv" HiveName="15.0_c9b36733" xmlns="http://microsoft.com/schemas/VisualStudio/SDK/Tools/IdeHostAdapter/2006/06" />
</Hosts>
<TestTypeSpecific>
<UnitTestRunConfig testTypeId="13cdc9d9-ddb5-4fa4-a97d-d965ccfc6d4b">
<AssemblyResolution>
<TestDirectory useLoadContext="true" />
</AssemblyResolution>
</UnitTestRunConfig>
</TestTypeSpecific>
<AgentRule name="AllAgentsDefaultRole">
</AgentRule>
</Execution>
<Properties />
</TestSettings>
I opened a issue on the mstest github page and after looking at the source code generating my warning I am seeing it looks like there is probably no work around to this. Here is the source code I was looking at in the MSTestDiscover.cs.
// Scenarios that include testsettings or forcing a run via the legacy adapter are currently not supported in MSTestAdapter.
if (MSTestSettings.IsLegacyScenario(logger))
{
return;
}
EDIT: 10/28/2018
The issue I linked to above was updated with the following response.
There are no plans to have remote execution capability through MSTVest2. You can use the test task in VSTS which supports distributed execution on multiple agents. https://learn.microsoft.com/en-us/vsts/pipelines/test/set-up-continuous-test-environments-builds?view=vsts
I am trying log some entries in a log file (Log.Trace / Log.Debug) while my VS unit test runs. I have also copied the file NLog.config to out directory through DeploymentItem attribute over class. Still my log file is not created. Any help as to how can we log entries in a file same as we do for normal web application.
I've just found a solution to this problem:
Add the App.config file to yout unit test project with the following contents:
<?xml version="1.0" encoding="utf-8" ?>
<configuration>
<configSections>
<section name="nlog" type="NLog.Config.ConfigSectionHandler, NLog"/>
</configSections>
<nlog xmlns="http://www.nlog-project.org/schemas/NLog.xsd"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<targets>
<target name="debugLog" xsi:type="Console" />
</targets>
<rules>
<logger name="*" minlevel="Debug" writeTo="debugLog"></logger>
</rules>
</nlog>
</configuration>
You may put any configuration you wish as with NLog.config file.
Unfortunately that is the only XML solution at the time. It is not only about Unit Testing, though. NLog.config does not work with just any Console Application.
Do not know if it is by design or just an oversight. Anyway I would not say that moving NLog.config to App.config section is any way satisfactory =/
Maybe it is worth to notice that there is a possibility of configuring nlog directly from code, what could be helpful in some scenarios.
One could be also glad to find nlog debug option, that will log the whole process of processing configuration file, registering targets and so on...
To turn it on, just add internalLogFile and internalLogLevel attributes to nlog element:
<nlog xmlns="http://www.nlog-project.org/schemas/NLog.xsd" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" internalLogFile="C:/logs/nlog.txt" internalLogLevel="Debug">
or from code:
InternalLogger.LogFile = #"c:\logs\nlog.txt";
InternalLogger.LogLevel = LogLevel.Trace;