I want to exclude some tests from my continuous integration build but I haven't found a way to do so.
One of the things I've tried was to set up the priority of those tests to -2 and then on the build I specified Minimum Test Priority = -1 but it still run those tests.
Any help would be greatly appreciated.
Instead of using "Test Lists" that have been described, you should use the "Test Category" method. The test lists & VSMDI functionality have actually been deprecated in Visual Studio 2010 and Microsoft may remove the feature completely in a future version of Visual Studio.
If you'd like some more information about how to use test categories especially with your automated build process, check out this blog post: http://www.edsquared.com/2009/09/25/Test+Categories+And+Running+A+Subset+Of+Tests+In+Team+Foundation+Server+2010.aspx
You can also exclude test categories from running by specifying the ! (exclamation point) character in front of the category name to further define your filter.
If you are using MSTest you can create a Test List for the tests that you need in you continuous integration.
With MSTest, you can simply create two test projects (assemblies) and only specify one in the build config to use for testing. In MSBuild, this was the way to go. For the new WF-Based build definitions, I currently don't have a sample at hand:
<ItemGroup>
<!-- TEST ARGUMENTS
If the RunTest property is set to true then the following test arguments will be used to run
tests. Tests can be run by specifying one or more test lists and/or one or more test containers.
To run tests using test lists, add MetaDataFile items and associated TestLists here. Paths can
be server paths or local paths, but server paths relative to the location of this file are highly
recommended:
<MetaDataFile Include="$(BuildProjectFolderPath)/HelloWorld/HelloWorld.vsmdi">
<TestList>BVT1;BVT2</TestList>
</MetaDataFile>
To run tests using test containers, add TestContainer items here:
<TestContainer Include="$(OutDir)\AutomatedBuildTests.dll" />
<TestContainer Include="$(SolutionRoot)\TestProject\WebTest1.webtest" />
<TestContainer Include="$(SolutionRoot)\TestProject\LoadTest1.loadtest" />
Use %2a instead of * and %3f instead of ? to prevent expansion before test assemblies are built
-->
</ItemGroup>
<PropertyGroup>
<RunConfigFile>$(SolutionRoot)\LocalTestRun.testrunconfig</RunConfigFile>
</PropertyGroup>
Tip: To use a generic build definition, we name all our Test projects "AutomatedBuildTests", i.e. there is no solution difference. So the build definition can be included in any existing build definition (or even be a common one) that always executes the right set of tests. It would be an easy task to prepend an "if exists" check in order to allow a build definition to only run tests when a Test assembly is present. We do not use this in order to get build errors when no test assembly is found as we absolutely want test with all those builds that use this definition.
My preference would be as above using a Test List, but some people have issued merging/editing the vsmdi files... We end up with separate solutions and use a pattern match to execute all tests in the appropriate DLL.
In Visual Studio 2012 and later you can configure your build definition using the Test case filter setting.
This setting is part of your build definition.
Open the build definition and navigate to the Process tab. In the section 3. Test you can define mutiple test sources. For each test source your can specify a Test case filter.
You can find the details in this MSDN article: Running selective unit tests in VS 2012 RC using TestCaseFilter
I have copied the supported operators and some examples from this article:
Operators supported in RC are:
1.= (equals)
2.!= (not equals)
3.~ (contains or substring only for string values)
4.& (and)
5.| (or)
6.( ) (paranthesis for grouping)
Expresssion can be created using these operators as any valid logical condition. & (and) has higher
precedence over | (or) while evaluating expression.
E.g.
"TestCategory=NAR|Priority=1"
"Owner=vikram&TestCategory!=UI"
"FullyQualifiedName~NameSpace.Class"
"(TestCategory!=UI&(Priority=1|Priority=2))|(TestCategory=UI&Priority=1)"
Another possibility would be to have some test sources in one build definition in some (i.e. more or fewer) test sources in other build definitions.
Related
Here is my test assembly patterns (configuration)
**\$(BuildConfiguration)\*test*.dll
!**\obj\**
!**\$(BuildConfiguration)\*Integration*
After triggering build, here is the log where integration test assembly is also there (this file must be filtered and should be here)
2019-04-23T13:10:33.6689787Z C:\VSTSAgent\A1\_work\1\s\myapp\myapp.Services.Test\bin\Release\myapp.Services.Test.dll
2019-04-23T13:10:33.6690018Z C:\VSTSAgent\A1\_work\1\s\myapp\myapp.Services.Integration.Test\bin\Release\myapp.Services.Integration.Test.dll
Becuase of this integration test cases are also running and I want to run only unit test cases.
Any idea?
I've found the solution, here is my latest configuration for the same which working absolutely as expected now.
**\$(BuildConfiguration)\*test*.dll
!**\obj\**
!**\myapp\*Integration*\**
!**\*Microsoft.Owin.Testing.dll*
!**\$(BuildConfiguration)\*Integration.Test*.dll
!**\$(BuildConfiguration)\*Microsoft.VisualStudio.TestPlatform*
!**\$(BuildConfiguration)\*MSTest*
!**\$(BuildConfiguration)\*Microsoft.Owin.Testing.dll*
!**\$(BuildConfiguration)\*Microsoft.VisualStudio.QualityTools.UnitTestFramework.dll*
If you notice the line which says that exclude path which contains this pattern;
!**\myapp\*Integration*\**
and below pattern matches and will not be included in the result.
2019-04-23T13:10:33.6690018Z C:\VSTSAgent\A1\_work\1\s\myapp\myapp.Services.Integration.Test\bin\Release\myapp.Services.Integration.Test.dll
I have ancestral project and my Nightly build fails. I can`t find out how to fix it. Problem is about test cases sequence dependent execution.
Environment:
There is used TFS2015 build definitions. Basically, I have a definition with a Visual Studio Build task and then a Visual Studio Test task. Visual Studio Test task is also overwritten with Powershell file and I see that in TFS Nightly build process my predefined VS Ordered Test statements is ignored.
Important
Test cases are sequence dependent (as I said, this project is ancestral).
Problem
Interesting is that build log files always shows test execution in sequence what I define in VS Ordered Test, but at TFS2015 Detailed report Test results sequence always is different. So I can`t find out what affects test case execution procedures in TFS. Also I am not sure how tests are executed - parallel or sequentially (As I see, both TFS and PowerShell has no indications to run test cases in Parallel).
I have 2 questions:
Powershell brake down all in VS defined conditions?
Which is the best way to define test execution order, so that it takes into account?
Actually, those Test method run in the order that you defined in Ordered Test file during TFS build process. The build log already shows the correct sequence.
Just like you mentioned above, in the test result page, the order is the same as what you defined in the Ordered Test, but you can see that in front of each test method, it has an order number. You could download the test result file to check again and you will find those test methods are run in the correct order.
In the higher version, like TFS 2017, you could click the Column title 'Test' to make it sort by order().
You could also add a Date started column to know which test method run the first.
We have a few thousand native and .NET unit tests. In Visual Studio 2012, I can run and see the results, grouped by the C++/C# project.
I'd like to get something like this view, preferably grouped by solution (product) and then project (.dll), to the business people. At the bare minimum I'd like to at least have number of tests run and failed per solution.
Is there any proper way to do this with TFS?
I've looked everywhere and keep running into walls,
TFS build test results don't seem to store any information about the test categories, so I can't use those to group by solution
.vsmdi lists and .testsettings files have been phased out in VS 2012 and TFS 2012. We had separate lists for each solution before...now it's just *test*.dll
Test Plans and Custom SSRS reports seem to be completely useless for this much granularity of test results (why?). TfsTestWarehouse has barely anything - just enough for total tests passed/failed per build.
Parsing TRX files and writing HTML reports seems to work best using tools like trx2html, but I still can't run tests by solution.
TRX files are just XMLs, there's no need to parse them. You can write an XSLT transformation to present the data in the format you need. A nice thing about XSLT is that it has built-in aggregation, grouping, sorting etc capabilities.
In case TRX files themselves do not contain solution information (which is likely), then you'll have to do a two-stage report generation: prepare the data, generate the report.
The preparation would be a relatively simple command line tool, which would go over your sln files and build a map of which projects belong to while solutions (search the web, i bet there're already a bunch of scripts for that).
And the generation part would be using that mapping as an argument to the transformation and report generation to properly aggregate the data.
I know, it's a bit of a generic response, but hope it helps at least a bit.
I ended up solving this by adding the Project and Solution information in a custom Assembly Attribute (i.e. to the test .dll) at build time, through a custom MSBuild task. Here are roughly the steps I followed (from memory).
First, I created the custom Attribute:
[AttributeUsage(AttributeTargets.Assembly)]
public class ProjectAttribute: Attribute {
public string Project { get; set; }
public string Solution { get; set; }
public ProjectAttribute(string project, string solution)
{
this.Project = project;
this.Solution = solution;
}
}
This custom attribute was defined in an Assembly that was referenced by all unit test projects.
I then created a very simple/rudimentary inline MSBuild task, CreateProjectAttribCs that would dynamically create an extra C# file with one line. Something like:
[assembly: ProjectAttribute(Project="$(ProjectName)") Solution="$(Solution)"]
And I then added this file to the <Compile> Item Group, inside a custom MSBuild target called before Compile (again, just going from memory):
<Target Name="CreateProjectAttribCs" BeforeTargets="Compile">
<CreateProjectAttribCs File="ProjectAttribute.cs" />
<ItemGroup>
<Compile Include="ProjectAttribute.cs" />
</ItemGroup>
</Target>
<Target Name="CleanupProjectAttribCs" AfterTargets="Compile>
<Delete Files="ProjectAttribute.cs" />
</Target>
For C++ projects I'd added the Project and Solution info to a String Table resource "injected" in a similar way to the ProjectAttrib.cs file.
One major annoyance with all of this was that developers would have to add this custom MSBuild .targets file (which would contain the custom targets and the assembly reference) by editing the .csproj or .vcxproj.
To circumvent that a bit, I also created a custom Visual Studio Project Template for our team's unit tests so that everything was already added and my fellow devs would never have to see the innards of an MSBuild project.
The hardest part was adding the Project/Solution info. Once I had that it was easy to read the custom attributes on the test assemblies or String Table resource in a native .dll, and add the info to data parsed/transformed from the test results to a custom test result database and report.
in gtets as we know, the moment control finds TEST or TEST_F function, it registers the test cases into gtest. But according to my requirement, after gtest registers all the testcases
I need to search whether the testcasename is there in the list or not?
If the tesetcasename is there then I need to unregister all the test cases and register only
the found testcasename.
How to do that???
Suppose
TEST_F(testcasename, testname){}
TEST_F(testcasename1, testname1){}
TEST_F(testcasename3, testname3){}
..
..
TEST_F(testcasenameN, testnameN){}
Suppose I am searching for "testcasename3" in the registered testcasename. and it's available.
Now I want gtest to execute only the found testcase not all...
How to do that?
Any answer is appreciated
This can be done using the command line as described in the advanced guide, so
./foo_test --gtest_filter=testcasename3.*
would only run testcasename3 and all its tests. The commandline syntax is extensive, tests and be included and excluded using wildcards. See the advanced documentation for more information
I've written several XMLUnit tests (that fit in to the JUnit framework) in groovy and can execute them easily on the command line as per the groovy doco but I don't quite understand what else I've got to do for it to produce the xml output that is needed by Jenkins/Hudson (or other) to display the pass/fail results (like this) and detailed report of the errors etc (like this). (apologies to image owners)
Currently, my kickoff script is this:
def allSuite = new TestSuite('The XSL Tests')
//looking in package xsltests.rail.*
allSuite.addTest(AllTestSuite.suite("xsltests/rail", "*Tests.groovy"))
junit.textui.TestRunner.run(allSuite)
and this produces something like this:
Running all XSL Tests...
....
Time: 4.141
OK (4 tests)
How can I make this create a JUnit test report xml file suitable to be read by Jenkins/Hudson?
Do I need to kick off the tests with a different JUnit runner?
I have seen this answer but would like to avoid having to write my own test report output.
After a little hackage I have taken Eric Wendelin's suggestion and gone with Gradle.
To do this I have moved my groovy unit tests into the requisite directory structure src/test/groovy/, with the supporting resources (input and expected output XML files) going into the /src/test/resources/ directory.
All required libraries have been configured in the build.gradle file, as described (in its entirety) here:
apply plugin: 'groovy'
repositories {
mavenCentral()
}
dependencies {
testCompile group: 'junit', name: 'junit', version: '4.+'
groovy module('org.codehaus.groovy:groovy:1.8.2') {
dependency('asm:asm:3.3.1')
dependency('antlr:antlr:2.7.7')
dependency('xmlunit:xmlunit:1.3')
dependency('xalan:serializer:2.7.1')
dependency('xalan:xalan:2.7.1')
dependency('org.bluestemsoftware.open.maven.tparty:xerces-impl:2.9.0')
dependency('xml-apis:xml-apis:2.0.2')
}
}
test {
jvmArgs '-Xms64m', '-Xmx512m', '-XX:MaxPermSize=128m'
testLogging.showStandardStreams = true //not sure about this one, was in official user guide
outputs.upToDateWhen { false } //makes it run every time even when Gradle thinks it is "Up-To-Date"
}
This applies the Groovy plugin, sets up to use maven to grab the specified dependencies and then adds some extra values to the built-in "test" task.
One extra thing in there is the last line which makes Gradle run all of my tests every time and not just the ones it thinks are new/changed, this makes Jenkins play nicely.
I also created a gradle.properties file to get through the corporate proxy/firewall etc:
systemProp.http.proxyHost=10.xxx.xxx.xxx
systemProp.http.proxyPort=8080
systemProp.http.proxyUser=username
systemProp.http.proxyPassword=passwd
With this, I've created a 'free-style' project in Jenkins that polls our Mercurial repo periodically and whenever anyone commits an updated XSL to the repo all the tests will be run.
One of my original goals was being able to produce the standard Jenkins/Hudson pass/fail graphics and the JUnit reports, which is a success: Pass/Fail with JUnit Reports.
I hope this helps someone else with similar requirements.
I find the fastest way to bootstrap this stuff is with Gradle:
# build.gradle
apply plugin: 'groovy'
task initProjectStructure () << {
project.sourceSets.all*.allSource.sourceTrees.srcDirs.flatten().each { dir ->
dir.mkdirs()
}
}
Then run gradle initProjectStructure and move your source into src/main/groovy and tests to test/main/groovy.
It seems like a lot (really it's <5 minutes of work), but you get lots of stuff for free. Now you can run gradle test and it'll run your tests and produce JUnit XML you can use in build/test-reports in your project directory.
Since you're asking for the purposes of exposing the report to Jenkins/Hudson, I'm assuming you have a Maven/Ant/etc build that you're able to run. If that's true, the solution is simple.
First of all, there's practically no difference between Groovy and Java JUnit tests. So, all you need to do is add the Ant/Maven junit task/plugin to your build and have it execute your Groovy junit tests (just as you'd do if they were written in Java). That execution will create test reports. From there, you can simply configure your Hudson/Jenkins build to look at the directory where the test reports get created during the build process.
You can write your own custom RunListener (or SuiteRunListener). It still requires you to write some code, but it's much cleaner than the script you've provided a link to. If you'd like, I can send you the code for a JUnit reporter I've written in JavaScript for Jasmine and you can 'translate' it into Groovy.