My project is composed of several DLLs that I must test with different unit tests. I generate an XML file for each of the tests then I transform them into a JUnit format to be read by Jenkins. It works very well.
On the other hand, the test results are all in the same topic on the Jenkins interface and I would like to separate them for each DLL. It is for a more practical aspect for the visualization of the tests. I haven't found the solution yet. That's why I'm asking you if there is a solution for this problem.
I tried several plugins like JUnit or Warning NG. But the result remains the same. The JUnit plugin puts all the results in the same section and makes no distinction and the Warning NG plugin fails to parse the XML report to display it in Jenkins.
Related
I have a TFS 2015 build which builds one of our applications (it's an ASP.NET Web API application). As part of the build process it runs our unit tests.
I have a Visual Studio Test build step which runs these unit tests and they all pass okay.
I then run dotCover from this same build to determine code coverage (no we don't use the built-in code coverage as we don't have enterprise licences). However, when run from dotCover all the same unit tests fail.
I use a script step to run a batch file which invokes dotCover as follows.
E:\JetBrains\Installations\dotCover05\dotCover.exe analyse coverage.xml /LogFile=dotcover.log
The dotCover log file doesn't seem to give any indications as to why the unit tests have failed.
Any ideas why the unit tests pass when run from the Visual Studio Test build step and then fail when run from dotCover?
It seems that the problem is related to the fact that my build uses an XML file to hold certain data values. This XML file is found by VSTest when run under TFS but not by dotCover.
When dotCover runs it creates a TestResults folder which it then copies all the necessary files to required to run the unit tests. All files are copied except the XML file. I have set the file to "Copy always" so can't understand why this file isn't copied. I tried copying the file manually as a batch file but the folder structure is created by dotCover so it doesn't exist until I actually run the code coverage.
The solution is to decorate my test classes with the DeploymentItem() attribute.
[TestClass]
[DeploymentItem("File.xml")]
This has resolved my problem.
Make sure your build service account have enough permission to run the dotcove.exe. According to your E:\JetBrains\Installations\dotCover05\dotCover.exe Seems you didn't install for all users on the build agent. Which should installed under %ProgramFiles(x86)% not %LOCALAPPDATA%\JetBrains\Installations.
Try to use CoreInstructionSet parameter in your dotCover as a workaround for your situation. Details see below picture.
After doing this try to run the build again.
You can also try this method if it is failed by the file copying: Shadow-copying in dotCover: if your NUnit tests fail during continuous testing .
In my solution I have c++ and c# projects with corresponding Unit test projects mixed. During a TFS build I only want to execute the C# unit tests. Unfortunately I can't find a way to exclude these test assemblies to be being used in the unit test runner. I could identify all c++ unit tests projects based on a naming patter like Native.Tests.dll.
I can't find a way to explicitly exclude certain test assemblies from being tested/executed from the test runner.
Is there a way through either
Test Case Filter
Test assembly file specification?
Environment:
TFS 2013
Process template: ReleaseTfvcTemplate.12.xaml
In case you are wondering: Why do I want to exclude certain assemblies?
I want to use the Test Category feature to exclude certain unit tests from being executed on the build server, which you do through the TestCaseFilter feature in the TFS template. You specify that per "Test batch". When I run all my unit tests (c# + c++) in one test settings the native unit tests throw an error, because they don't understand/support the TestCategory-filter feature (remember: Test Case Filter). Therefore I want to split the it in 2 test runs/batches: c++ and c#.
Test Category is only worked in C# test project. (Test in my environment: TFS2015 VS2015)
If you only want to execute the C# unit test. The simplest way you can specify the C# test assembly file in Test assembly file specification.
You just need to identify all C# unit tests projects based on a naming patter totally different as C++ tests projects.
Note: You must make sure you are using totally different naming rules between C++ an C# test project.
For example:
C++test project name : including ABC like ABC1 ,ABC2, ABC3
C# test project name: including XYZ like XYZ1, XYZ2 ,XYZ3
Then using ***XYZ*.dll in Test assembly file specification to run the all C# test project.
I have project that is being built using NAnt.
Several projects are written in Delphi, several - C++, others - C#. They have unit tests. NAnt is smart enough to execute these unit tests.
As a result I have folder TestsResults containing one XML files (NUnit format) per one project with unit tests.
How can I insert tests results from these XML files into sonar DB? I tried to use Maven with sonar to do the trick but still no luck.
Have you read the Sonar .NET plugin details here? The Delphi, C++ and C# are also documented.
I have a maven build process that publishes executable jars and their tests to Nexus.
I have another maven build process that needs to access these jars (executable + test) and run the tests.
How do I go about it? So far I have managed to do this only if the jar is exploded to class files.
I am new to maven and completely lost in the documentation.
Update 2022-03-11
The feature has been implemented, see https://stackoverflow.com/a/17061755/1589700 for details
Original answer
Surefire and failsafe do not currently support running tests from within a jar.
This is largely a case of not being able to identify the tests.
There are two ways to get the tests to run.
Use a test suite that lists all the tests from the test-jar. Because the test suite will be in src/test/java (more correctly will be compiled into target/test-classes) that will be picked up and all the tests in the suite will be run by Surefire/failsafe (assuming the suite class name matches the includes rule: starts or ends with Test)
Use the maven dependency plugin's unpack-dependencies goal to unpack the test-jar into target/test-classes (this screams of hack, but works quite well)
The main issue with the first option is that you cannot easily run just one test from the suite, and you need to name every test from the test-jar
For that reason I tend to favour option 2... There is the added benefit that option 2 does not mean writing code to work around a limitation in a build tool plugin... The less you lock yourself into a specific build tool, the better IMHO
This actually works quite fine with the newer surefire and failsafe plugins, see related questions:
Run JUnit Tests contained in dependency jar using Maven Surefire
run maven tests from classpath
So you don't need to unpack the jar anymore, you just provide the group and artifact id for the dependencies to scan (this works with both "main jar" dependencies, as well as "test-jar" dependencies)
The attached test-jar can be used as a usual dependency in other project which supports reuse of code in the test area but you can't run tests out of the jar. If you really need the solution you have to write at least a single suite (etc.?) to start the tests from the jar.
What are the steps to get Team Foundation Server running unit tests when a given build runs?
What are the caveats / pitfalls / workarounds a dev or sysadmin should be aware of when setting up a TFS server to do this for the first time?
What are common troubleshooting steps for unit test problems during builds?
it depends on which version of TFS you are running, so I will assume it is 2008.
Firstly, you must have Team Edition for Testers installed on the computer that will act as your build agent, as stated in How To: Create a Build Definition
There are a couple of ways to tell Team Build to run tests for your build.
Unit tests can be run from a defined Test List within the Solution being built. This list is referenced by the build definition and all tests within the chosen list(s) are executed. More info here
WildCard test exectution is also available by defining a wildcard mask (i.e. Test*.dll) that instructs Team Build to run any tests present in assemblies that match the mask. This is done when defining the build definition as well.
Things to note:
If you intend to use the wildcard method and want to enable code coverage for your test configuration, you must add the following to your build definition file to enable it.
<RunConfigFile>$(SolutionRoot)\TestRunConfig.testrunconfig</RunConfigFile>
See my previous question on this for more info here
If you don't want to use test configs (A Pain in the ass to manage) just run all the test in a .dll by adding this to the build config:
<ItemGroup>
<TestContainerInOutput Include="MyProject.UnitTests.dll" />
</ItemGroup>
The whole process is smooth and fairly simple. You can inspect the unit tests that filaed on the build server by opening the test result file locally (a bit of a pain) but generally you'll just run the unit tests locally and you can see the results immediately.
If you are used to NUnit, you may opt to sort the tests by classname, it gives a similar view.
Careful with code coverage, it makes complete copies of your binaries on compile. If your binaries are sufficiently large and you compile often, it will eat through drive space quickly.
http://msdn.microsoft.com/en-us/library/cc981972(v=vs.90).aspx
I like this defination as it gives you a complete 'walkthrough' from
Creating the Project
Creating the Unit Test Project
To configuring Team build to use it Unit Test