I use Coverity to scan my project for security issues.
What I would like to know is how to exclude any java test (NOTE: both integration and unit) from the analysis results that are available after a defect commit.
I did use maven to build the project and I excluded Unit tests using the flag -Dmaven.skip.test=true.
Though that made Coverity not scanning unit tests, it still makes it scan integration ones.
All the integration tests in my project contain the word "Test" in the file titles. Therefore I started looking at the filter section made available in Coverity. What I tried then was a regex (.*(!?(Test).*)$) but it did not work. It seems that coverity supports two matching character (* and ? - see image below) while it does not seem to support any negative look-around.
Is there any good way to accomplish this task in an easy and clean fashion?
Since Coverity relies on your Maven build, you can exclude:
compilation and execution of both unit tests (by Surefire plugin) and integration tests (by Failsafe plugin) by adding -Dmaven.skip.test=true
execution of both unit and integration tests via -DskipTests
execution of integration tests via -DskipITs
Instead, if you have your integration tests in a separate Maven module, you can directly exclude that from the Maven build via profile, like in below example -- see extract of aggregator's pom.xml and maven command line to be launched:
<modules>
<!-- remove 'my-it-module' from this list -->
</modules>
<profiles>
<profile><id>build-it</id>
<activation><activeByDefault>true</activeByDefault></activation>
<modules><module>my-it-module</module></modules>
</profile>
</profiles>
and then mvn install -P !build-it
Related
My project is composed of several DLLs that I must test with different unit tests. I generate an XML file for each of the tests then I transform them into a JUnit format to be read by Jenkins. It works very well.
On the other hand, the test results are all in the same topic on the Jenkins interface and I would like to separate them for each DLL. It is for a more practical aspect for the visualization of the tests. I haven't found the solution yet. That's why I'm asking you if there is a solution for this problem.
I tried several plugins like JUnit or Warning NG. But the result remains the same. The JUnit plugin puts all the results in the same section and makes no distinction and the Warning NG plugin fails to parse the XML report to display it in Jenkins.
I have a maven build process that publishes executable jars and their tests to Nexus.
I have another maven build process that needs to access these jars (executable + test) and run the tests.
How do I go about it? So far I have managed to do this only if the jar is exploded to class files.
I am new to maven and completely lost in the documentation.
Update 2022-03-11
The feature has been implemented, see https://stackoverflow.com/a/17061755/1589700 for details
Original answer
Surefire and failsafe do not currently support running tests from within a jar.
This is largely a case of not being able to identify the tests.
There are two ways to get the tests to run.
Use a test suite that lists all the tests from the test-jar. Because the test suite will be in src/test/java (more correctly will be compiled into target/test-classes) that will be picked up and all the tests in the suite will be run by Surefire/failsafe (assuming the suite class name matches the includes rule: starts or ends with Test)
Use the maven dependency plugin's unpack-dependencies goal to unpack the test-jar into target/test-classes (this screams of hack, but works quite well)
The main issue with the first option is that you cannot easily run just one test from the suite, and you need to name every test from the test-jar
For that reason I tend to favour option 2... There is the added benefit that option 2 does not mean writing code to work around a limitation in a build tool plugin... The less you lock yourself into a specific build tool, the better IMHO
This actually works quite fine with the newer surefire and failsafe plugins, see related questions:
Run JUnit Tests contained in dependency jar using Maven Surefire
run maven tests from classpath
So you don't need to unpack the jar anymore, you just provide the group and artifact id for the dependencies to scan (this works with both "main jar" dependencies, as well as "test-jar" dependencies)
The attached test-jar can be used as a usual dependency in other project which supports reuse of code in the test area but you can't run tests out of the jar. If you really need the solution you have to write at least a single suite (etc.?) to start the tests from the jar.
I have a maven super pom, consisting of multiple modules. I run mvn emma:emma on the super pom, and I get emma code coverage for all the modules (that have tests defined.)
Now I wish to exclude one of the modules from the emma code coverage run. Such that when I run mvn emma:emma on the super pom, the specific module does not get code coverage measured. But when I do mvn test all module tests will be executed as normally.
How do I do that?
As I see in Maven Emma plugin docs, there's not such property to exclude some projects (modules) from coverage measurement. It seems that all reactor projects will be handled by Emma plugin.
What can be useful in your case is the plugin's ability to exclude some test classes (maven.emma.filter.excludes property) from coverage report, but it would work only of you're able to specify masks that catch test classes of modules' you want to effectively exclude, so it might not work in your case.
I have a maven project and lots of junit classes in it. I develop with Eclipse. I want to separate functional test classes and integration testing classes.
When I build the project in Eclipse then I want only the functional test classes to be executed.
By jenkins both of them should be executed.
Which approach should i follow?
I find it more convenient to put integration tests in separate projects, and then run them as if they were unit tests by relying on Maven's default life cycle. As I have to run my tests against different environments, this approach makes it easier to manage environment specific tests.
Let's assume I have an application, represented by the application Maven aggregator project, which contains a jar module called project. I keep unit tests within project itself, so that they get executed whenever I build my application. This is also built every night by Jenkins; ideally successful builds should be automatically deployed to one or more test environments, for both manual and automatic tests. Currently this is done by hand.
For every environment where I need to run my integration tests I have an applicationTestEnvX Maven aggregator project. This contains at least a projectTest module, where I keep those integration tests that are environment independent, as well as any test support code. Tests for my project module that are specific to environment X are kept in a projectTestEnvX module. I have a Jenkins job for each applicationTestEnvX project, which runs my tests every night. Ideally these should be run against the result of the application build, but I'm not there yet.
There is also a direct correspondence with how my projects are stored in Subversion and my Eclipse workspaces, but that's another story ;-)
Have a look at two Maven plugins: Surefire (for unit tests) and Failsafe (for integration tests). They closely resemble each other, Failsafe is a clone of Surefire.
Organize your tests so that their naming schema goes with proposed configuration: **/*Test.java for unit tests and **/*IT.java for integration. Surefire is run by default, for Failsafe you'll need extra excerpt in POM — example and more info in this answer.
Then it's down to mvn test or mvn integration-test.
If you want to run integration test only in certain environments (Jenkins), you could make Failsafe executing only in a profile, for example:
<profiles>
<profile>
<id>env-itest</id>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-failsafe-plugin</artifactId>
<executions>
<execution>
<id>integration-test</id>
<goals><goal>integration-test</goal></goals>
</execution>
<!-- other executions, if needed -->
</executions>
</plugin>
</plugins>
</profile>
</profiles>
Then, on Jenkins, you run mvn clean install -P env-itest and on your local environment only mvn clean install (or simliar).
Have a look at the maven documentation around the integration test phase - you can use different plugins to control which tests are run, simply by naming the tests appropriately.
Then running mvn test will build & run your code & unit tests, which mvn verify will run your integration tests as well.
You can separate your unit and integration tests into separate packages (or perhaps even separate source folders, but then you'd have to updated your Maven configuration to recognize that you have two separate source folders for tests).
To take advantage of this, in Eclipse's Run Configurations (Run > Run Configurations), create a new JUnit run configuration that "Run all tests in the selected project, package or source folder:", select the the package/source folder containing only the tests you want to run.
When I first read your question, I got it backwards. I thought you wanted to run the full suite in Eclipse, and only a subset in Jenkins. I'm going to just leave my old answer up in case you find this useful some how:
The way I've done this before is through naming convention of the JUnit Test Cases.
I would name all the unit test test cases ...UnitTest (e.g., RegistrationManagerUnitTest) and integration test test cases, I'd name ...IntegrationTest (e.g., RegistrationDaoIntegrationTest).
Then in Maven, you can configure it to run all the test cases whose classes end with ...UnitTest (by default it's looking for classes whose name end with ...Test. Something along the lines of:
<plugin>
<artifactId>maven-surefire-plugin</artifactId>
<configuration>
<!-- Run only tests whose name end with "UnitTest" -->
<includes>
<include>**/*UnitTest.java</include>
</includes>
</configuration>
</plugin>
FYI, with TestNG, you would simply use groups, e.g. #Test(groups = "integration") and #Test(groups = "unit"). Then you simply run different groups depending on what you need.
What are the steps to get Team Foundation Server running unit tests when a given build runs?
What are the caveats / pitfalls / workarounds a dev or sysadmin should be aware of when setting up a TFS server to do this for the first time?
What are common troubleshooting steps for unit test problems during builds?
it depends on which version of TFS you are running, so I will assume it is 2008.
Firstly, you must have Team Edition for Testers installed on the computer that will act as your build agent, as stated in How To: Create a Build Definition
There are a couple of ways to tell Team Build to run tests for your build.
Unit tests can be run from a defined Test List within the Solution being built. This list is referenced by the build definition and all tests within the chosen list(s) are executed. More info here
WildCard test exectution is also available by defining a wildcard mask (i.e. Test*.dll) that instructs Team Build to run any tests present in assemblies that match the mask. This is done when defining the build definition as well.
Things to note:
If you intend to use the wildcard method and want to enable code coverage for your test configuration, you must add the following to your build definition file to enable it.
<RunConfigFile>$(SolutionRoot)\TestRunConfig.testrunconfig</RunConfigFile>
See my previous question on this for more info here
If you don't want to use test configs (A Pain in the ass to manage) just run all the test in a .dll by adding this to the build config:
<ItemGroup>
<TestContainerInOutput Include="MyProject.UnitTests.dll" />
</ItemGroup>
The whole process is smooth and fairly simple. You can inspect the unit tests that filaed on the build server by opening the test result file locally (a bit of a pain) but generally you'll just run the unit tests locally and you can see the results immediately.
If you are used to NUnit, you may opt to sort the tests by classname, it gives a similar view.
Careful with code coverage, it makes complete copies of your binaries on compile. If your binaries are sufficiently large and you compile often, it will eat through drive space quickly.
http://msdn.microsoft.com/en-us/library/cc981972(v=vs.90).aspx
I like this defination as it gives you a complete 'walkthrough' from
Creating the Project
Creating the Unit Test Project
To configuring Team build to use it Unit Test