I've just started working on a Java project (as I usually use .net), and one of the first things that strikes me as odd is that in the Maven project there is a /src and a /test directory where obviously the source code and the tests should go.
In .net I preferred to have the tests in a separate assembly/project, so for example I would have:
MyProject
MyProject.Tests
That way I dont have to bloat my deployed code with any tests and it makes it easier to test my code in true isolation and in a lot of cases I didn't bother writing tests per project, I would just have solution wide unit/integration/acceptance tests i.e MySolution.UnitTests, MySolution.IntegrationTests.
However in Java it just seems to be bundled together, and I would rather separate it out, however I hear that Maven is a cruel mistress when you want to do things differently to the default structures.
So to reign this post back in, my main questions are:
Is there a way to separate out the tests from the project
Based on the above information are there any pros for actually having the tests within the project? (other than when you check it out you always have the tests there)
I dont have to bloat my deployed code with any tests
The deployable artifacts (jar file, war files) will not contain the test classes or data.
Is there a way to separate out the tests from the project
You could split it into two projects, with a "Test" project containing only the tests, and depending on the "real" project.
However, especially with Maven, you probably want to follow the same project layout conventions that everyone else (or at least the majority) has. This will make your life easier (less configuration).
Again, the tests will not make it into the product to be deployed, so the current layout should not be a problem.
I would just have solution wide unit/integration/acceptance tests i.e MySolution.UnitTests, MySolution.IntegrationTests.
For integration tests, that actually makes sense. In this case, define a "Test" project that depends on all the other projects that make up the solution (I'd still keep the unit tests with the project they test).
In a default maven setup, the tests are only executed, but not deployed.
Per convention, everything inside src/main lands in the target archive, while everything else doesn't.
Per default, a maven JAR project creates a jar with just the classes that are compiled from src/main/java. You can use different plugin goals to create:
test jar (jar of compiled test classes)
source jar (jar of main sources)
test source jar (jar of test sources)
javadoc jar (jar of javadoc api documentation)
But all of these require extra steps.
Related
I have a big code base in TFS which has multiple .sln files, each with many projects and at least one unit test project. Most of the unit tests rely on common XML and XSD files, and there are several other types of files (.config, .xaml, etc) that are needed by the code when unit testing.
Because of the way that TFS builds and gathers the files for unit testing, most of those files are missing from the TestResults folder, so the tests are failing during our CI builds [this has been happening for a while, but I'm new to the project, and am trying to fix the errors]. What TFS appears to do is this: First, it checks out all the code to a src folder (with Solution1, Solution2, etc) and builds it, just like the developers do locally. Second, it copies the build outputs to a bin\Binaries folder. Third, it looks for all the test.dll files, copies them and their dependencies (but only the dependencies), plus the App.config file to TestResults\Deploy_[date/time]\Out folder, and runs the unit tests there.
I am encountering two problems with this. Because the second step is combining all the build outputs into one folder, all the files with duplicate names are overwriting each other. So, there is only one App.config file, even though each solution has its own. This is happening with other config/xml files too, and with two poorly named unit test .dlls. I can live with this if I have to because most of those config files are duplicates, and other files can be renamed.
The second problem is that most of those extra files don't make it into the TestResults folder, and when they aren't there the unit tests will fail. I know about using the [DeploymentItem] attribute; if that is the only solution, I will go that route, but there are so many extra files that I am looking for a different approach.
So my question is, how can I configure my tfs-run builds & unit tests to include all of the files that they need, without all the work & maintenance problems of adding a lot of [DeploymentItem] attributes, and also without affecting the local builds & unit tests for the developers?
Update
One thing I've found is that adding a [DeploymentItem] attribute to a unit test actually causes the deployment folder to be used. Without that, it runs the unit tests in the binaries folder. See http://msdn.microsoft.com/en-us/library/ms182475.aspx, under "When is a separate deployment folder used?"
Also on that page, it says you can specify files to deploy in a .testsettings file, but then says you should avoid using it because your tests will run slower. The newer alternative, a .runsettings file, does not let you list what files to deploy.
It also appears that deployment becomes enabled if code coverage is enabled, which we don't currently do, but plan to once the tests are passing.
In TFS 2013 you can execute a powershell pre-test to organize the files in a way that you need. You can get files to be deployed ad part of the tests with a test settings file for pre2013.
If however the test setting file is not enough you can use the Community Build tools to call a powershell directly in previous versions to 2013.
If you are stuck on 2012 then you will need to use the .testsettings file to push the bits you need. Yes it will make your build slower but that's your only choice other than customising the build process as above.
E.g. I have a file ConnectionDummy in Maven project Common. It is used only by tests in Email-Lib, Scanner and Common. Note Common itself also includes unit tests, but these files are just helpers to unit tests. Should it go Common's test or main?
Maven setting but I think this is probably common to many technologies.
It depends a little bit on how strong you feel about having test (utility) code in your production system, down the road. Is code that's just sitting there, doing nothing, bad for your production environment? Imagine that your test utility has nifty annotations that e.g. Spring picks up during run-time?
I've come across people who don't care. Personally I feel strong about it, and don't want test code in my production environments. If that's your thinking too, there's two ways to go with Maven that I've seen in our projects:
The test utility code goes in Common's test folder. You'll end up building test JARs (test-jar) for your Common project and you may run into transitive dependency issues where any <scope>test</scope> dependencies of your Common project that are needed by your test utility code are not going to be there when someone depends on your test JAR. This is a potential dependency nightmare where I've seen the "including project" re-defining a whole lot of dependencies that were already nicely defined by the "included project", but weren't picked up by Maven for being transitive at scope test.
The test utility code goes in a separate test project, e.g. CommonTest which produces jar, not test-jar packaging. That way the test utility becomes "utility" to not only your dependent projects, but to the original Common project itself, just the same. By doing so, your test folder becomes truly an internal matter of a project. Maybe that's how the Maven guys have meant it, why else would making a test-jar not be a standard option? :)
I have a maven build process that publishes executable jars and their tests to Nexus.
I have another maven build process that needs to access these jars (executable + test) and run the tests.
How do I go about it? So far I have managed to do this only if the jar is exploded to class files.
I am new to maven and completely lost in the documentation.
Update 2022-03-11
The feature has been implemented, see https://stackoverflow.com/a/17061755/1589700 for details
Original answer
Surefire and failsafe do not currently support running tests from within a jar.
This is largely a case of not being able to identify the tests.
There are two ways to get the tests to run.
Use a test suite that lists all the tests from the test-jar. Because the test suite will be in src/test/java (more correctly will be compiled into target/test-classes) that will be picked up and all the tests in the suite will be run by Surefire/failsafe (assuming the suite class name matches the includes rule: starts or ends with Test)
Use the maven dependency plugin's unpack-dependencies goal to unpack the test-jar into target/test-classes (this screams of hack, but works quite well)
The main issue with the first option is that you cannot easily run just one test from the suite, and you need to name every test from the test-jar
For that reason I tend to favour option 2... There is the added benefit that option 2 does not mean writing code to work around a limitation in a build tool plugin... The less you lock yourself into a specific build tool, the better IMHO
This actually works quite fine with the newer surefire and failsafe plugins, see related questions:
Run JUnit Tests contained in dependency jar using Maven Surefire
run maven tests from classpath
So you don't need to unpack the jar anymore, you just provide the group and artifact id for the dependencies to scan (this works with both "main jar" dependencies, as well as "test-jar" dependencies)
The attached test-jar can be used as a usual dependency in other project which supports reuse of code in the test area but you can't run tests out of the jar. If you really need the solution you have to write at least a single suite (etc.?) to start the tests from the jar.
Here is an example of my source control structure in TFS 2010:
TFS Project
Development
Branches
Source
Tests
Under my Source directory I have created a directory for my Data Access Layer. The solution is currently not dependent on any other project outside its own solution. I did this so I could branch just this code. I wanted to avoid having to branch the entire Source folder.
I personally like having my unit tests as part of a solution so I can easily create and run tests against the code without having to go into another solution.
If I put my unit test projects in the same solution as my DAL solution, what is the best way to branch my entire DAL without the unit test project dependencies? Should I create another solution without unit Tests for branching? Should I include the unit tests in the branch after I branch the solution?
Since the Tests are located in the Tests folder, there isn't a way to selectively branch folders in TFS.
How do other users approach this?
The unit tests also help to act as documentation for the code and examples of how it behaves and how to use it.
Therefore, if you're branching to change the code's behavior, I recommend branching the unit tests along with the code. That way your documentation and examples are always up to date.
We develop a C++ application using Visual Studio 2008 and unit test using Boost.Test. At the moment, we have a separate solution which contains our unit tests.
Many of our projects in the core solution produce DLL's. We're limited in test coverage because we cannot test non-exported classes.
I have two ideas on how these could be tested:
Export everything
Put the tests inside the DLL (same project and solution) and use Boost.Test's external runner
I'm not entirely sure what the drawbacks would be. Number 1 above breaks module level encapsulation, and number 2 could result in a much larger DLL, unless it's possible to only include the test code in certain configurations.
So, are there any severe drawbacks to the above methods, or can you think of other solutions?
Expanding on Tom Quarendon's answer to this question, I have used a slight variant of Simon Steele's response:
Create a test project (using whatever test framework you like, I use CppUnit).
In your test_case.cpp, #include <header/in/source/project.h>.
In the test project properties:
In Linker->General, add the source project's $(IntDir) to the Additional Library Directories.
In Linker->Input, add the .obj files to the Additional Dependencies.
Add the dependency from the test project to the source project in Project->Project Dependencies.
Again, the only maintenance overhead is the standard one for unit tests - to create the dependency on the unit(s) you want to test.
The solution I use for this is to build the same non-exported code into my tests DLL as well. This does increase build time and means adding everything to both projects, but saves exporting everything or putting the tests in the main product code.
Another posibility would be to compile the non-exported code into a lib which is used by both the DLL with exports, and the unit test project.
Was searching a solution as well, maybe the following will be easier to maintain.
Add a new build configuration e.g. "Unit testing Debug" to the DLL project and change the Configuration Type to be "Static Library .lib" ("General"->"Configuration Type").
Then just add a dependency of your unit tests on this project, now everything should link together when you use new build configuration "Unit testing Debug".
If you are using release builds for unit tests then you need to add another configuration with release optimizations.
So the benefits of this solution are:
low maintanability cost
single DLL/Static library project
don't have to manually link to .obj files
Drawbacks:
Extra configuration profile(s) will require some changes in your build environment (CI)
Greater compilation times
Update:
We actually ended up using a different approach.
We added new "Test debug"/"Test release' configurations for every existing project that we have.
For .exe/.dll projects we disable the original main.cpp from compiling and replaced it with the one that instantiates the test framework (e.g. gtest) and runs all the tests, the tests are in separate .cpp files which are also excluded from compilation in regular configurations (Release/Debug) and enabled only in Test configurations.
For .lib projects we also have new "Test debug"/"Test release" configurations and there we convert the static library to be an .exe file and provide a main.cpp which instantiates the testing framework and runs the tests and tests themselves. Test related files are excluded from compilation on Release/Debug configurations.
Try making a define such as the following somewhere all files will include:
#define EXPORTTESTING __declspec(dllexport)
And use it in place of the dllexport, like this:
class EXPORTTESTING Foo
{
...
};
Then you will be able to turn off the flag for building a release DLL, but keep it on for a unit-testable DLL.