I have a big code base in TFS which has multiple .sln files, each with many projects and at least one unit test project. Most of the unit tests rely on common XML and XSD files, and there are several other types of files (.config, .xaml, etc) that are needed by the code when unit testing.
Because of the way that TFS builds and gathers the files for unit testing, most of those files are missing from the TestResults folder, so the tests are failing during our CI builds [this has been happening for a while, but I'm new to the project, and am trying to fix the errors]. What TFS appears to do is this: First, it checks out all the code to a src folder (with Solution1, Solution2, etc) and builds it, just like the developers do locally. Second, it copies the build outputs to a bin\Binaries folder. Third, it looks for all the test.dll files, copies them and their dependencies (but only the dependencies), plus the App.config file to TestResults\Deploy_[date/time]\Out folder, and runs the unit tests there.
I am encountering two problems with this. Because the second step is combining all the build outputs into one folder, all the files with duplicate names are overwriting each other. So, there is only one App.config file, even though each solution has its own. This is happening with other config/xml files too, and with two poorly named unit test .dlls. I can live with this if I have to because most of those config files are duplicates, and other files can be renamed.
The second problem is that most of those extra files don't make it into the TestResults folder, and when they aren't there the unit tests will fail. I know about using the [DeploymentItem] attribute; if that is the only solution, I will go that route, but there are so many extra files that I am looking for a different approach.
So my question is, how can I configure my tfs-run builds & unit tests to include all of the files that they need, without all the work & maintenance problems of adding a lot of [DeploymentItem] attributes, and also without affecting the local builds & unit tests for the developers?
Update
One thing I've found is that adding a [DeploymentItem] attribute to a unit test actually causes the deployment folder to be used. Without that, it runs the unit tests in the binaries folder. See http://msdn.microsoft.com/en-us/library/ms182475.aspx, under "When is a separate deployment folder used?"
Also on that page, it says you can specify files to deploy in a .testsettings file, but then says you should avoid using it because your tests will run slower. The newer alternative, a .runsettings file, does not let you list what files to deploy.
It also appears that deployment becomes enabled if code coverage is enabled, which we don't currently do, but plan to once the tests are passing.
In TFS 2013 you can execute a powershell pre-test to organize the files in a way that you need. You can get files to be deployed ad part of the tests with a test settings file for pre2013.
If however the test setting file is not enough you can use the Community Build tools to call a powershell directly in previous versions to 2013.
If you are stuck on 2012 then you will need to use the .testsettings file to push the bits you need. Yes it will make your build slower but that's your only choice other than customising the build process as above.
Related
So, we are in this process of migrating XAML Builds to vNext (2015) Builds on TFS, and we are trying to "do things as clean as possible", since we had many, many customizations on the XAML builds that could be avoided and actually gave us problems along the way.
One major issue we are facing is with paths and "global files". Let me explain:
There are some files that, for commodity reasons, we have on a single place and every SLN file on that Collection refers them. Those files are such ones as Code Analysis RuleSets, Signing Files (SNK), etc. So the change is made in one place only and it affects every build.
Well, in XAML Builds we have a Build that runs with CI that downloads (Gets) those files, and since we hammered-in the same exact pathing for TFS and Machine (with a environment variable for the beginning of the path), the path is the same on the Developers and Build machines. However, this creates dependencies between builds and workspace issues.
My question here is, is there a configuration that I am missing that allows referring to files in other branches other than the build one? Since I’m trying to keep the build machines as “disposable” as possible, it’s running with an Agent Config Out of the Box: No custom paths, no hardwiring.
I already tried referring the files directly with their source control path, for example. The only options I’m seeing are either creating a PowerShell/CMD Script that downloads those files right into the same folder as the SLN or keeping it “as it is” and use relative paths putting a “Build” Build Step before the actual Build Step so it downloads the files to the server.
Isn’t there an “Elegant” way of doing this? Or is our methodology wrong from the get go?
You can add a Copy Files step to copy the files that the build needs:
We have a Microsoft TeamFoundationServer 2013.
Now we have many different TeamProjects with solutions.
Now I have to create a build-definition which builds many of those solutions at once.
I can specify the build-output in the definition as follows: Solution Specific Build Outputs: True
So this puts the build-outputs of every single project to a folder with the according name.
But now, I have to run the unit-tests for every solution. If I just enable it, many of the tests fail.
This is because my BuildDefinition throws all binaries which it wants to test into the folder TestResults\Deploy_USERNAME 2015-03-26 11_30_00\Out.
But now, there's the problem: Many of those projects contain a app.config for connection-strings (or a web.config). But obviously there can only be one such file in a single directory, so it overrides every file until the last project copies its app.config on top of all others. Now the unit-tests for that last project just run well, because it's got all necessary connection-string. But for all other projects, many will fail.
Is there an option to split those test-bins into different folders, just as the build-output?
Or is there any other workaround for this problem?
Environment:
Team Foundation Server 2013
Windows Server 2012
Build-Template: MultiBuildTemplate.xaml
We are using VS2012 and TFS2012 and write unit tests for our code. We want to report code coverage, and also using .config files in our unit tests for test appsettings, and also some other settings for logging, MS Enterprise library settings etc. etc.
App.config not working in new test framework
New test framework of MS should be great, but to me it is not so great at all.
How I'm i suppose to set some basic configuration in config files, when the new framework does not use config files anymore?
We had a problem with mixed mode dlls, and found a fix: adding
<startup useLegacyV2RuntimeActivationPolicy="true">
to the app.config. But this did not work for our unit test project. Becuase config files are not there anymore. Searching the internet came up with a solution
'Problems with .Net 2.0 Mixed Mode Assemblies inside Visual Studio .Net 4.5 Test Projects'
This means editing a file of Visual Studio 11 itself in the program files directory, not a great solution i think....
And how about some basic appsetting? How am I supposed to set this?
Do not use the .testSettings file
Using the old .testsettings file is also not recommended by MS, becuase then the old test framework is used. And if I use the .testsettings file, i cannot setup Code Coverage on my tfs2012 build service.
Another issue is that we have code that need a dll (system.data.sqlite.dll), but only at runtime the unit test code needs this dll. So a reference is not needed. We fixed this by using the Deployment tab on the testsettings file. But in the new framework, you should not use the testsettings file. You have the [deploymentitem] attribute if you need files. But the deploymentitem attribute can only be used on a [testmethod] not on a [testinitialize] or [assemblyinitialize] method. But our code needs the dll in the [testinitialize] method. So there is no way to get the dll in place.
Just copy it with File.Copy in the [assemblyinitialize] (or testinitialize) method does not work.
Adding the dll as file to the project, and set the 'copy to output directory' to 'Copy Always' as mentioned in 'Configuring Unit Tests by using a .runsettings File' also does not work at all.
The (really not great) solution for this is to add the dll as a reference, then instantiate a class and do nothting with it. This way the dll is needed otherwise it is not building, and thus the dll will deploy itself to the right directories.
how to solve my problem(s)???
- I want to use config files in my unit test.
- I want to deploy some files that are neede in the 'assemblyinitialise' and/or 'classinitialize' methods.
- I want Code Coverage on my TFS2012 nightly build enabled.
a) App.config not working in new test framework
This should still work. What I think is missing in this case is that this .config file is not being copied with your test dll. Could you please set this as a deployment item and try again?
b) Do not use the .testSettings file
.testsettings and code coverage.
Setting up code coverage with the .testsettings file IS still supported in VS 2012 build. You simply need to select the mstest 2010 test runner and specify your .testsettings file in your build definition
If you dont have anything except code coverage settings in the .testsettings file then you can easily migrate to the 2012 test runner and select "enable code coverage" in the drop down items
copying a file required by test initalize
You could do this via the .testsettings file or you can have a post-build file copy task. It is pretty straight forward to do so and has no impact on anything else. Using the "copy to output directory = copy always" does work. Please try it with a sample solution and see if you can narrow down on why this does not work on your setup.
We develop a C++ application using Visual Studio 2008 and unit test using Boost.Test. At the moment, we have a separate solution which contains our unit tests.
Many of our projects in the core solution produce DLL's. We're limited in test coverage because we cannot test non-exported classes.
I have two ideas on how these could be tested:
Export everything
Put the tests inside the DLL (same project and solution) and use Boost.Test's external runner
I'm not entirely sure what the drawbacks would be. Number 1 above breaks module level encapsulation, and number 2 could result in a much larger DLL, unless it's possible to only include the test code in certain configurations.
So, are there any severe drawbacks to the above methods, or can you think of other solutions?
Expanding on Tom Quarendon's answer to this question, I have used a slight variant of Simon Steele's response:
Create a test project (using whatever test framework you like, I use CppUnit).
In your test_case.cpp, #include <header/in/source/project.h>.
In the test project properties:
In Linker->General, add the source project's $(IntDir) to the Additional Library Directories.
In Linker->Input, add the .obj files to the Additional Dependencies.
Add the dependency from the test project to the source project in Project->Project Dependencies.
Again, the only maintenance overhead is the standard one for unit tests - to create the dependency on the unit(s) you want to test.
The solution I use for this is to build the same non-exported code into my tests DLL as well. This does increase build time and means adding everything to both projects, but saves exporting everything or putting the tests in the main product code.
Another posibility would be to compile the non-exported code into a lib which is used by both the DLL with exports, and the unit test project.
Was searching a solution as well, maybe the following will be easier to maintain.
Add a new build configuration e.g. "Unit testing Debug" to the DLL project and change the Configuration Type to be "Static Library .lib" ("General"->"Configuration Type").
Then just add a dependency of your unit tests on this project, now everything should link together when you use new build configuration "Unit testing Debug".
If you are using release builds for unit tests then you need to add another configuration with release optimizations.
So the benefits of this solution are:
low maintanability cost
single DLL/Static library project
don't have to manually link to .obj files
Drawbacks:
Extra configuration profile(s) will require some changes in your build environment (CI)
Greater compilation times
Update:
We actually ended up using a different approach.
We added new "Test debug"/"Test release' configurations for every existing project that we have.
For .exe/.dll projects we disable the original main.cpp from compiling and replaced it with the one that instantiates the test framework (e.g. gtest) and runs all the tests, the tests are in separate .cpp files which are also excluded from compilation in regular configurations (Release/Debug) and enabled only in Test configurations.
For .lib projects we also have new "Test debug"/"Test release" configurations and there we convert the static library to be an .exe file and provide a main.cpp which instantiates the testing framework and runs the tests and tests themselves. Test related files are excluded from compilation on Release/Debug configurations.
Try making a define such as the following somewhere all files will include:
#define EXPORTTESTING __declspec(dllexport)
And use it in place of the dllexport, like this:
class EXPORTTESTING Foo
{
...
};
Then you will be able to turn off the flag for building a release DLL, but keep it on for a unit-testable DLL.
I've just started working on a Java project (as I usually use .net), and one of the first things that strikes me as odd is that in the Maven project there is a /src and a /test directory where obviously the source code and the tests should go.
In .net I preferred to have the tests in a separate assembly/project, so for example I would have:
MyProject
MyProject.Tests
That way I dont have to bloat my deployed code with any tests and it makes it easier to test my code in true isolation and in a lot of cases I didn't bother writing tests per project, I would just have solution wide unit/integration/acceptance tests i.e MySolution.UnitTests, MySolution.IntegrationTests.
However in Java it just seems to be bundled together, and I would rather separate it out, however I hear that Maven is a cruel mistress when you want to do things differently to the default structures.
So to reign this post back in, my main questions are:
Is there a way to separate out the tests from the project
Based on the above information are there any pros for actually having the tests within the project? (other than when you check it out you always have the tests there)
I dont have to bloat my deployed code with any tests
The deployable artifacts (jar file, war files) will not contain the test classes or data.
Is there a way to separate out the tests from the project
You could split it into two projects, with a "Test" project containing only the tests, and depending on the "real" project.
However, especially with Maven, you probably want to follow the same project layout conventions that everyone else (or at least the majority) has. This will make your life easier (less configuration).
Again, the tests will not make it into the product to be deployed, so the current layout should not be a problem.
I would just have solution wide unit/integration/acceptance tests i.e MySolution.UnitTests, MySolution.IntegrationTests.
For integration tests, that actually makes sense. In this case, define a "Test" project that depends on all the other projects that make up the solution (I'd still keep the unit tests with the project they test).
In a default maven setup, the tests are only executed, but not deployed.
Per convention, everything inside src/main lands in the target archive, while everything else doesn't.
Per default, a maven JAR project creates a jar with just the classes that are compiled from src/main/java. You can use different plugin goals to create:
test jar (jar of compiled test classes)
source jar (jar of main sources)
test source jar (jar of test sources)
javadoc jar (jar of javadoc api documentation)
But all of these require extra steps.