We are using VS2012 and TFS2012 and write unit tests for our code. We want to report code coverage, and also using .config files in our unit tests for test appsettings, and also some other settings for logging, MS Enterprise library settings etc. etc.
App.config not working in new test framework
New test framework of MS should be great, but to me it is not so great at all.
How I'm i suppose to set some basic configuration in config files, when the new framework does not use config files anymore?
We had a problem with mixed mode dlls, and found a fix: adding
<startup useLegacyV2RuntimeActivationPolicy="true">
to the app.config. But this did not work for our unit test project. Becuase config files are not there anymore. Searching the internet came up with a solution
'Problems with .Net 2.0 Mixed Mode Assemblies inside Visual Studio .Net 4.5 Test Projects'
This means editing a file of Visual Studio 11 itself in the program files directory, not a great solution i think....
And how about some basic appsetting? How am I supposed to set this?
Do not use the .testSettings file
Using the old .testsettings file is also not recommended by MS, becuase then the old test framework is used. And if I use the .testsettings file, i cannot setup Code Coverage on my tfs2012 build service.
Another issue is that we have code that need a dll (system.data.sqlite.dll), but only at runtime the unit test code needs this dll. So a reference is not needed. We fixed this by using the Deployment tab on the testsettings file. But in the new framework, you should not use the testsettings file. You have the [deploymentitem] attribute if you need files. But the deploymentitem attribute can only be used on a [testmethod] not on a [testinitialize] or [assemblyinitialize] method. But our code needs the dll in the [testinitialize] method. So there is no way to get the dll in place.
Just copy it with File.Copy in the [assemblyinitialize] (or testinitialize) method does not work.
Adding the dll as file to the project, and set the 'copy to output directory' to 'Copy Always' as mentioned in 'Configuring Unit Tests by using a .runsettings File' also does not work at all.
The (really not great) solution for this is to add the dll as a reference, then instantiate a class and do nothting with it. This way the dll is needed otherwise it is not building, and thus the dll will deploy itself to the right directories.
how to solve my problem(s)???
- I want to use config files in my unit test.
- I want to deploy some files that are neede in the 'assemblyinitialise' and/or 'classinitialize' methods.
- I want Code Coverage on my TFS2012 nightly build enabled.
a) App.config not working in new test framework
This should still work. What I think is missing in this case is that this .config file is not being copied with your test dll. Could you please set this as a deployment item and try again?
b) Do not use the .testSettings file
.testsettings and code coverage.
Setting up code coverage with the .testsettings file IS still supported in VS 2012 build. You simply need to select the mstest 2010 test runner and specify your .testsettings file in your build definition
If you dont have anything except code coverage settings in the .testsettings file then you can easily migrate to the 2012 test runner and select "enable code coverage" in the drop down items
copying a file required by test initalize
You could do this via the .testsettings file or you can have a post-build file copy task. It is pretty straight forward to do so and has no impact on anything else. Using the "copy to output directory = copy always" does work. Please try it with a sample solution and see if you can narrow down on why this does not work on your setup.
Related
I have a Visual Studio web test that is using a custom JSON Extraction Rule. Initially the rule was in the web test's project/assembly. When running the test, it fails during the extraction, saying
Could not load file or assembly `ProjectName, Version=1.0.0.0` or one of its dependencies. The system cannot find the file specified.
In the same test I'm using a custom validation rule defined in a different project and this is working, so I copied the Extraction Rule to the other project. I manually edited the webtest XML to use the other project's namespace and assembly name. I re-run the web test after saving/cleaning/building.
I get the same error.
I confirm in my testsettings file that I've said to deploy the bin folder of the project. And the other project. And some other random stuff just out of desperation, it seems.
Same error.
If I take the non-working web test and click Generate Code it produces a TestNameCoded.cs file in the root of the test project which does exactly what the webtest does. If I run that, it works fine.
Why do Visual Studio Web Tests ever have problems loading their own assembly? How can I get this web test to work without having to generate (and maintain) a coded test?
I have a big code base in TFS which has multiple .sln files, each with many projects and at least one unit test project. Most of the unit tests rely on common XML and XSD files, and there are several other types of files (.config, .xaml, etc) that are needed by the code when unit testing.
Because of the way that TFS builds and gathers the files for unit testing, most of those files are missing from the TestResults folder, so the tests are failing during our CI builds [this has been happening for a while, but I'm new to the project, and am trying to fix the errors]. What TFS appears to do is this: First, it checks out all the code to a src folder (with Solution1, Solution2, etc) and builds it, just like the developers do locally. Second, it copies the build outputs to a bin\Binaries folder. Third, it looks for all the test.dll files, copies them and their dependencies (but only the dependencies), plus the App.config file to TestResults\Deploy_[date/time]\Out folder, and runs the unit tests there.
I am encountering two problems with this. Because the second step is combining all the build outputs into one folder, all the files with duplicate names are overwriting each other. So, there is only one App.config file, even though each solution has its own. This is happening with other config/xml files too, and with two poorly named unit test .dlls. I can live with this if I have to because most of those config files are duplicates, and other files can be renamed.
The second problem is that most of those extra files don't make it into the TestResults folder, and when they aren't there the unit tests will fail. I know about using the [DeploymentItem] attribute; if that is the only solution, I will go that route, but there are so many extra files that I am looking for a different approach.
So my question is, how can I configure my tfs-run builds & unit tests to include all of the files that they need, without all the work & maintenance problems of adding a lot of [DeploymentItem] attributes, and also without affecting the local builds & unit tests for the developers?
Update
One thing I've found is that adding a [DeploymentItem] attribute to a unit test actually causes the deployment folder to be used. Without that, it runs the unit tests in the binaries folder. See http://msdn.microsoft.com/en-us/library/ms182475.aspx, under "When is a separate deployment folder used?"
Also on that page, it says you can specify files to deploy in a .testsettings file, but then says you should avoid using it because your tests will run slower. The newer alternative, a .runsettings file, does not let you list what files to deploy.
It also appears that deployment becomes enabled if code coverage is enabled, which we don't currently do, but plan to once the tests are passing.
In TFS 2013 you can execute a powershell pre-test to organize the files in a way that you need. You can get files to be deployed ad part of the tests with a test settings file for pre2013.
If however the test setting file is not enough you can use the Community Build tools to call a powershell directly in previous versions to 2013.
If you are stuck on 2012 then you will need to use the .testsettings file to push the bits you need. Yes it will make your build slower but that's your only choice other than customising the build process as above.
I have just created a small project with AppHarbor, composed of a single .csproj that uses xUnit. It's using NuGet to recover the xUnit framework. AppHarbor recognized it was a test project and executed the tests that were present.
I want to know how AppHarbor select which projects to unit test. Does it look at the referenced assemblies of the dll/exe (perhaps through reflection)? (for example if the dll/exe references xunit.dll then it's probably a test project). Does it look at the .csproj file looking for the references? Does it run all the assemblies through all the runners?
Appharbor uses Galio http://www.gallio.org/ I'm pretty sure its going to run everything that is marked with [Fact]
Using Visual Studio 2010 C++. I'm experimenting with unit testing and decided to try Google Test (gtest). I have an existing project which compiles to an MFC executable (I'm also interested in how to test a project that compiles to a DLL). My understanding of the convention for unit testing is that you should create a new separate project for your tests. So I created a new project in the same solution for my unit tests. But how do I link the projects? Can I test arbitrary functions/methods of my exe project from my test project?
What is the conventional way to do this?
I think the best way to organize unitary test is:
Don't change your main project. The structure should be independent of your test actions. In my opinion, changing your project to a big static lib AND an executable is really not elegant.
Instead, add a post build action to aggregate all obj files into a static lib file that will be used ONLY by your test project.
Create a simple test project, linking to your test framework AND the static library you have previously generated.
Enjoy.
The main advantages is you don't touch the project you want to test and you don't include all source code to your test project.
To see how you can do that for visual studio, you can see this post: Linking to multiple .obj for unit testing a console application
Either put the functionality you want to test into a static library which is linked into both your test project and your MFC project, or put your files in both projects. The first is more complicated, but the second will cause you to compile everything twice....
I have prepared a github repo including Visual Studio 2015 solution in parralel of Billy's answer. You can use it directly without any additional requirement or dependency.
https://github.com/fuatcoskun/GoogleTestVS2015
I hope it helps...
We develop a C++ application using Visual Studio 2008 and unit test using Boost.Test. At the moment, we have a separate solution which contains our unit tests.
Many of our projects in the core solution produce DLL's. We're limited in test coverage because we cannot test non-exported classes.
I have two ideas on how these could be tested:
Export everything
Put the tests inside the DLL (same project and solution) and use Boost.Test's external runner
I'm not entirely sure what the drawbacks would be. Number 1 above breaks module level encapsulation, and number 2 could result in a much larger DLL, unless it's possible to only include the test code in certain configurations.
So, are there any severe drawbacks to the above methods, or can you think of other solutions?
Expanding on Tom Quarendon's answer to this question, I have used a slight variant of Simon Steele's response:
Create a test project (using whatever test framework you like, I use CppUnit).
In your test_case.cpp, #include <header/in/source/project.h>.
In the test project properties:
In Linker->General, add the source project's $(IntDir) to the Additional Library Directories.
In Linker->Input, add the .obj files to the Additional Dependencies.
Add the dependency from the test project to the source project in Project->Project Dependencies.
Again, the only maintenance overhead is the standard one for unit tests - to create the dependency on the unit(s) you want to test.
The solution I use for this is to build the same non-exported code into my tests DLL as well. This does increase build time and means adding everything to both projects, but saves exporting everything or putting the tests in the main product code.
Another posibility would be to compile the non-exported code into a lib which is used by both the DLL with exports, and the unit test project.
Was searching a solution as well, maybe the following will be easier to maintain.
Add a new build configuration e.g. "Unit testing Debug" to the DLL project and change the Configuration Type to be "Static Library .lib" ("General"->"Configuration Type").
Then just add a dependency of your unit tests on this project, now everything should link together when you use new build configuration "Unit testing Debug".
If you are using release builds for unit tests then you need to add another configuration with release optimizations.
So the benefits of this solution are:
low maintanability cost
single DLL/Static library project
don't have to manually link to .obj files
Drawbacks:
Extra configuration profile(s) will require some changes in your build environment (CI)
Greater compilation times
Update:
We actually ended up using a different approach.
We added new "Test debug"/"Test release' configurations for every existing project that we have.
For .exe/.dll projects we disable the original main.cpp from compiling and replaced it with the one that instantiates the test framework (e.g. gtest) and runs all the tests, the tests are in separate .cpp files which are also excluded from compilation in regular configurations (Release/Debug) and enabled only in Test configurations.
For .lib projects we also have new "Test debug"/"Test release" configurations and there we convert the static library to be an .exe file and provide a main.cpp which instantiates the testing framework and runs the tests and tests themselves. Test related files are excluded from compilation on Release/Debug configurations.
Try making a define such as the following somewhere all files will include:
#define EXPORTTESTING __declspec(dllexport)
And use it in place of the dllexport, like this:
class EXPORTTESTING Foo
{
...
};
Then you will be able to turn off the flag for building a release DLL, but keep it on for a unit-testable DLL.