I have just created a small project with AppHarbor, composed of a single .csproj that uses xUnit. It's using NuGet to recover the xUnit framework. AppHarbor recognized it was a test project and executed the tests that were present.
I want to know how AppHarbor select which projects to unit test. Does it look at the referenced assemblies of the dll/exe (perhaps through reflection)? (for example if the dll/exe references xunit.dll then it's probably a test project). Does it look at the .csproj file looking for the references? Does it run all the assemblies through all the runners?
Appharbor uses Galio http://www.gallio.org/ I'm pretty sure its going to run everything that is marked with [Fact]
Related
I have a big code base in TFS which has multiple .sln files, each with many projects and at least one unit test project. Most of the unit tests rely on common XML and XSD files, and there are several other types of files (.config, .xaml, etc) that are needed by the code when unit testing.
Because of the way that TFS builds and gathers the files for unit testing, most of those files are missing from the TestResults folder, so the tests are failing during our CI builds [this has been happening for a while, but I'm new to the project, and am trying to fix the errors]. What TFS appears to do is this: First, it checks out all the code to a src folder (with Solution1, Solution2, etc) and builds it, just like the developers do locally. Second, it copies the build outputs to a bin\Binaries folder. Third, it looks for all the test.dll files, copies them and their dependencies (but only the dependencies), plus the App.config file to TestResults\Deploy_[date/time]\Out folder, and runs the unit tests there.
I am encountering two problems with this. Because the second step is combining all the build outputs into one folder, all the files with duplicate names are overwriting each other. So, there is only one App.config file, even though each solution has its own. This is happening with other config/xml files too, and with two poorly named unit test .dlls. I can live with this if I have to because most of those config files are duplicates, and other files can be renamed.
The second problem is that most of those extra files don't make it into the TestResults folder, and when they aren't there the unit tests will fail. I know about using the [DeploymentItem] attribute; if that is the only solution, I will go that route, but there are so many extra files that I am looking for a different approach.
So my question is, how can I configure my tfs-run builds & unit tests to include all of the files that they need, without all the work & maintenance problems of adding a lot of [DeploymentItem] attributes, and also without affecting the local builds & unit tests for the developers?
Update
One thing I've found is that adding a [DeploymentItem] attribute to a unit test actually causes the deployment folder to be used. Without that, it runs the unit tests in the binaries folder. See http://msdn.microsoft.com/en-us/library/ms182475.aspx, under "When is a separate deployment folder used?"
Also on that page, it says you can specify files to deploy in a .testsettings file, but then says you should avoid using it because your tests will run slower. The newer alternative, a .runsettings file, does not let you list what files to deploy.
It also appears that deployment becomes enabled if code coverage is enabled, which we don't currently do, but plan to once the tests are passing.
In TFS 2013 you can execute a powershell pre-test to organize the files in a way that you need. You can get files to be deployed ad part of the tests with a test settings file for pre2013.
If however the test setting file is not enough you can use the Community Build tools to call a powershell directly in previous versions to 2013.
If you are stuck on 2012 then you will need to use the .testsettings file to push the bits you need. Yes it will make your build slower but that's your only choice other than customising the build process as above.
We had a unit test project (exe, using NUnitLite) that referenced the main application to unit test code within the main application (an exe).
This used to work, but now Xamarin is giving an error that we can't reference an exe project.
Why is this?
What alternatives do people suggest for unit testing application code? It seems wrong to have to move all application code from the exe to an assembly just for the purposes of unit testing.
It looks like the feature was accidentally removed :-( Someone must have overlooked that valid, if uncommon, scenario. Please file a bug report on Xamarin.Studio and request this feature to be *re*enabled.
Until then I see two workarounds:
Build/execute the tests from a script (e.g. a Makefile). You can automate this to require less steps than using the IDE;
Add a pre-build step to your project to make a copy of the .exe and name it .dll. Then reference this copy of the assembly.
We are using VS2012 and TFS2012 and write unit tests for our code. We want to report code coverage, and also using .config files in our unit tests for test appsettings, and also some other settings for logging, MS Enterprise library settings etc. etc.
App.config not working in new test framework
New test framework of MS should be great, but to me it is not so great at all.
How I'm i suppose to set some basic configuration in config files, when the new framework does not use config files anymore?
We had a problem with mixed mode dlls, and found a fix: adding
<startup useLegacyV2RuntimeActivationPolicy="true">
to the app.config. But this did not work for our unit test project. Becuase config files are not there anymore. Searching the internet came up with a solution
'Problems with .Net 2.0 Mixed Mode Assemblies inside Visual Studio .Net 4.5 Test Projects'
This means editing a file of Visual Studio 11 itself in the program files directory, not a great solution i think....
And how about some basic appsetting? How am I supposed to set this?
Do not use the .testSettings file
Using the old .testsettings file is also not recommended by MS, becuase then the old test framework is used. And if I use the .testsettings file, i cannot setup Code Coverage on my tfs2012 build service.
Another issue is that we have code that need a dll (system.data.sqlite.dll), but only at runtime the unit test code needs this dll. So a reference is not needed. We fixed this by using the Deployment tab on the testsettings file. But in the new framework, you should not use the testsettings file. You have the [deploymentitem] attribute if you need files. But the deploymentitem attribute can only be used on a [testmethod] not on a [testinitialize] or [assemblyinitialize] method. But our code needs the dll in the [testinitialize] method. So there is no way to get the dll in place.
Just copy it with File.Copy in the [assemblyinitialize] (or testinitialize) method does not work.
Adding the dll as file to the project, and set the 'copy to output directory' to 'Copy Always' as mentioned in 'Configuring Unit Tests by using a .runsettings File' also does not work at all.
The (really not great) solution for this is to add the dll as a reference, then instantiate a class and do nothting with it. This way the dll is needed otherwise it is not building, and thus the dll will deploy itself to the right directories.
how to solve my problem(s)???
- I want to use config files in my unit test.
- I want to deploy some files that are neede in the 'assemblyinitialise' and/or 'classinitialize' methods.
- I want Code Coverage on my TFS2012 nightly build enabled.
a) App.config not working in new test framework
This should still work. What I think is missing in this case is that this .config file is not being copied with your test dll. Could you please set this as a deployment item and try again?
b) Do not use the .testSettings file
.testsettings and code coverage.
Setting up code coverage with the .testsettings file IS still supported in VS 2012 build. You simply need to select the mstest 2010 test runner and specify your .testsettings file in your build definition
If you dont have anything except code coverage settings in the .testsettings file then you can easily migrate to the 2012 test runner and select "enable code coverage" in the drop down items
copying a file required by test initalize
You could do this via the .testsettings file or you can have a post-build file copy task. It is pretty straight forward to do so and has no impact on anything else. Using the "copy to output directory = copy always" does work. Please try it with a sample solution and see if you can narrow down on why this does not work on your setup.
I have some projects developed with C++ builder XE.
I would like to add some unit test, but the DUnit framework installed is nice for Delphi, but the integration with C++ builder is not so good (and very limited).
What other xUnit framework can I easily work with ?
In your case I'd start by asking Embarcadero for assistance. They want to fully support the developers who use their stuff, and automated unit testing is really critical to keeping them happy.
Until then, CppUnit works on any C++ code, but does not really integrate all that well with IDEs. The approach we've used is to create a new project to contain the tests, and have its linker include the path to the existing production project's .OBJ files. We set up a a project dependency so the test project depends on the production project.
In the Test project, we'll use different main.cpp files, one each for Debug and Release, and use conditionals to include/exclude the appropriate one from the Debug and Release builds.
For some "fake" integration, at least as far as running the tests go, in the DebugMain.cpp we'll load up the MFC TestRunner GUI, so the developer can click to select the tests they want to execute. In the ReleaseMain.cpp, we'll use the command line test runner, using the CompilerOutputter object which will let the build process know of success or failure. The output is also compatible with IDEs that interpret stuff like that, so you can click on a failed test report in the Output window, and the IDE takes you to the failing test assertion.
Long time viewer, first time questioner here. Using Unit Tests in a separate project, I can't figure out how to separate them at the release build?
Surely when I remove the reference to the Unit Test project, all the references to the Unit Test interfaces, won't be able to find the interfaces and will cause compiler errors?
Thanks for any help.
Edit: By separating I mean removing them on the release build.
Assuming .NET, your unit test project should have a reference to the production project - but not the other way round. Your production code shouldn't depend on your unit tests at all.
You should have built your two projects, so that the base code (That in which you are testing) has no dependencies on the unit test. It is the Unit Test that has the dependencies on your code.
From what you wrote, I gather you wrote your code in .NET? (i.e. Assemblies?). Your project dependencies should look like this.
// Business code
My_Project.dll
-> References
---> System.dll etc..
// Test code
My_Project_Test.dll
-> References
---> My_Project.dll
If you make your projects that way, than I think you won't have to do anything special in your release build configuration.