I have 80+ VS2010 solutions, each contains some unit test projects.
All solutions are merged into one big solution before build process.
All tests (or some test subset) are executed after the successful build.
I want to enable test code coverage for all tests to get exact code coverage for all output assemblies.
My question is: how to enable code coverage without using Visual Studio?
Note: I'm using TFS2010 (MSBuild) to build merged solution. VS 2010 Premium is installed on the build server. MSTest is used for test execution.
You can use JetBrain's TeamCity Professional. It is a CI server that supports executing unit tests and calculating code coverage. It is free for small installations.
I think you need to consider deploying a code coverage tool, see here for a comparison (provided you implement .net).We use NCover, which integrated in our TFS-Build in it's console variant and, although it's not trivial to set it up, we 're very satisfied with it.In this post I had briefly described how we inserted NCoverin our build, this might we useful to you even if you go with another tool.
If you create a Vsmdi file in your large solution (ms test will usually do this for you) you can use this to tell the build which assemblies you want to instrument.
This will only provide code coverage for assemblies that have tests run against them. If you're using testrun.config files to decide which tests you want to run, this should be all you need. The code coverage results should then be published to the build drop location
Edit:
This blog post looks like it covers setting up code coverage
Related
I am somewhat new to TeamCity. I have set it up for a Visual Studio solution.
I am using NUnit step to run unit tests and dotCover for code coverage. Problem is, there are particular parts of my code that I know are covered by unit tests but in the code coverage report, it shows them with zero coverage.
I know the DLLs are created just fine because I run a dir /s command and can see all the test DLLs.
The NUnit step uses MSIL 4.0 platform and the test path looks like this:
**/bin/Debug/*Tests.dll
There are no filters under code coverage.
Many tests are run. But there are some specific tests that are not present in the search report when I search.
Why is this happening and how do I fix it?
Can you please provide information about your NUnit step, especially about .Net Coverage (Filters).
Can you also confirm that tests are actually run?
Full build log (pasted in pastebin.com or such provider) would be also appreciated.
Stupid me. The DLL that was being ignored was spelled *Test.dll instead of *Tests.dll. I use *Tests.dll in my test path.
The issue got fixed once I changed it to *Test*.dll
[I am very much new to the visual studio environment. After working with visual studio for 2-3 days I was able to get some knowledge about it.]
Well. I am working in a testing team. I just wanna setup a code coverage for cpp code developed in visual studio 2010. There are merely 2 doubts I wanna ask
1.My company has purchased visual studio 2010 ultimate edition and using that I was able to generate the code coverage for a sample test project (.dll) created using Test Project template in visual studio.
The problem here is that developers had written a test project using
cppunit as an .exe. So will I be able to get the code coverage for
an .exe too (with PROFILER enabled) using the usual method to get
the code coverage in visual studio ultimate edition. Or should I ask
them to write the unit tests using the Microsoft's unit test
framework(ie .dll). I just cant simply say do it, I need to present
them the genuine reason to do so.
2.The second thought I had was, there are lot more open source tools to get code coverage for cpp compiled using gcc or g++. So I thought of compiling the cpp code using g++ and use the open source tools to get code coverage. Is this a good idea?
When I shared this idea with a senior developer he said there are lot
more cpp code designed to be MSDN specific, you cant do by this way.If
you wanna do so you should make the code cross platform compatible or
portable[something I couldn't understand].
So here is the final thing, I am trying to present them a immediate solution of getting code coverage using the visual studio ultimate edition which we have. And later on search for the best way to achieve this without purchasing any paid tools like ncover, squish coco, bullseye coverage, etc only so I opted for the above mentioned second option.
My ultimate aim is to display the final report in the Jenkins so that everyone is able to see it without any manual intervention after the build process [which is not possible using the visual studio I guess,].
There is a plugin called MSTestRunner plugin for jenkins which can generate unit tests report but not the coverage report.
I think I made it clear. If this is a duplicate question please mention me the link to.Thanks in advance
I forgot to say this I was also able to find some tool called vcxproj2cmake I think it is clear from the name itself. So if i use this tool to generate a cmake corresponding to the actual vcxproj will i be able to the my second choice?
NOTE:
As for as now, I created a test project (lets say, SampleTestProject) as said in method 1 which has
only one test method (lest say, TestMethod) which in turn triggers all the unit tests written
using cppunit by the below mentioned method.
I asked the devs to make the .exe (which they have already with cppunit unit tests written) as .dll in a specific format like it
should contain a class called TestInitiator which contains the
CppUnit::TextUI::TestRunner (the runner instance which triggers the
cppunit tests, ideally this method should be present in order to trigger cppunit tests) with all the test methods defined in a suite.
This method is called by the above mentioned TestMethod in the SampleTestProject and thereby obtaining the code coverage
By this way the code coverage was obtained with very less change done to the code.Hope you like it!
I am just starting to work with QuantLib project. There are quite a few BOOST unit tests defined, hence the question: is there an easy way to run these tests selectively from Visual Studio (and see results)?
I've only used unit testing in .Net languages so far, and there you can either use the built-in test runner if you use MsTest, or some add-on (I used Resharper and TestDriven.NET) for other frameworks. There you can choose which tests you run, and you can run them with the debugger attached (which is unfortunately useful sometimes). Are there similar add-ons for BOOST tests in unmanaged C++?
Yes you can!
If your boost unit test project is defined correctly, then you can run the entire test project from VS10 debugger by clicking
right click on project --> Debug --> Start new instance
This will run the tests inside VS framework which will give you all the debugger goodies.
For running specific test suites/cases, just edit the project command line arguments under
right click on project --> Properties --> General --> Debugging
and add the parameters according to the links in the other answers
you can run any test case or test suite via command line. running tests by name
it even allows wildcards.
ReSharper C++ supports discovery and selective running/debugging of Boost tests (with Boost 1.59 and later).
I'm using the Boost Unit Test Adapter. It's free and has a very nice output. It can also list the time the single tests consumed and you can start single tests and group of tests.
You can download it here:
https://visualstudiogallery.msdn.microsoft.com/5f4ae1bd-b769-410e-8238-fb30beda987f?SRC=VSIDE
In our solution we have several test projects. Most of them contain unit tests that run fast but some are for integration, acceptance and UI tests which take quite some time to finish.
Now I'd like to have a convenient way of running only the unit tests. Either by using the hotkey to run all or using the new feature to run tests after each build.
Unfortunately VS also runs all the slow tests as well. In VS10 I grouped the tests in Test View by project and selected only the ones containing unit tests. It's not as easy as I wished it to be but at least it was possible. Now there doesn't seem to be a way around running all tests in a solution.
The only alternative that I see at the moment is to remove all but the unit tests from the solution but this would limit the refactoring capabilities.
So does anyone know how to easily group and run only unit tests in the new VS?
In Visual Studio 2012 Update 1 CTP, Test Explorer has added the option to group tests by Project or Traits (Categories).
Please go to this link to see what's new there.
You can use the search filter in Unit Text Explorer.
In the new Test Exlorer shipped with 2012, you can group tests by 'Test Results', 'FileName',... etc but in RTM build group/search by 'TestCategory' is not present from Test Explorer. However there is a filter in commandline and TeamBuild based on TestCategory (Running selective unit tests in VS 2012 RC using TestCaseFilter) in VS2012.
From a member of VS dev Team, they understood the importance of this missing feature. It is in the backlog, and it should be available in furutre builds/updates.
The following blog provides more details: http://blogs.msdn.com/b/vikramagrawal/archive/2012/07/23/running-selective-unit-tests-in-vs-2012-rc-using-testcasefilter.aspx
As of Visual Studio 2012 Update 2 (currently on CTP) now we are able to run a subset of unit test using "Playlists" or based on "Lists & Classes". More information here: http://blogs.msdn.com/b/visualstudioalm/archive/2013/01/30/first-ctp-for-visual-studio-update-2.aspx#playlists
Original Answer:
AFAIK, The new unit testing framework in VS2012 "supports selective execution based on filtering condition through TestCaseFilter". Unfortunately, currently there is not support to do this through the user interface but it is in the product group's backlog. You can use this approach through Test Build or command line. Please refer to the following blog: http://blogs.msdn.com/b/vikramagrawal/archive/2012/07/23/running-selective-unit-tests-in-vs-2012-rc-using-testcasefilter.aspx
EDIT: Following the MSDN documentation to use test categories to group tests.
http://msdn.microsoft.com/en-us/library/dd286595.aspx
Also, refer to this question: How to exclude certain tests in the Visual Studio 2012 Test Runner?
It shows an example of filtering based on the test path.
I'd recommend using TestDriven.Net. Group all Unit Testing projects in one solution folder, and all other tests in another. Then just right click the "Unit Tests" folder and choose "Run All Tests". I always favor this great test runner of the VS native testing tools.
Another approach is using NCrunch - It runs all you tests in the background while you code. It handles integration tests pretty well, as they run in parallel, in the background so it doesn't block you (you also choose which test projects are run). One catch though - I believe it doesn't support VS2012 yet, but probably soon.
I was also missing this feature... (We did the same as you did: grouping by project or sometimes by namespace). But I found that I could somehow workaround this using the test explorer search capabilities:
https://msmvps.com/blogs/deborahk/archive/2012/09/16/searching-and-grouping-in-vs-2012-test-explorer.aspx
In my case I could identify the specific tests I needed to run using the search filter on FullName (use dropdown on search box, or type FullName:"UnitTest")
Its not the same, but it works for me
From reasons I won't get into, all our unit tests are using the VSTS test framework. I now want to create an MSBuild script that runs the tests, but I don't want to use mstest.exe from various reasons (it's slower, requires Visual Studio installation everywhere, I need to maintain testrunconfig, etc.)
I've seen that TestDriven.net and TeamCity are able to run VSTS tests 'NUnit style', without using mstest.exe. Are you aware of any standalone command line utility that does this?
You can execute Team System Tests (MSTest) in NUnit if you use a special NUnit Addin that recognizes the MS Test Attributes (TestClass, etc).
Exact Magic Software has an open-source "test-adapter" that can do this.
UPDATE: I've reworked Exact Magic's Msts NUnit Adapter for NUnit 2.5.2.
Download here: http://snippetware.googlecode.com/files/ExactMagic.MstsAdapter.zip
Read more about it here: http://www.bryancook.net/2009/10/mstest-nunit-adapter.html
It seems like TeamCity is simply leveraging Gallio to run VS tests. Gallio appears to have msbuild integration and sounds perfect but after a closer look it seems that it would require a VS install just like MSTest as it appears to depend on MS exes:
The plugin enable condition was not satisfied: '${process:DEVENV.EXE} or
${process:VSTESTHOST.EXE} or
${process:QTAGENT.EXE} or
${process:QTAGENT32.EXE} or
${process:QTDCAGENT.EXE} or
${process:QTDCAGENT32.EXE}'.
Host process exited with code: 0
That being said it sounds like at least one person has got it working:
Christoph De Baene - Running MSTest without Visual Studio
It is possible to run MSTests without installing Visual Studio. See how-do-i-use-mstest-without-visual-studio.
I did this so that I could run my tests as part of my CI process. (I am using CC.NET for my CI solution).
I am in a similar situation as you, in that I want to use TestDriven.NET to get code coverage stats. But, I am running into problems. My first problem is that I am using AssemblyInitialize attributes to initialize a database connection. This isn't supported by NUnit so about half of my tests fail whereas they run fine under MSTest.
So, it seems that translating tests from one test framework to another has pitfalls. If you are aware of that, then go forth, but it might be better to try and keep consistent on one test framework.
We run VSTS tests using msbuild TestToolsTask on a Cruise Control server. This does not use the MSTEST executable -- the condition you ask for -- but does use a variety of TFS dependencies.
Note that we are migrating tests off of the VSTS test framework for NUnit, mostly because we can create extensions for NUnit to perform useful tasks.