I am using gtest for testing my code in C++ with Visual studio 2010. But I could not able to makeout that I have reached 100% code coverage. To make sure that I have covered 100% code coverage, I would like to know that, is there any way to find out the code coverage gtest or not? Because I have Googled a lot but I did not find any possible way to get the code coverage result by using gtest in Windows enviornment. If it is possible please let me know.
Thanks A Lot..
You can try OpenCppCoverage: https://github.com/OpenCppCoverage/OpenCppCoverage.
It is open source, designed for Visual Studio C++ and works well with Google Test.
I already used successfully for mid size project (~500 000 lines of code).
Hope that help.
Disclaimer: I am the author of the tool.
Code coverage in C++ can not be handled by the testing framework solely, because a coverage analysis tool has to know the whole extent of the code (wich the testing framework has not) and it has to instrument the code under test somehow to monitor wich parts of the code get executed.
I had the same desire like you once, wanting to measure my test coverage in MSVC. This is what I learned:
MSVC ships with some command line tools for these instrumentations, googling a bit will get you one or two msdn blog posts about how to use them. Frankly, its not very convenient and easy to use. If you look for third party tools, you will probably not find any free ones. Any tools I found at all were enterprise tools with license fees in the range of several hundred to more than a thousand dollars, so not really an option if you are not a company.
Related
I've been using MSTest so far for my unit-tests, and found that it would sometimes randomly break my builds for no reason. The builds would fail in VS but compile fine in MSBuild - with error like 'option strict does not allow IFoo to cast to type IFoo'. I believe I have finally fixed it, but after the bug coming back and struggling to make it go away again, and little help from MS, it left a bad taste in my mouth. I also noticed when looking at this forum and other blogs and such, that most people are using NUnit, xUnit, or MBUnit.. We are on VS2008 at work BTW.. So now I am looking to explore other options..
I'm working on moving our team to start doing TDD and real unit testing and have some training planned, but first would like to come up with a set of standard tools & best practices. To this end I've been looking online to come up with the right infrastructure for both a build server and dev machines...I was looking at the typemock website as I've heard great things about their mocking framework, and noticed that it seems like they promote MSTest, and even have some links of people moving TO MSTest from NUnit..
This is making me re-think my decision.. so I guess I'm asking - is anyone using MSTest as part of their TDD infrastructure? Any known limitiations it has, if I want to integrate with a build / CI server, or code coverage or any other kind of TDD tool I may need? I did search these forums and mostly find people comparing the 3rd party frameworks to eachother and not even giving MSTest much of a chance... Is there a good reason why.. ?
Thanks for the advice
EDIT: Thanks to the replies in this thread, I've confirmed MSTest works for my purposes and integreated gracefully with CI tools and build servers.
But does anyone have any experience with FinalBuilder?? This is the tool that I'd like us to use for the build scripts to prevent having to write a ton of XML compared to other build tools. Any limitiations here that I should be aware of before committing to MS Test?
I should also note - we are using VSS =(. I'm hoping we can ax this soon - hopefully as part of, maybe even the first step, of setting up all of this infrastructure.
At Safewhere we currently use MSTest for TDD, and it works out okay.
Personally, I love the IDE integration, but dislike the API. If it ever becomes possible to integrate xUnit.NET with the VS test runner, we will migrate very soon thereafter.
At least with TFS, MSTest works pretty well as part of our CI.
All in all I find that MSTest works adequately for me, but I don't cling to it.
If you are evaluating mock libraries, take a look at this comparison.
I've been using MS Test since VS 2008 came out, but I haven't managed to strong-arm anything like TDD or CI here at work, although I've messed with Cruise Control a little in an attempt to build a CI server on my local box.
In general I've found MS Test to be pretty decent for testing locally, but there are some pain points for institutional use.
First, MS Test adds quite a few things that probably don't belong in source control. The .VSMDI files are particularly annoying; just running MS Test creates anywhere from 1 to 5 of them and adds them to the solution file. Which means churn on your .SLN in source control, and churn of that sort is bad.
I understand the supposed point behind these extra files -- tracking test run history and such -- but I don't find them particularly useful for anything but a single developer. You should use your build service and CI for that sort of thing!
Second, you either must have Team Foundation Server to run your unit tests as part of CI, or you have to have a copy of Visual Studio installed on your build server if you use, for example, Cruise Control.NET. See this Stack Overflow question for details.
In general, there's nothing wrong with MS Test. But going CI will not be as smooth as it could be.
I have been using MSTest very successfully in our company. We are currently setting up standardised build processes within our company and so far, we have had good success with TeamCity. For Continuous integration, we use out the box TeamCity configurations. For the actual release builds, we set up large msbuild scripts that automate the entire process.
I really like mstest because of the IDE integration and also that all our devs automatically can use it without installing any 3rd party dependencies. I would not recommend switching just because of the problem you are experiencing. I have come full circle, where we went over to nunit and then came back again. These frameworks are all the same at the end of the day so pick the one that is easiest for most your devs to get access to and start using.
What I suspect your problem might be... sounds like an obscure problem I have had before where incorrect references of dll's (eg: adding explicit references (via browse) to projects in your solution, and not using the project reference) leads to out-of-date problems that only come up after clean checkouts or builds.
The other really suspect issue that I have found before is if you have some visual component or control that has a public property of some custom type that is being serialised in the forms .resx file. I typically need to flag them with an attribute that says SerializationVisibility.Hidden. This means that the IDE will not try to generate setters for the property value (which is typically some object graph). Just a thought. Could be way out.
I trust the tools and they don't really lie about there being a genuine problem. They only misrepresent them or report them as something completely obscure. It sounds to me like you have this. I suspect this because the error message doesn't make sense if all is in order, but it does make sense if some piece of code has loaded up an out of date or modified version of the dll at that point.
I have successfully deployed several FinalBuilder installations and the customers have been very happy with the outcome. I can highly recommend it.
I've recently decided to start using some light unit testing to see if it adds any value to our project, but I'm having trouble finding documentation for SimplyVBUnit. Any suggestions?
There's some discussion of VB6 unit testing in this question, with Gutzofter's mini tutorial on SimplyVBUnit.
If you downloaded the binary installer for SimplyVBUnit it comes with a couple of example projects in the installation folder. Both the source code zip file and binary installer include a Help.chm showing many of the classes and how to apply them in your tests.
Have you considered collecting test coverage? This determines what part of your code has run, often as exercised by your unit tests. To do this, you need a test coverage data collection tool, as well as your tests.
EDIT: The only VB6 test coverage tool I know is http://www.aivosto.com/vbwatch.html
Aivisto seems to have a generally good reputation for thier VB tools.
I've spent 4 years developing C++ using Visual Studio 2008 for a commercial company; it's now time for me to upgrade my development process.
Here's the problem: I dont have a 1 button build automation. I also dont have a CI server that automatically builds when a commit happens, and emails me whether a build is broken or not. Worse we dont even have a single unit test!!
Can someone please point to me how I can get started?
I have looked at many many tools and I think I might go with:
Visual Build (for build automation) (Note: I also considered Final Builder)
Cruise (for CI server)
I also now am just starting to practice TDD...so I will want to automate my unit tests as well. I chose Google Test/Mock for their extensive documentation. (Cant go wrong with Google brand can I? =p)
Price is not the issue, I want what's best and easiest to get started.
Can people that use real CI/automation tool for unmanaged MSVC++ tell me their tools and how I can go about starting?
Our source control is Subversion.
Last point: I'm also considering project management/tracking tool that integrates right into VSTD ..and thinking about using OnTime. VSTS costs too much. I tried FogBugz, but I think it's too simple. Any others?
I would take some time to seriously consider TeamCity. We used CruiseControl.NET for a while and TeamCity completely demolishes it. Plus it has built-in plugins for Boost and CppUnit, so your unit testing will come for free.
Best of all, the tool is free for < 20 users and gives you three build agents.
I just finished implementing our C++ product at work and it was fairly simple. We did it with msbuild and basically use the msbuild task to compile the solution. Other targets can be used to copy files, run unit tests, etc.
The last time I worked on an unmanaged MSVC++ project (which was moderately sized I might add), we used FinalBuilder to do the automated build & versioning (and even executing PCLint and other profiling tools as well).
Having said that, if you're willing to invest the time, MSBuild (or nAnt perhaps?) can do everything you need - even for unmanaged solutions.
Which brings us to the trade-off: Tools like Visual Build Pro and Final Builder get you up and running quickly. If you want something which offers a greater range of customization, you'll probably be spending a decent amount of time learning and understanding it - i.e. MSBuild, CIFactory, nAnt etc are no cake walk.
So if price isn't an issue - is time an issue? If time is at a premium, I'd investigate the GUI driven tools, they'll get you to where you want to go quickly. If you know you're going to need to extend on the simple one button build + unit tests + deploy scenario (which happens a lot!) then decide if you can invest the time into the more complex tools like MSBuild?
We use a combination of Boost.Build, NAnt, CPPUnit and either Cruise Control.NET or Hudson (we've used them both for various projects but are starting to prefer Hudson).
They are all good tools though we're considering replacing CPPUnit - the Google unit test system is pretty good from what I've seen.
If you're happy running on just Windows you can lose Boost.Build and just call out to Visual Studio from NAnt.
As for issue tracking/project management we settled on Vision Project after a long investigation. It's not well known (yet) but we've found it a very good fit in our environment. Fogbugz is great, a nice, clear interface but we came to the conclusion you did too; way too simple for our needs.
Although the .NET world is spoilt for these kinds of tools Continuous Integration is still pretty easy to set up for C++! I wouldn't think of starting a non-trivial project without putting these systems in place.
we use subversion + cruisecontrol + wix to accomplich CI automated builds outputting one-click installers. this combo has worked very well for us. we've created out own site for admin of svn user groups and permissioning and added the web interface to cc to it. we have a sql server storing all the collected stats from svn and cc and use them for custom reports available on our site. we are looking to add other tools to the mix for checking various attributes of the code stored in svn. this combo has worked very well for us.
At my company we use CruiseControl (http://cruisecontrol.sourceforge.net/). The Java version, not .NET, to build our wxWidgets application on Windows and OS X. Working great for us so far.
I am looking for any and all suggestions of the best and effective resources that the StackOverflow community has used to better learn MSBuild with an emphasis on integrating unit tests and later static code analysis tools such as FxCop and StyleCop into the build process.
I have tried to find good clear documentation on adding unit tests into my build but I still am searching - even Google searches have come up empty or with just bits and pieces. Ideally I want to add unit tests, report results, and eventually add code coverage statistics, etc into the build results.
I know it has to be in MSDN somewhere but I seem unable to find anything which explains and teaches well. I am using Visual Studio Team System 2008.
Continuous Integration From Theory to Practice by Carel Lotz. It covers the entire scope of your problem, and then some. Well written, complete, and a full sample are all there.
Hands down best resource. Use it as a tutorial first, then use it as a guide, then use it as a reference.
MSDN and others are good for clarifying (or confusing) the details.
Edit: The guide by Carel Lotz uses MBUnit for unit tests (see his earlier document version for NUnit, though you can replace the MBUnit with NUnit pretty easily if you follow the NUnit help files).
Also, it is written to use Cruise Control.NET to run the MSBuild script in various configurations.
Personally, I run unit tests in a secondary MSBuild script, but have found that wrapping the NUnit calls in MSBuild gives more flexibility than running from CCNet directly.
Here's a book that might help: http://blogs.msdn.com/microsoft_press/archive/2009/01/31/sayed.aspx
I guess I need to ask if you are sure you want to use MSBuild directly? Might want to check out WIX as the MSI producing tool - there is an extensive manual and it is built on top of MSBuild.
As for the automating of your tests with reporting and integrating with NUnit, FxCop, NCover, FitNesse, etc - I think the best (free) tool out there is CruiseControl .Net. It works with all of these tools and more. Can do versioning, automated builds with automated testing, creates the reports for each...
Here is a sample of one of my builds...
http://img84.imageshack.us/img84/3664/cruisecontrolnetsamplezn0.jpg http://img84.imageshack.us/img84/3664/cruisecontrolnetsamplezn0.jpg
So I'm interested in hearing different thoughts about what is the best way to go about unit testing XNA Game/Applications. Astute googlers can probably figure out why I'm asking, but I didn't want to bias the topic :-)
I would that this question is geared more toward the approach of unit testing in game development. I mean, XNA is a framework. Plug in NUnit, and begin writing test cases while you develop.
Here is a post on SO about unit testing a game. It'll give you a little insight into how you need to think while progressing.
XNA BOOK
This book shows how to code in XNA but the entire book is based on NUNIT testing. So while you are coding the projects in the book, he also shows you how to write the scripts for NUNIT to test the XNA code.
VS2008 has a nicely integrated unit testing framework. (I assume you're using using the XNA 3.0 CTP with your Zune.)
The Microsoft testing framework is now available in Visual Studio 2008 Professional and up. If you have this software you already have all the software that you need to start testing your games.
Here are two links that will get you started:
Unit tests Overview - http://msdn.microsoft.com/en-us/library/ms182516.aspx
Creating Unit Tests - http://msdn.microsoft.com/en-us/library/ms182523.aspx
If you only have the Visual Studio 2008 Express than you need to use some other testing framework. NUnit is probably the best one, some people even prefer it to MSTest.
After you have all the software you need you can start adding tests for your code.
Here I've posted some basic techniques about unit testing games that you might find useful.
Have you done unit testing before?
If you haven't I could posbily give you some more hints and resources
You should give Scurvy Test a try. Have not used it my self but it looks promising.
I know this is an old post, but for other people wondering how to best go about testing their XNA Games, there is another option. The built-in testing in Visual Studio is definitely great, but is not well suited for games. Everytime a value is needed, you have to pause the game, and then either hover over the variable, go to quick watch, or add a watch. Then you can see the value of the variable at that frame. To see the value again during another frame, you must pause your game, again. This can be a big hassle. Therefore I have created a debugging terminal to run on top of your game. It allows you to see values of variables, invoke methods, and even watch a variable change in real-time all while you game is running! To find out more, visit: http://www.protohacks.net/xna_debug_terminal/
The project is completely free and open source. If you like it, feel free to tell others using XNA Game Studio about it. Hopes this helps out!