I am looking for any and all suggestions of the best and effective resources that the StackOverflow community has used to better learn MSBuild with an emphasis on integrating unit tests and later static code analysis tools such as FxCop and StyleCop into the build process.
I have tried to find good clear documentation on adding unit tests into my build but I still am searching - even Google searches have come up empty or with just bits and pieces. Ideally I want to add unit tests, report results, and eventually add code coverage statistics, etc into the build results.
I know it has to be in MSDN somewhere but I seem unable to find anything which explains and teaches well. I am using Visual Studio Team System 2008.
Continuous Integration From Theory to Practice by Carel Lotz. It covers the entire scope of your problem, and then some. Well written, complete, and a full sample are all there.
Hands down best resource. Use it as a tutorial first, then use it as a guide, then use it as a reference.
MSDN and others are good for clarifying (or confusing) the details.
Edit: The guide by Carel Lotz uses MBUnit for unit tests (see his earlier document version for NUnit, though you can replace the MBUnit with NUnit pretty easily if you follow the NUnit help files).
Also, it is written to use Cruise Control.NET to run the MSBuild script in various configurations.
Personally, I run unit tests in a secondary MSBuild script, but have found that wrapping the NUnit calls in MSBuild gives more flexibility than running from CCNet directly.
Here's a book that might help: http://blogs.msdn.com/microsoft_press/archive/2009/01/31/sayed.aspx
I guess I need to ask if you are sure you want to use MSBuild directly? Might want to check out WIX as the MSI producing tool - there is an extensive manual and it is built on top of MSBuild.
As for the automating of your tests with reporting and integrating with NUnit, FxCop, NCover, FitNesse, etc - I think the best (free) tool out there is CruiseControl .Net. It works with all of these tools and more. Can do versioning, automated builds with automated testing, creates the reports for each...
Here is a sample of one of my builds...
http://img84.imageshack.us/img84/3664/cruisecontrolnetsamplezn0.jpg http://img84.imageshack.us/img84/3664/cruisecontrolnetsamplezn0.jpg
Related
The company I'm working at uses xUnit to write Integration tests. xUnit works perfectly for us but we will like to extract more statistical information out of runs. For example - "How many times did this specific Test-Case has been failing in the last month", and maybe even sort it out nicely on a pie chart.
Since we have a Microsoft Test Manager license, I did some research on it, and it seems like it does support more detailed reports. I also like the coupling between Manual Test-Cases to Automation Test-Cases, and the fact you easily identify how much of your Test-Cases are automated.
Sadly enough, Test Manager only support MSTest integration out of the box. I did noticed however that the MSTest.ext alternative - VSTest.exe is able to run xUnit tests, and even output TRX result file. Is there any way to integrate xUnit (or nUnit) to the Test Manager somehow? Has anyone done so in the past? we prefer to use Test Manager, but I'm interesting to know if there is an alternative that support a couple Test-Cases with Automated Test-Cases and a way to get statistical information about multiple-runs.
Thank you.
This tool will allow you to associate NUnit and xUnit test cases with Microsoft Test Manager.
https://github.com/JakeGinnivan/TestCaseAutomationAssigner
After some research on the Microsoft Forum (and personally speaking with Microsoft representative), It seems like it's not possible to use xUnit with MS Test Manager.
We decided not to use MS Test Manager and handle all our test runs using VSTest.exe and xUnit categories.
Edit: It's now possible to use an external tool for that called "TestCaseAutomationAssigner". See Jeff's answer for more information.
I am beginning a fairly large new project using ColdFusion. This new project will include several developers and as such documentation of code will be key.
Another issue I am hoping to avoid (either with adequate code documentation or some other tool) is the duplication of code. A tool that would be able to "index" the code for searching or diagramming would likely help here.
What are others out there using either specifically for ColdFusion or language agnostic. We will likely be using ColdBox for the underlying framework if that makes a difference.
Thanks for any any all suggestions.
-c
Well, it's impossible to tell you which framework to use without knowing more about your project, but I can list out some tools that will be useful no matter which framework you use.
Language-agnostic tools:
GitHub.com organization+teams
Jenkins continuous integration
Apache ANT build scripts
Apache Maven for project management
Coldfusion-specific tools:
MXunit unit testing framework
MockBox for unit testing (if you use ColdBox: ColdBox-specific tutorial)
ColdDoc documentation generator
Javascript-specific tools:
JSLint or JSHint for JS code cleanup
Jasmine unit testing
Ideally, your Jenkins build server should:
Do a fresh checkout from source control
Run all unit tests and stop the build if they fail
Generate documentation
Generate a production-ready package of your project
At a minimum, I highly recommend using source control, setting up Jenkins with MXunit tests, and scheduling daily automated builds.
We used the Atlassian suite at my last job. Mostly Jira for tracking and Greenhopper for agile but the other tools may help, fisheye, bamboo, and crucible. If you host it yourself I believe they have a one time $10/product price tag that. Depending on your team's size may or may not work. If money is no subject the suite worked really nicely. It also has built in support for svn and maybe more by now.
http://www.atlassian.com/software
Sounds to me what you need is a methodology, not a tool. If you have a clearly defined set of objects/responsibilities. There should be no crossover in scripting, and if you determine a common API for the objects being coded, I would think you'd be fine.
I've been using MSTest so far for my unit-tests, and found that it would sometimes randomly break my builds for no reason. The builds would fail in VS but compile fine in MSBuild - with error like 'option strict does not allow IFoo to cast to type IFoo'. I believe I have finally fixed it, but after the bug coming back and struggling to make it go away again, and little help from MS, it left a bad taste in my mouth. I also noticed when looking at this forum and other blogs and such, that most people are using NUnit, xUnit, or MBUnit.. We are on VS2008 at work BTW.. So now I am looking to explore other options..
I'm working on moving our team to start doing TDD and real unit testing and have some training planned, but first would like to come up with a set of standard tools & best practices. To this end I've been looking online to come up with the right infrastructure for both a build server and dev machines...I was looking at the typemock website as I've heard great things about their mocking framework, and noticed that it seems like they promote MSTest, and even have some links of people moving TO MSTest from NUnit..
This is making me re-think my decision.. so I guess I'm asking - is anyone using MSTest as part of their TDD infrastructure? Any known limitiations it has, if I want to integrate with a build / CI server, or code coverage or any other kind of TDD tool I may need? I did search these forums and mostly find people comparing the 3rd party frameworks to eachother and not even giving MSTest much of a chance... Is there a good reason why.. ?
Thanks for the advice
EDIT: Thanks to the replies in this thread, I've confirmed MSTest works for my purposes and integreated gracefully with CI tools and build servers.
But does anyone have any experience with FinalBuilder?? This is the tool that I'd like us to use for the build scripts to prevent having to write a ton of XML compared to other build tools. Any limitiations here that I should be aware of before committing to MS Test?
I should also note - we are using VSS =(. I'm hoping we can ax this soon - hopefully as part of, maybe even the first step, of setting up all of this infrastructure.
At Safewhere we currently use MSTest for TDD, and it works out okay.
Personally, I love the IDE integration, but dislike the API. If it ever becomes possible to integrate xUnit.NET with the VS test runner, we will migrate very soon thereafter.
At least with TFS, MSTest works pretty well as part of our CI.
All in all I find that MSTest works adequately for me, but I don't cling to it.
If you are evaluating mock libraries, take a look at this comparison.
I've been using MS Test since VS 2008 came out, but I haven't managed to strong-arm anything like TDD or CI here at work, although I've messed with Cruise Control a little in an attempt to build a CI server on my local box.
In general I've found MS Test to be pretty decent for testing locally, but there are some pain points for institutional use.
First, MS Test adds quite a few things that probably don't belong in source control. The .VSMDI files are particularly annoying; just running MS Test creates anywhere from 1 to 5 of them and adds them to the solution file. Which means churn on your .SLN in source control, and churn of that sort is bad.
I understand the supposed point behind these extra files -- tracking test run history and such -- but I don't find them particularly useful for anything but a single developer. You should use your build service and CI for that sort of thing!
Second, you either must have Team Foundation Server to run your unit tests as part of CI, or you have to have a copy of Visual Studio installed on your build server if you use, for example, Cruise Control.NET. See this Stack Overflow question for details.
In general, there's nothing wrong with MS Test. But going CI will not be as smooth as it could be.
I have been using MSTest very successfully in our company. We are currently setting up standardised build processes within our company and so far, we have had good success with TeamCity. For Continuous integration, we use out the box TeamCity configurations. For the actual release builds, we set up large msbuild scripts that automate the entire process.
I really like mstest because of the IDE integration and also that all our devs automatically can use it without installing any 3rd party dependencies. I would not recommend switching just because of the problem you are experiencing. I have come full circle, where we went over to nunit and then came back again. These frameworks are all the same at the end of the day so pick the one that is easiest for most your devs to get access to and start using.
What I suspect your problem might be... sounds like an obscure problem I have had before where incorrect references of dll's (eg: adding explicit references (via browse) to projects in your solution, and not using the project reference) leads to out-of-date problems that only come up after clean checkouts or builds.
The other really suspect issue that I have found before is if you have some visual component or control that has a public property of some custom type that is being serialised in the forms .resx file. I typically need to flag them with an attribute that says SerializationVisibility.Hidden. This means that the IDE will not try to generate setters for the property value (which is typically some object graph). Just a thought. Could be way out.
I trust the tools and they don't really lie about there being a genuine problem. They only misrepresent them or report them as something completely obscure. It sounds to me like you have this. I suspect this because the error message doesn't make sense if all is in order, but it does make sense if some piece of code has loaded up an out of date or modified version of the dll at that point.
I have successfully deployed several FinalBuilder installations and the customers have been very happy with the outcome. I can highly recommend it.
I've recently decided to start using some light unit testing to see if it adds any value to our project, but I'm having trouble finding documentation for SimplyVBUnit. Any suggestions?
There's some discussion of VB6 unit testing in this question, with Gutzofter's mini tutorial on SimplyVBUnit.
If you downloaded the binary installer for SimplyVBUnit it comes with a couple of example projects in the installation folder. Both the source code zip file and binary installer include a Help.chm showing many of the classes and how to apply them in your tests.
Have you considered collecting test coverage? This determines what part of your code has run, often as exercised by your unit tests. To do this, you need a test coverage data collection tool, as well as your tests.
EDIT: The only VB6 test coverage tool I know is http://www.aivosto.com/vbwatch.html
Aivisto seems to have a generally good reputation for thier VB tools.
I've spent 4 years developing C++ using Visual Studio 2008 for a commercial company; it's now time for me to upgrade my development process.
Here's the problem: I dont have a 1 button build automation. I also dont have a CI server that automatically builds when a commit happens, and emails me whether a build is broken or not. Worse we dont even have a single unit test!!
Can someone please point to me how I can get started?
I have looked at many many tools and I think I might go with:
Visual Build (for build automation) (Note: I also considered Final Builder)
Cruise (for CI server)
I also now am just starting to practice TDD...so I will want to automate my unit tests as well. I chose Google Test/Mock for their extensive documentation. (Cant go wrong with Google brand can I? =p)
Price is not the issue, I want what's best and easiest to get started.
Can people that use real CI/automation tool for unmanaged MSVC++ tell me their tools and how I can go about starting?
Our source control is Subversion.
Last point: I'm also considering project management/tracking tool that integrates right into VSTD ..and thinking about using OnTime. VSTS costs too much. I tried FogBugz, but I think it's too simple. Any others?
I would take some time to seriously consider TeamCity. We used CruiseControl.NET for a while and TeamCity completely demolishes it. Plus it has built-in plugins for Boost and CppUnit, so your unit testing will come for free.
Best of all, the tool is free for < 20 users and gives you three build agents.
I just finished implementing our C++ product at work and it was fairly simple. We did it with msbuild and basically use the msbuild task to compile the solution. Other targets can be used to copy files, run unit tests, etc.
The last time I worked on an unmanaged MSVC++ project (which was moderately sized I might add), we used FinalBuilder to do the automated build & versioning (and even executing PCLint and other profiling tools as well).
Having said that, if you're willing to invest the time, MSBuild (or nAnt perhaps?) can do everything you need - even for unmanaged solutions.
Which brings us to the trade-off: Tools like Visual Build Pro and Final Builder get you up and running quickly. If you want something which offers a greater range of customization, you'll probably be spending a decent amount of time learning and understanding it - i.e. MSBuild, CIFactory, nAnt etc are no cake walk.
So if price isn't an issue - is time an issue? If time is at a premium, I'd investigate the GUI driven tools, they'll get you to where you want to go quickly. If you know you're going to need to extend on the simple one button build + unit tests + deploy scenario (which happens a lot!) then decide if you can invest the time into the more complex tools like MSBuild?
We use a combination of Boost.Build, NAnt, CPPUnit and either Cruise Control.NET or Hudson (we've used them both for various projects but are starting to prefer Hudson).
They are all good tools though we're considering replacing CPPUnit - the Google unit test system is pretty good from what I've seen.
If you're happy running on just Windows you can lose Boost.Build and just call out to Visual Studio from NAnt.
As for issue tracking/project management we settled on Vision Project after a long investigation. It's not well known (yet) but we've found it a very good fit in our environment. Fogbugz is great, a nice, clear interface but we came to the conclusion you did too; way too simple for our needs.
Although the .NET world is spoilt for these kinds of tools Continuous Integration is still pretty easy to set up for C++! I wouldn't think of starting a non-trivial project without putting these systems in place.
we use subversion + cruisecontrol + wix to accomplich CI automated builds outputting one-click installers. this combo has worked very well for us. we've created out own site for admin of svn user groups and permissioning and added the web interface to cc to it. we have a sql server storing all the collected stats from svn and cc and use them for custom reports available on our site. we are looking to add other tools to the mix for checking various attributes of the code stored in svn. this combo has worked very well for us.
At my company we use CruiseControl (http://cruisecontrol.sourceforge.net/). The Java version, not .NET, to build our wxWidgets application on Windows and OS X. Working great for us so far.