I am trying to be a "good" programmer and have unit tests for my ColdFusion application but haven't been able to find a code coverage tool that can tie into the test that I'm using. For those of you who do unit tests on your ColdFusion code, how have you approached this problem?
With ColdFusion Builder you can use Rancho for code coverage.
http://forta.com/blog/index.cfm/2012/5/25/ColdFusion-Code-Coverage-With-Rancho
Many have asked, but to date there remains no ColdFusion code coverage tool.
The latest ColdFusion features .NET integration in addition to J2EE integration, so if you require code coverage metrics (ie, customer requirements) then (aside from choosing something other than ColdFusion) you might define what parts need such coverage and break them out into .NET or J2EE objects which you can profile with readily available tools.
Alternately, you can do some analysis yourself - use a parser to insert cftrace tags after every control statement and then parse the results.
Lastly, ColdFusion does have some basic stack trace ability - I don't know if it goes deeper than that, but it's worth exploring to see what you can parse from that.
I hope this helps!
-Adam Davis
Related
The company I'm working at uses xUnit to write Integration tests. xUnit works perfectly for us but we will like to extract more statistical information out of runs. For example - "How many times did this specific Test-Case has been failing in the last month", and maybe even sort it out nicely on a pie chart.
Since we have a Microsoft Test Manager license, I did some research on it, and it seems like it does support more detailed reports. I also like the coupling between Manual Test-Cases to Automation Test-Cases, and the fact you easily identify how much of your Test-Cases are automated.
Sadly enough, Test Manager only support MSTest integration out of the box. I did noticed however that the MSTest.ext alternative - VSTest.exe is able to run xUnit tests, and even output TRX result file. Is there any way to integrate xUnit (or nUnit) to the Test Manager somehow? Has anyone done so in the past? we prefer to use Test Manager, but I'm interesting to know if there is an alternative that support a couple Test-Cases with Automated Test-Cases and a way to get statistical information about multiple-runs.
Thank you.
This tool will allow you to associate NUnit and xUnit test cases with Microsoft Test Manager.
https://github.com/JakeGinnivan/TestCaseAutomationAssigner
After some research on the Microsoft Forum (and personally speaking with Microsoft representative), It seems like it's not possible to use xUnit with MS Test Manager.
We decided not to use MS Test Manager and handle all our test runs using VSTest.exe and xUnit categories.
Edit: It's now possible to use an external tool for that called "TestCaseAutomationAssigner". See Jeff's answer for more information.
So, testing with Sitecore. It's a special topic and I've found allot of reading material regarding it already. (Sitecore Development Chapter 8, Alistair Deneys blog, NextDigital blog, iStern blog,...) but in most of these cases they're going with NUnit and custom test runners etc... The most useful (to me in my context) this far was the iStern blog for mocking out Sitecore using Microsoft Fakes. But is this really the way to go?
I'm surprised that with Hedgehog TDS system to integrate so deeply with TFS and be able to do CI in Sitecore development that there isn't more on how to utilize this system for setting up solid testing executed by TFS (yet).
We're gearing up for a large new project now that uses Sitecore to handle front-end user interaction where the data being used is 95% behind a WCF service. So this part can be easily tested and TDD developed. It's the last 5% (which sadly includes like the highest business value, being online payments) that resides within Sitecore that needs to be tested. Can we ever have enough intimate knowledge of sitecore to mock it out? I'd be inclined to think not... if so, how then do we run conclusive tests on our TFS CI build against sitecore?
Last but not least I get the feeling that the information currently to be found is getting a bit out of date perhaps (maily seeing the remarks on the NextDigital blog), does Sitecore 7 open new ways to tackle this issue?
For those who'd see this more as a philosophical rather then a technical question: There can only be one answer to this and that is a technical accurate definition of a method of using the Microsoft test framework that is capable of running in the TFS CI environment to test code written for Sitecore.
Is Microsoft Fakes the way to go? In my opinion, no. Microsoft fakes allows you to test code that is not designed to be testable. If you design you solution properly, a standard mocking framework should be sufficient.
Can we ever have enough intimate knowledge of sitecore to mock it out? This is kind of a trick question. Unless a third-party library was specifically designed for it and is something that you would consider a "stable dependency", you shouldn't try to mock it. Instead, wrap it with your own classes and abstractions and mock those.
Take a look at Synthesis and Glass Mapper. They are object-mapping frameworks that allow you to map Sitecore items to interfaces while maintaining page editor support. Glass, in particular has a wrapper around Sitecore.Context that can be mocked. Synthesis is supposed to be pretty testable as well, but I haven't tried it yet.
Using one of those mapping frameworks and a good SOLID design, you should be able to make most of your code testable. Just remember that the classes on the edges of your solution should be simple enough to not require testing.
I was in the exact same situation as you, IvanL, a few weeks ago. I wanted to test some of my business logic running against Sitecore 7 without a mocking framework. I managed to do it, but only in a very specific scenario. Unfortunately, I haven't published my prototype solution or the slides explaining it yet, but I'll explain the basics of what I did.
In Sitecore 7, the move towards querying against the index with the Sitecore.ContentSearch namespace and using LINQ opened up a way for me to very easily unit test with fake index data.
There are some unit test examples out there, as you've seen, that use mocking frameworks. However, the classes they mock are actually quite simple to fake out yourself. If you implement ISearchIndex, you really only need to implement the CreateSearchContext method in order to start returning an IQueryable to work with in your tests.
To implement CreateSearchContext, you will likely need to create a fake provider search context implementation that will do the GetQueryable implementation.
Once you have those two classes set up, you've essentially got your 'index' covered. Add a property onto it where you can set the data collection from the unit test and then make sure the context returns that data collection.
That will let you build up a fake index with whatever data POCOs you want, and then pass that through to your standard provider implementations that are running your business data.
The big thing to remember is that this only works for any code you may be writing that will use the new Sitecore 7 way of using LINQ and the IQueryable implementation. Older style code that is running using the Sitecore.Data.Item API still works the way it used to, and has the same limitations as before.
Update: The prototype I mentioned is now available for download: http://blog.nonlinearcreations.com/2014/02/sitecore-7-developers-quest-successful-unit-testing/
We're developing alot of enhancements to salesforce using Visualforce and Apex as part of a larger system, as part of our quality metrics we have to provide a report to management on our Code Coverage.
I'd like to get a report similar to the one produced by Run All Tests in the Force.com IDE but in HTML so I can display it easily via a web interface.
For the rest of our system we use Sonar http://www.sonarsource.org/ to produce the reports.
Does anybody know the best approach to this?
I've explored the API documentation but am unable to find out if the Coverage Percentages is stored against the classes so querying that isn't an option.
Any help or pointers would be greatly appreciated.
If you run the Apex tests yourself via the API there are objects returned indicating which lines hasn't been covered by the tests in that run. You can run the tests via either the synchronous or asynchronous methods.
Then you can use the data to create a report in a format that you require. For example, I've used it to create a basic report in the FuseIT SFDC Explorer (Windows based and free). I'm just dumping out line ranges that aren't covered.
You will probably want to run all the tests in one run to get the complete code coverage of all the tests. For example, in the screenshot above I only ran one out of a much greater number of test classes. As a result it looks like the code coverage was much lower than the cumulative tests give. It does however show which lines an individual test class reaches.
I've also heard good things about MavensMate for the Sublime Text Editor. Being open source you should be able to find how it integrates with the testing api and then generates the reports.
I am beginning a fairly large new project using ColdFusion. This new project will include several developers and as such documentation of code will be key.
Another issue I am hoping to avoid (either with adequate code documentation or some other tool) is the duplication of code. A tool that would be able to "index" the code for searching or diagramming would likely help here.
What are others out there using either specifically for ColdFusion or language agnostic. We will likely be using ColdBox for the underlying framework if that makes a difference.
Thanks for any any all suggestions.
-c
Well, it's impossible to tell you which framework to use without knowing more about your project, but I can list out some tools that will be useful no matter which framework you use.
Language-agnostic tools:
GitHub.com organization+teams
Jenkins continuous integration
Apache ANT build scripts
Apache Maven for project management
Coldfusion-specific tools:
MXunit unit testing framework
MockBox for unit testing (if you use ColdBox: ColdBox-specific tutorial)
ColdDoc documentation generator
Javascript-specific tools:
JSLint or JSHint for JS code cleanup
Jasmine unit testing
Ideally, your Jenkins build server should:
Do a fresh checkout from source control
Run all unit tests and stop the build if they fail
Generate documentation
Generate a production-ready package of your project
At a minimum, I highly recommend using source control, setting up Jenkins with MXunit tests, and scheduling daily automated builds.
We used the Atlassian suite at my last job. Mostly Jira for tracking and Greenhopper for agile but the other tools may help, fisheye, bamboo, and crucible. If you host it yourself I believe they have a one time $10/product price tag that. Depending on your team's size may or may not work. If money is no subject the suite worked really nicely. It also has built in support for svn and maybe more by now.
http://www.atlassian.com/software
Sounds to me what you need is a methodology, not a tool. If you have a clearly defined set of objects/responsibilities. There should be no crossover in scripting, and if you determine a common API for the objects being coded, I would think you'd be fine.
I am looking for any and all suggestions of the best and effective resources that the StackOverflow community has used to better learn MSBuild with an emphasis on integrating unit tests and later static code analysis tools such as FxCop and StyleCop into the build process.
I have tried to find good clear documentation on adding unit tests into my build but I still am searching - even Google searches have come up empty or with just bits and pieces. Ideally I want to add unit tests, report results, and eventually add code coverage statistics, etc into the build results.
I know it has to be in MSDN somewhere but I seem unable to find anything which explains and teaches well. I am using Visual Studio Team System 2008.
Continuous Integration From Theory to Practice by Carel Lotz. It covers the entire scope of your problem, and then some. Well written, complete, and a full sample are all there.
Hands down best resource. Use it as a tutorial first, then use it as a guide, then use it as a reference.
MSDN and others are good for clarifying (or confusing) the details.
Edit: The guide by Carel Lotz uses MBUnit for unit tests (see his earlier document version for NUnit, though you can replace the MBUnit with NUnit pretty easily if you follow the NUnit help files).
Also, it is written to use Cruise Control.NET to run the MSBuild script in various configurations.
Personally, I run unit tests in a secondary MSBuild script, but have found that wrapping the NUnit calls in MSBuild gives more flexibility than running from CCNet directly.
Here's a book that might help: http://blogs.msdn.com/microsoft_press/archive/2009/01/31/sayed.aspx
I guess I need to ask if you are sure you want to use MSBuild directly? Might want to check out WIX as the MSI producing tool - there is an extensive manual and it is built on top of MSBuild.
As for the automating of your tests with reporting and integrating with NUnit, FxCop, NCover, FitNesse, etc - I think the best (free) tool out there is CruiseControl .Net. It works with all of these tools and more. Can do versioning, automated builds with automated testing, creates the reports for each...
Here is a sample of one of my builds...
http://img84.imageshack.us/img84/3664/cruisecontrolnetsamplezn0.jpg http://img84.imageshack.us/img84/3664/cruisecontrolnetsamplezn0.jpg