I've read a few threads on SO about usefulness of unit-testing various applications. The opinions can range from "test everything all the time" to "unit tests are useless", and everything in between ("test where it makes sense"). I tend to lean towards the middle one.
That leads me to my question. I am trying to decide if it would be beneficial or practical to have some basic unit-tests testing 3rd party ORM as suggested in this SO post:
link text
some baseline tests may be useful as insurance against future breaking changes, depending on how you are using the tool. For example, instead of mocking up the entire n-tier chain (I'm not a fan of mocking when it is not necessary), just use the ORM tool to create, read, update, and delete a typical object/record, and verify the operation using direct SQL statements on the (test) database. That way if the 3rd-party vendor later updates something that breaks the basic functionality you'll know about it, and new developers to your project can easily see how to use the ORM tool from the unit test examples.
My main reservations following this advise is that it would require way too much setup, would be a headache to maintain, and over all it would not be practical in our environment. Here's the summary of some points to consider:
The ORM we're using requires static datasource object(s) to be created and registered with its Data Access Layer and associated with authenticated user. This would require a lot of test setup, and probably would be problematic on the build server where no user is logged on.
ORM vendor has a pretty good track record of releasing new updates and not breaking basic functionality. Furthermore whenever it's time to update ORM to the latest version, I would imagine that application wouldn't go straight to production, but would be thoroughly regression tested anyway.
Maintaining test db for Unit testing is kind of problematic in this environment. Test db gets wiped out after each major release and replaced with db backup from staging with sensative data obfuscated. I would imagine in order to have a test db for ORM unit testing, we would need to run some scripts/code that would set the database in a "test" state. Again too much setup and maintenance.
And finally ORM documentation/help for new developers. I can see how something like that could be useful. But ORM vendor provides pretty good documentation/help with demo apps. So writing unit tests on top of that doesn't seem to be worth all the efforts.
So, is it worth to go through all these troubles just to make sure that ORM does what it supposed to do (which is CRUD)? Shouldn't it be a responsibility of the vendor anyway?
You said it yourself. Test where it makes sense. If it will make you "feel" better to test the 3rd party ORM, then do it. But, ultimately, you're putting your trust in their tool. What are you going to do if the ORM suddenly stops working? Have you written enough code against it that you can't easily rip it out? You'd wait for them to fix it, probably.
Basically, you have to treat 3rd party tools as the proverbial black boxes and let them do what you bought them to do. That was the reason you paid the money you did, right? To keep from having to write that yourself.
In this particular case, I wouldn't bother. I think you are correct in assuming a bad ROI here.
And yes, I consider it the responsibility of the vendor. I expect (and assume) their stuff works. That's how I treat my vendors.
It is the responsibility of the vendor to make sure the ORM does what it's supposed to do, but it's your responsibility to ensure that your application does what it's supposed to do, and if it fails for whatever reason, your clients will be unhappy, even if it's "just" because the ORM failed.
Your tests could ensure that the ORM works the way you expect it to given the way you're calling it. It's possible that the ORM will change in a way that isn't "broken" but that doesn't play nicely with your application.
That being said, if you're confident in the ORM, and feel that setting up and maintaining any kind of automated tests of the ORM is not worth the effort, it's probably not, especially if you've got other levels of testing that are likely to reveal the problems if they arise.
I personally think that real unit tests should only test the application itself, and everything that needs to be separately deployed and configured should be mocked up.
What you are saying is to write some integration/functional tests, that test the whole system end-to-end. These will never be lightweight, but probably are still useful in some cases (e.g. if your system doesn't change too much and is critical for your company at the same time). I have seen such tests automated as well, using virtual servers (either VMWare or microsoft equivalent), and an example database which was restored from file before every test run. You can also just set the ORM once, and accept that the tests will fail mainly because the configuration will break. Obviously you can test more, but be aware that the cost is higher.
Testing that 3rd party ORM library does its job is not unit testing at all. However, that's not the point of your question.
As was said numerous times in such a books like "Working Effectively with Legacy Code" by Michael Feathers or "Domain-Driven Design" by Eric Evans or "Clean Code" by Robert Martin your 3rd party ORM library is a technical detail which should be abstracted away from your codebase, precisely because you have no control over 3rd party libraries by definition. If they change, you accommodate.
So your solution is to make a wrapper around this ORM library, publishing ideally domain-related interface to rest of your application but generic interface is probably will do, too. This wrapper you need to test using fullstack automated tests, which inevitably should setup your application along with the database and all configuration and preparation required for it. This tests are not unit-level and expected to be really slow.
You can read about the different levels of tests and how they should be set up in the chapter 6 of the book "Continuous Integration" by Paul M. Duvall.
When writing true unit-level tests for your application level, you mock the wrapper above the ORM library, which you are able to do because you control the code of the wrapper.
This is a standard practice. The obvious benefit of it is that when you decide to update the ORM library or (which is highly possible) when your client/boss decide to switch to another ORM or database which this ORM is not compatible with, you will have the instant feedback about regression errors from the tests of your wrapper and all you'll need to do is to accommodate to changes inside your wrapper.
"Too much maintenance burden" is a fallacy created by lack of automation, by the way.
Related
I know what the advantages are and I use fake data when I am working with more complex systems.
What if I am developing something simple and I can easily set up my environment in a real database and the data being accessed is so small that the access time is not a factor, and I am only running a few tests.
Is it still important to create fake data or can I forget the extra coding and skip right to the real thing?
When I said real database I do not mean a production database, I mean a test database, but using a real live DBMS and the same schema as the real database.
The reasons to use fake data instead of a real DB are:
Speed. If your tests are slow you aren't going to run them. Mocking the DB can make your tests run much faster than they otherwise might.
Control. Your tests need to be the sole source of your test data. When you use fake data, your tests choose which fakes you will be using. So there is no chance that your tests are spoiled because someone left the DB in an unfamiliar state.
Order Independence. We want our tests to be runnable in any order at all. The input of one test should not depend on the output of another. When your tests control the test data, the tests can be independent of each other.
Environment Independence. Your tests should be runnable in any environment. You should be able to run them while on the train, or in a plane, or at home, or at work. They should not depend on external services. When you use fake data, you don't need an external DB.
Now, if you are building a small little application, and by using a real DB (like MySQL) you can achieve the above goals, then by all means use the DB. I do. But make no mistake, as your application grows you will eventually be faced with the need to mock out the DB. That's OK, do it when you need to. YAGNI. Just make sure you DO do it WHEN you need to. If you let it go, you'll pay.
It sort of depends what you want to test. Often you want to test the actual logic in your code not the data in the database, so setting up a complete database just to run your tests is a waste of time.
Also consider the amount of work that goes into maintaining your tests and testdatabase. Testing your code with a database often means your are testing your application as a whole instead of the different parts in isolation. This often result in a lot of work keeping both the database and tests in sync.
And the last problem is that the test should run in isolation so each test should either run on its own version of the database or leave it in exactly the same state as it was before the test ran. This includes the state after a failed test.
Having said that, if you really want to test on your database you can. There are tools that help setting up and tearing down a database, like dbunit.
I've seen people trying to create unit test like this, but almost always it turns out to be much more work then it is actually worth. Most abandoned it halfway during the project, most abandoning ttd completely during the project, thinking the experience transfer to unit testing in general.
So I would recommend keeping tests simple and isolated and encapsulate your code good enough it becomes possible to test your code in isolation.
As far as the Real DB does not get in your way, and you can go faster that way, I would be pragmatic and go for it.
In unit-test, the "test" is more important than the "unit".
I think it depends on whether your queries are fixed inside the repository (the better option, IMO), or whether the repository exposes composable queries; for example - if you have a repository method:
IQueryable<Customer> GetCustomers() {...}
Then your UI could request:
var foo = GetCustomers().Where(x=>SomeUnmappedFunction(x));
bool SomeUnmappedFunction(Customer customer) {
return customer.RegionId == 12345 && customer.Name.StartsWith("foo");
}
This will pass for an object-based fake repo, but will fail for actual db implementations. Of course, you can nullify this by having the repository handle all queries internally (no external composition); for example:
Customer[] GetCustomers(int? regionId, string nameStartsWith, ...) {...}
Because this can't be composed, you can check the DB and the UI independently. With composable queries, you are forced to use integration tests throughout if you want it to be useful.
It rather depends on whether the DB is automatically set up by the test, also whether the database is isolated from other developers.
At the moment it may not be a problem (e.g. only one developer). However (for manual database setup) setting up the database is an extra impediment for running tests, and this is a very bad thing.
If you're just writing a simple one-off application that you absolutely know will not grow, I think a lot of "best practices" just go right out the window.
You don't need to use DI/IOC or have unit tests or mock out your db access if all you're writing is a simple "Contact Us" form. However, where to draw the line between a "simple" app and a "complex" one is difficult.
In other words, use your best judgment as there is no hard-and-set answer to this.
It is ok to do that for the scenario, as long as you don't see them as "unit" tests. Those would be integration tests. You also want to consider if you will be manually testing through the UI again and again, as you might just automated your smoke tests instead. Given that, you might even consider not doing the integration tests at all, and just work at the functional/ui tests level (as they will already be covering the integration).
As others as pointed out, it is hard to draw the line on complex/non complex, and you would usually now when it is too late :(. If you are already used to doing them, I am sure you won't get much overhead. If that were not the case, you could learn from it :)
Assuming that you want to automate this, the most important thing is that you can programmatically generate your initial condition. It sounds like that's the case, and even better you're testing real world data.
However, there are a few drawbacks:
Your real database might not cover certain conditions in your code. If you have fake data, you cause that behavior to happen.
And as you point out, you have a simple application; when it becomes less simple, you'll want to have tests that you can categorize as unit tests and system tests. The unit tests should target a simple piece of functionality, which will be much easier to do with fake data.
One advantage of fake repositories is that your regression / unit testing is consistent since you can expect the same results for the same queries. This makes it easier to build certain unit tests.
There are several disadvantages if your code (if not read-query only) modifies data:
- If you have an error in your code (which is probably why you're testing), you could end up breaking the production database. Even if you didn't break it.
- if the production database changes over time and especially while your code is executing, you may lose track of the test materials that you added and have a hard time later cleaning it out of the database.
- Production queries from other systems accessing the database may treat your test data as real data and this can corrupt results of important business processes somewhere down the road. For example, even if you marked your data with a certain flag or prefix, can you assure that anyone accessing the database will adhere to this schema?
Also, some databases are regulated by privacy laws, so depending on your contract and who owns the main DB, you may or may not be legally allowed to access real data.
If you need to run on a production database, I would recommend running on a copy which you can easily create during of-peak hours.
It's a really simple application, and you can't see it growing, I see no problem running your tests on a real DB. If, however, you think this application will grow, it's important that you account for that in your tests.
Keep everything as simple as you can, and if you require more flexible testing later on, make it so. Plan ahead though, because you don't want to have a huge application in 3 years that relies on old and hacky (for a large application) tests.
The downsides to running tests against your database is lack of speed and the complexity for setting up your database state before running tests.
If you have control over this there is no problem in running the tests directly against the database; it's actually a good approach because it simulates your final product better than running against fake data. The key is to have a pragmatic approach and see best practice as guidelines and not rules.
Background
I work in a team of 7 developers and 2 testers that work on a logistics system.
We use Delphi 2007 and modeldriven development with Bold for Delphi as framework.
The system has been in production about 7 years now and has about 1,7 millions lines of code.
We release to production after 4-5 weeks and after almost every release we have to do some patches for bugs we don’t found. This is of course irritating both for us and the customers.
Current testing
The solution is of course more automatic testing. Currently we have manual testing. A Testdbgenerator that starts with an empty database and add data from the modelled methods. We also have Testcomplete that runs some very basic scripts for testing the GUI. Lack of time stop us from add more tests, but scripts is also sensitive for changes in the application. For some years ago I really tried unit testing with DUnit, but I gave up after some days. The units have too strong connections.
Unit testing preconditions
I think I know some preconditions for unit testing:
Write small methods that do one thing, but do it well.
Don’t repeat yourself.
First write the test that fails, then write the code so the test pass.
The connections between units shold be loose. They should not know much about each other.
Use dependency injection.
Framework to use
We may upgrade to Delphi XE2, mainly because of the 64-bit compiler.
I have looked at Spring a bit but this require an update from D2007 and that will not happen now. Maybe next year.
The question
Most code is still not tested automatically. So what is the best path to go for increasing testability of old code ? Or maybe it is best to start writing tests for new methods only ?
I’m not sure what is the best way to increase automatic testing and comments about it is welcome. Can we use D2007 + DUnit now and then easily change to Delphi XE2 + Spring later ?
EDIT: About current test methodology for manual testing is just "pound on it and try to break it" as Chris call it.
You want the book by Michael Feathers, Working Effectively with Legacy Code. It shows how to introduce (unit) tests to code that wasn't written with testability in mind.
Some of the chapters are named for excuses a developer might give for why testing old code is hard, and they contain case studies and suggested ways to address each problem:
I don't have much time and I have to change it
I can't run this method in a test harness
This class is too big and I don't want it to get any bigger
I need to change a monster method and I can't write tests for it.
It also covers many techniques for breaking dependencies; some might be new to you, and some you might already know but just haven't thought to use yet.
The requirements for automated unit testing are exactly this:
use an unit testing framework (for example, DUnit).
use some kind of mocking framework.
Item 2 is the tough one.
DRY, small methods, start with a test, and DI are all sugar. First you need to start unit testing. Add DRY, etc. as you go along. Reduced coupling helps to make stuff more easily unit tested, but without a giant refactoring effort, you will never reduce coupling in your existing code base.
Consider writing tests for stuff that is new and stuff that is changed in the release. Over time you will get a reasonable base of unit tests. New and changes stuff can also be refactored (or written nicely).
Also, consider an automated build process that runs unit tests and sends email when the build breaks.
This only covers unit testing. For QA testers, you will need a tool (they exist, but I can't think of any) that allows them to run automated tests (which are not unit tests).
Your testing team is too small, IMO. I've worked in teams where the QA dept outnumbers the Developers. Consider working in "sprints" of manageable chunks (features, fixes) that fit in smaller cycles. "Agile" would encourage 2-week sprints, but that may be too tight. Anyway, it would keep the QA constantly busy, working farther ahead of the release window. Right now, I suspect that they are idle until you give them a huge amount of code, then they're swamped. With shorter release cycles, you could keep more testers busy.
Also, you didn't say much about their testing methodology. Do they have standard scripts that they run, where they verify appearance and behavior against expected appearance and behavior? Or do they just "pound on it and try to break it"?
IMO, Dunit testing is hard to do with lots of dependencies like databases, communication, etc.. But it's do-able. I've created DUnit classes that automatically run database setup scripts (look for a .sql file with the same name as the class being tested, run the sql, then the test proceeds), and it's been very effective. For SOAP communications, I have a SoapUI mockservice running that returns canned results, so I can test my communications.
It does take work, but it's worth it.
I've been writing software for years, but have never mastered the art of testing. My typical testing includes thorough run-throughs on my machines, and then testing in various operating systems via VMware. Mainly a brute-force play-with-it-until-it-breaks-or-doesn't approach. Where possible I work on actual hardware, but this isn't always practical.
My question is twofold:
How do medium-sized professional development houses do their testing?
What common techniques or procedures (outside of unit testing) can apply to a developer team of one. I'm looking for practicality.
Thank you for your time and input.
Step 1: Unit Testing
Divide your software into components (which can be anything from single functions up to whole programs) and unit-test those components thoroughly, especially as relates to the API and behavior that the rest of the application can see. (Don't forget to check for failure modes too, but beware of binding too carefully to the exact nature of a failure; it's often good enough to just test for the presence of the right class of exception rather than its exact message.) Make sure those tests pass; you're honing them in on a specification of what the component should be doing. (Automated test running helps here, as does a CI system.) This is important because of…
Step 2: Integration Testing
Test that the compositions of components that make up the application work (this is integration testing). Ideally you'll only be finding bugs in the specifications of things at this point (hah!) and wherever you identify that a component is wrong despite passing its unit tests, that tells you that there is a bug. Whenever things fail to work together despite being told to do so, you've probably got a bug in your specs from the previous step so you typically resolve these things by adding more detail to your unit tests and fixing the components until they work.
Note that to make good integration, you want to keep this stage so that the integration itself is sufficiently simple that it is in the “Obviously No Bugs” class of programs instead of the larger “No Obvious Bugs” class. An integration framework like Spring or a scripting language can help a lot here (though with the latter you have to guard against creating components on the sly; if you create a component then admit it and make sure it has a proper usage contract and unit tests to ensure that it meets its contract).
Where you can, you can make components by composing others together; these higher-level components need to be unit tested as characterized in Step 1 above. This might sound like extra work – it probably is – but it does have the advantage of meaning that you can use automated tests for larger parts of the program. (Alas, it's harder to do all integration tests with an automated test tool; such things tend to work better doing unit tests where you can mock out all the irrelevant parts.) But this doesn't save you from…
Step 3: Acceptance Testing
This is where the overall application is tested to see if it actually does what is desired. This might be automatable, but usually isn't. This is level where you bring in users to let them see whether things are what they expected, though you might want to use internal testers a bit first. How easy this all is depends on the nature of the application.
Note also that user interfaces tend to spend more time in this step than the others, precisely because what makes for a good UI is difficult to impossible to pin down in algorithms (it relates much more to human psychology after all).
A final note: What I've written here sounds like testing is meant to be a laborious process that takes ages at the end of a project. It isn't! You can often get parts of an application done before others, do an integration of those parts (with mocks for the other bits) and test quite a bit of how acceptable this sub-application really is. Of course, when doing this take care to stop users from believing that everything is done; one way is to have dialog boxes that pop up and say things like “magic to happen here”. Silly but effective. :-)
For a small team unit tests or automatic integration tests are crucial. Because there are no hands and time for manual testing - the more you automate the better. This includes Continuous Integration.
Set up a separate 'beta' environment that is as close to your production environment as possible. Do most of your tests there - this way you will pick up all the things you've forgotten in your 'release plan'.
As a proffesional tester my suggestion is that you should have a healthy mix of automated and manual testing. The Examples below are in .net but it should be easy to find a tool for whatever technique you are using.
AUTOMATED TESTING
Unit Testing
Use NUnit to test your classes, functions and interaction between them.
http://www.nunit.org/index.php
Automated Functional Testing
If it's possible you should automate a lot of the functional testing. Some frame works have functional testing built into them. Otherwise you have to use a tool for it. If you are developing web sites/applications you might want to look at Selenium.
http://www.peterkrantz.com/2005/selenium-for-aspnet/
Continuous Integration
Use CI to make sure all your automated tests run every time someone in your team makes a commit to the project.
http://martinfowler.com/articles/continuousIntegration.html
MANUAL TESTING
As much as I love automated testing it is, IMHO, not a substitute for manual testing. The main reason being that an automated can only do what it is told and only verify what it has been informed to view as pass/fail. A human can use it's intelligence to find faults and raise questions that appear while testing something else.
Exploratory Testing
ET is a very low cost and effective way to find defects in a project. It take advantage of the intelligence of a human being and a teaches the testers/developers more about the project than any other testing technique i know of. Doing an ET session aimed at every feature deployed in the test environment is not only an effective way to find problems fast, but also a good way to learn and fun!
http://www.satisfice.com/articles/et-article.pdf
My testing tool examples are Java based, but I will try to suggest tools which are ported to multiple languages or are language agnostic.
Use unit testing tools like junit (ported to a variety of languages). This will allow you to refactor your code safely. Most code bugs should result in the addition or correction of at least one test.
Use revision control, and setup an automated build environment that check out the code and builds the code. It should then run the automated test suite. If the application uses a database the build environment should have its own database. Use different code branched for production (released) and development code.
Use integration testing tools like HTTPunit or Synergy to test web applications. Tools of this type are basically language agnostic, but your may want to choose a tool which can be extended in the language(s) you are using. For non-web applications, there may not be an equivelent tool for your platform. You may also want to use a performance tool like JMeter.
These tools have some setup costs, but a quick payback. Overall development time may be the same or less than if you didn't use the tools.
Acceptance tests generally don't lend themselves to automated testing. Where they do, included them in the integrated testing. Get acceptance feedback as early and often as possible.
How do the pros do it? That all depends on who the 'Pro' is... There are dozens of different approaches to testing, and plenty of experts to tell you that their way is the one true way. Agile gurus will tell you a very different story from the waterfall gurus. The ISTBQ guys will tell you a very different from the Context-Driven guys.
Unfortunately there is no one true way, and you have to figure it out for yourself. Your approach to testing depends on too many factors. That's probably not very helpful, but you just need to be aware that any answer you get here will be only one option of many, and it may be completely inappropriate for your situation.
Personally, after several years in software testing, I have decided to align myself with the Context Driven school of software testing. See: http://www.context-driven-testing.com
Secondly, from your description of you current approach, that sounds a lot like exploratory testing to me. You may find this material interesting: satisfice.com/sbtm/
One thing you can do (combined with all the previous suggestions) is identify the risky and critical areas of your app and try to focus your testing efforts on these areas.
Recently I have been trying to use unit tests in my code, and I like the idea in principle. However, the parts of my code that I am most eager to test are those error-prone areas which unit tests alone don't handle very well; for example:
Network code
Filesystem interactions
Database interactions
Communication with hardware (e.g. specialized devices that talk over RS-232)
Calls to quirky third-party libraries
I understand that mock objects are typically used in these situations, but I'm looking for a way to feel confident that the mock objects are correctly simulating the situations I want to test.
For example, suppose I want to write a mock that simulates what happens when the database server is restarted. To do this, I would want to first verify that the database library I'm using will actually throw a particular exception if the DB server is restarted. Right now, I write code like:
def checkDatabaseDropout():
connectToDatabase()
raw_input("Shut down the database and press Enter")
try:
testQuery()
assert False, "Database should have thrown an exception"
except DatabaseError, ex:
pass
Running this requires a fair amount of manual intervention, but it at least it gives me a verifiable set of assumptions I can work with in my code, and it lets me check those assumptions when I upgrade the library, switch to a different underlying database, etc.
My question is: are there better ways of handling this? Are there frameworks that support this kind of semi-automated testing? Or do people generally use other techniques at this end of the testing spectrum?
I try to not foresee these kinds of things.
Even though I'm doing close to 100% TDD, at the end of the day, I'm still building a complete system, so I also test that the entire application runs as expected. Such System Tests can capture and reproduce the kind of scenarios you talk about.
Once I know how to reproduce a given scenario, I can always write a unit test that reproduces it.
So in other words, I currently tend to work with two configurations:
Fully automated unit tests
Manual system tests.
These can interact and feed each other, iteratively making each easier and better to work with.
Unit tests are often deployed with software releases to validate the install - i.e. do the install, run the tests and if they pass then the install is good.
I'm about to embark on a project that will involve delivering prototype software library releases to customers. The unit tests will be delivered as part of each release and in addition to using the tests to validate the install, I plan on using the unit tests that test the API as a "contract" for how the release should be used. If the user uses the release in a similar manner to how it is used by the unit tests then great. If they use it some other way then all bets are off.
Has anybody tried this before? Any thoughts on whether this is a good/bad idea?
Edit: To highlight a good point raised by ChrisA and Dan in replies below, the "unit tests that test the API" are better called the integration tests and their intent is to exercise the API and the software to demonstrate the functionality of the software from a customer perspective.
Sounds like a good idea to me. I (we all?) routinely use unit tests internally to do just that. In using my unit tests to validate that I haven't broken anything I'm also implicitly verifying that my API contract hasn't changed. It seems like a natural usage of unit tests to deploy them in the fashion you're talking about.
Agile methodologies say: Tests are specifications, so this is a very good idea.
I fully expect to be flamed for this, but I don't understand how a set of unit tests proves anything at all about the kind of things a customer cares about, namely whether the application meets his business requirements.
Here's an example: I've just finished converting a chunk of code to fix a big mistake we made. It was a classic case of over-engineering, and the changes have touched about a dozen windows forms and about as many classes.
It's taken me a couple of days, it's now a lot simpler, we gained some features for free, and we lost a ton of code that did stuff that we now know we never really needed.
Every single one of those forms worked perfectly before. The public methods did exactly what they needed to do, and the underlying data accesses were just fine.
So any unit test would have passed.
Except, sadly, they did the wrong thing - which we didn't realise, except in retrospect. It's as if we'd built a prototype and only after trying to use it, realised that it wasn't right.
So now we have a leaner, meaner, fitter application.
But the things that were wrong, were wrong at a level where unit tests could never have revealed them, so I'm just not understanding how shipping a set of unit tests with an install does anything except give a false sense of security.
Maybe I'm not understanding something, but it seems to me that unless the thing that is shipped functions at the same level as the tests supplied, they prove nothing.
It's actually a pretty good idea, and extremely pleasant as an API user.
This technique can actually also be used the other way round : when you're using a "legacy" API, you can use unit tests to document the way you think the API behaves and to validate that it actually behaves as planned.
If you are releasing a code library, this sounds great.
If you are releasing an ordinary software product with which your users will interact only via a GUI, your unit tests may not be working at the same level of abstraction or may not the most useful tool to assess the behaviour of your product. A really good user manual (yes, this is possible) might be better for that.
If you're interested in providing a set of specifications with your code, perhaps you should investigate some of the behavior-driven development tools (nbehave, jbehave, rspec, etc.). These frameworks provide support for describing your tests in given/when/then syntax and outputting formatted results that are in a natural language. See nbehave for an example of a BDD tool for .NET. You can find an excellent description of BDD here
Another option may be for you to write tests using an acceptance testing framework such as fit or fitnesse (or the java-only concordion) and deliver these acceptance tests with the code. Both fit/fitnesse and concordion allow specification of the tests in plain HTML or even Word documents.
The benefit of either approach (BDD or acceptance testing frameworks) is that the results the user sees are more human-readable and understandable.
Tests will check requirements.
Requirements define functionality
=> Tests will check functionality.
The problem is, that only functionality can be checked that can be covered by unit tests. Integration or whole system tests won't work.
Otherwise, it's the main approach of TDD to check functionality via unit tests.
Meszaros calls this "Tests as documentation"