I've set up unit tests that test a fake repository and tests that make use of a fake repository.
But what about testing the real repository that hits the database ? If this is left to integration tests then it would be seem that it isn't tested directly and problems could be missed.
Am I missing something here?
Well, the integration tests would only test the literal persistence or retrieval of data to and from the layer of persistence. If your repository is doing any kind of logic concerning that data (validation, throwing exceptions if an object isn't found, etc.), that can be unit tested by faking what the persistence layer returns (whether it returns the queried object, a return code, or something else). Your integration test will assure you that the code can physically persist/retrieve data from persistence, and that's it. Any sort of logic to test ought to belong in a unit test.
Sometimes, however, logic could exist in the persistence layer itself (e.g. stored procedures). This could be for the sake of efficiency, or it could merely be legacy code. This is harder to properly unit test, as you can only access the logic by getting to the database. In this scenario, it'd probably be best to try and move the logic to your code base as much as possible, so that it can be tested more easily. There probably exist unit testing frameworks for scenarios such as these, but I'm not aware of them (merely out of inexperience).
Can you set up a real repository that tests against a fake database?
Regardless of what you do, this is integration testing, not unit testing.
I'd definitely suggest integration tests against the DAL, within reason.
We don't use the Repository pattern per se (to our chagrin), but our policy for similar classes (Searchers) is as follows:
If the method does a simple retrieve from the database using an O/RM call, don't test it.
If the method uses query-building features of the O/RM, test it.
If the method contains a string (such as a column name), test it.
If the method calls a stored procedure, test it.
If the method contains logic, test it. But try to avoid logic.
If the method bypasses the O/RM and uses raw SQL, test it. But really try to avoid this.
The gist is you should know your O/RM works (and hopefully has tests), so there's no reason to test basic CRUD behavior.
You'll definitely want a "test deck" - an in-memory database, a local file-backed database that can be checked into source control, or (if you have to) a shared database. Some testing frameworks offer rollback facilities to restore the database state; just be careful if you're hitting multiple databases in the same test or (in some cases) if you have embedded transactions.
EDIT: Note that these integration tests will still test your repository in "isolation" (save for the database). All your other unit tests will use a fake repository.
I recently covered a very similar question over here.
In summary: test your concrete Repository implementations if there's value in doing so. If you are doing something complex in your implementation, it is probably a good idea to test it. If you are using an ORM with no custom logic, there may not be much value in writing tests at that level.
Related
I am just starting to get into unit testing and cant see an easy way to do a lot of test cases due to the interaction with a database.
Is there a standard method/process for unit testing where database access (read and write) is required in order to assert tests?
The best I can come up with so far is to have a config file used to bootstrap my app using a different db connection and then use the startup method to copy over the live db to a db used in isolation for tests?
Am I close? Or is there a better approach to this?
Your business logic shouldn't directly interact with the Database. Instead it should go through a data access layer that you can fake and mock in the context of unit testing. Look into mocking frameworks to do the mocking for you. Your tests should not depend on a database at all. Instead you should specify the data returned from your data access layer explicitly, and then ensure that your business logic behaves correctly with that information.
Testing that the program works with a DB attached is more of an integration test, and those have a lot of costs associated with them. They are slower (so it's harder to run them every time you compile), and more complicated (so they require more time and effort to maintain). If you can get simpler unit tests in place, I would recommend you do that first. Later you can add integration tests that might use the DB as well, but you'll get the most value from adding simpler unit tests first.
As far as unit-test go, I think whatever works for you in practice is the way to go. It's important that unit tests give you some value and improve the quality of your system and your ability to develop and maintain it.
I would suggest you probably don't want to be copying the live db over to your test db. There's probably no guarantees that your live database will contain suitable data that will cause your unit-tests to consistently run. The unit-tests should test that your code works, they shouldn't be testing that the live database happens to contain suitable data that causes them to pass, because as it's live, your users might change the content of it so that your tests fail.
You're unit test code itself should probably populate your test db with data required that simulates the scenarios you want to write unit tests for. I messed around with some ruby on rails code a few years ago; the test framework for that would have a test class which setup the db with some fake data, then multiple test methods from the class would be written to run against that data, and the tear-down method would wipe data from the database. So, different test-classes (or sometimes people call them fixtures) would run against a certain data setup, that meant you could run a number of tests against the same data setup instead of creating it for every test case you wanted to run. Setting up data for every test could end up causing your tests to run slowly, such that you get bored of waiting for them to run and stop bothering with them.
If I'm having Data Access Layer (nHibernate) for example a class called UserProvider
and a Business Logic class UserBl, should I both test their methods SaveUser or GetUserById, or any other public method in DA layer which is called from BL layer. Is this a redundancy or a common practice to do?
Is it common to unit test DA layer, or that belongs to Integration test domain?
Is it better to have test database, or create database data during test?
Any help is appreciated.
There's no right answer to this, it really depends. Some people (e.g Roy Osherove) say you should only test code which has conditional logic (IF statements etc), which may or may not include your DAL. Some people (often those doing TDD) will say you should test everything, including the DAL, and aim for 100% code coverage.
Personally I only test it if it has logic in, so end up with some DAL methods tested and some not. Most of the time you just end up checking that your BL calls your DAL, which has some merit but I don't find necessary. I think it makes more sense to have integration tests which cover the app end-to-end, including the database, which covers things like GetUserById.
Either way, and you probably know this already, but make sure your unit tests don't touch an actual database. (No problem doing this, but that's an integration test not a unit test, as it takes a lot longer and involves complex setup, and should be run separately).
It is a good practice to write unit test for every layer, even the DAL.
I don't think running tests on the real db is a good idea, you might ruin important data. We used to set up a copy of the db for tests with just enough data in it to run tests on.
In our test project we had a special web.config file with test settings, like a ConnectionString to our test db.
In my experience it was helpful to test each layer on its own. Integrating it and test again. Integration test normally does not test all aspects. Sometimes if the Data Access Layer (I don't know nHibernate) is generated code or sort of generic code it looks like overkill. But I have seen it more than once that systematic testing pays off.
Is it redundancy? In my opinion it is not.
Is it common practice? Hard to tell. I would say no. I have seen it in some projects but not in all projects I worked in. Was often dependend on time/resources and mentality of the team / individiual developer.
Is it better to have test database, or create database data during test? This is quite a different question. Cannot be answered easily. Depends on your project. Create a new one is good but sometimes throws up unreal bugs (although bugs). It is depending on your project (product development or a proprietary development). Usually in an proprietary on site development a database gets migrated into from somewhere. So a second test is definitely needed with the migrated data. But this is rather at a system test level.
Unit testing the DAL is worth it as mentioned if there is logic in there, for example if using the same StoredProc for insert & update its worth knowing that an insert works, a subsequent call updates the previous and a select returns it and not a list. In your case SaveUser method probably inserts first time around and subsequently updates, its nice to know that this is whats being done at unit test stage.
If you're using a framework like iBatis or Hibernate where you can implement typehandlers its worth confirming that the handlers handle values in a way that's acceptable to your underlying DB.
As for testing against an actual DB if you use a framework like Spring you can avail of the supported database unit test classes with auto rollback of transactions so you run your tests and the DB is unaffected afterwards. See here for information. Others probably offer similiar support.
When unit testing, is it a must to use a database when testing CRUD operations?
Can sql lite help with this? Do you have to cre-create the db somehow in memory?
I am using mbunit.
No. Integrating an actual DB would be integration testing. Not unit testing.
Yes you could use any in-memory DB like SQLite or MS SQL Compact for this if you can't abstract (mock) your DAL/DAO in any other way.
With this in mind I have to point out, that unit testing is possible all the way to DAL, but not DAL itself. DAL will have to be tested with some sort of an actual DB in integration testing.
As with all complicated question, the answer is: It depends :)
In general you should hide your data access layer behind an interface so that you can test the rest of the application without using a database, but what if you would like to test the data access implementation itself?
In some cases, some people consider this redundant since they mostly use declarative data access technologies such as ORMs.
In other cases, the data access component itself may contain some logic that you may want to test. That can be an entirely relevant thing to do, but you will need the database to do that.
Some people consider this to be Integration Tests instead of Unit Tests, but in my book, it doesn't matter too much what you call it - the most important thing is that you get value out of your fully automated tests, and you can definitely use a unit testing framework to drive those tests.
A while back I wrote about how to do this on SQL Server. The most important thing to keep in mind is to avoid the temptation to create a General Fixture with some 'representative data' and attempt to reuse this across all tests. Instead, you should fill in data as part of each test and clean it up after.
When unit testing, is it a must to use a database when testing CRUD operations?
Assuming for a moment that you have extracted interfaces round said CRUD operations and have tested everything that uses said interface via mocks or stubs. You are now left with a chunk of code that is a save method containing a bit of code to bind objects and some SQL.
If so then I would declare that a "Unit" and say you do need a database, and ideally one that is at least a good representation of your database, lest you be caught out with vender specific SQL.
I'd also make light use of mocks in order to force error conditions, but I would not test the save method itself with just mocks. So while technically this may be an integration test I'd still do it as part of my unit tests.
Edit: Missed 2/3s of your question. Sorry.
Can sql lite help with this?
I have in the past used in memory databases and have been bitten as either the database I used and the live system did something different or they took quite some time to start up. I would recommend that every developer have a developer local database anyway.
Do you have to cre-create the db somehow in memory?
In the database yes. I use DbUnit to splatter data and manually keep the schema up to date with SQL scripts but you could use just SQL scripts. Having a developer local database does add some additional maintenance as you have both the schema and the datasets to keep up to data but personally I find is worth while as you can be sure that database layer is working as expected.
As others already pointed out, what you are trying to achieve isn't unit testing but integration testing.
Having that said, and even if I prefer unit testing in isolation with mocks, there is nothing really wrong with integration testing. So if you think it makes sense in your context, just include integration testing in your testing strategy.
Now, regarding your question, I'd check out DbUnit.NET. I don't know the .NET version of this tool but I can tell you that the Java version is great for tests interacting with a database. In a few words, DbUnit allows you to put the database in a known state before a test is run and to perform assert on the content of tables. Really handy. BTW, I'd recommend reading the Best Practices page, even if you decide to not use this tool.
Really, if you are writing a test that connects to a database, you are doing integration testing, not unit testing.
For unit testing such operations, consider using some typed of mock-database object. For instance, if you have a class that encapsulates your database interaction, extract an interface from it and then create an inheriting class that uses simple in-memory objects instead of actually connecting to the database.
As mentioned above, the key here is to have your test database in a known state before the tests are run. In one real-world example, I have a couple of SQL scripts that are run prior to the tests that recreate a known set of test data. From this, I can test CRUD operations and verify that the new row(s) are inserted/updated/deleted.
I wrote a utility called DBSnapshot to help integration test sqlserver databases.
If your database schema is changing frequently it will be helpful to actually test your code against a real db instance. People use SqlLite for speedy tests (because the database runs in memory), but this isn't helpful when you want to verify that your code works against an actual build of your database.
When testing your database you want to follow a pattern similar to: backup the database, setup the database for a test, exercise the code, verify results, restore database to the starting state.
The above will ensure that you can run each test in isolation. My DBSnapshot utility will simplify your code if your writing it .net. I think its easier to use than DbUnit.NET.
What is the best practice for testing an API that depends on data from the database?
What are the issues I need to watch out for in a "Continuous Integration" environment that runs Unit Tests as part of the build process? I mean would you deploy your database as part of the build scripts (may be run your installer) or should I go for hardcoded data [use MSTest Data Driven Unit Tests with XML]?
I understand I can mock the data layer for Business Logic layer but what if I had issues in my SQL statements in DAL? I do need to hit the database, right?
Well... that's a torrent of questions :)... Thoughts?
As far as possible you should mock out code to avoid hitting the database altogether, but it seems to me you're right about the need to test your SQL somewhere along the line. If you do write tests that hit the database, one key tip for avoiding headaches is to make sure that your setup gets the data into a known state, rather than relying on there already being suitable data available.
And of course, never test against your live database! But that goes without saying :)
As mentioned, use mocking to simulate DB calls in unit tests unless you want to fiddle with your tests and data endlessly. Testing sql statements implies more of an integration test. Run that separate from unit tests, they are 2 different beasts.
It's a good idea to automatically wipe the test database and then populate it with test harness data that will be assumed to be there for all of the tests that need to connect to the database. The database needs to be reset before each test for proper isolation - a failing test that puts in bad data could cause false failures on tests that follow and it gets messy if you have to run tests in a certain order for consistent results.
You can clear and populate the database with tools (DBUnit, DBUnit.NET, others) or just make your own utility classes to do the same thing.
As you said, other layers should be sufficiently decoupled from classes that actually hit the database, so the need for any kind of database being involved in testing is limited to tests run a small subset of your codebase. Your database accessing components can be mocked/stubbed for everything that depends on them.
One thing I did was create static methods that returned test data of a known state. I would then use a "fake" DAL to return this data as if I was actually calling the database. As for testing the sql/stored procedure, I tested it using SQL Management Studio. YMMV!
How are people unit testing their business applications? I've seen a lot of examples of unit testing with "simple to test" examples. Ex. a calculator. How are people unit testing data-heavy applications? How are you putting together your sample data? In many cases, data for one test may not work at all for another test which makes it hard to just have one test database?
Testing the data access portion of the code is fairly straightforward. It's testing out all the methods that work against the data that seem to be hard to test. For example, imagine a posting process where there is heavy data access to determine what is posted, numbers are adjusted, etc. There are a number of interim steps that occur (and need to be tested) along with tests afterwards that ensure the posting was successful. Some of those steps may actually be stored procedures.
In the past I've tried inserting the test data in a test database, then running the test, but honestly it's pretty painful to write this kind of code (and error prone). I've also tried just building a test database up front and rolling back the changes. That works OK but in a number of places you can't easily do this either (and many people would say that's integration testing; so be it, I still need to be able to test this somehow).
If the answer is that there isn't a nice way of handling this and it currently just sort of sucks, that would be useful to know as well.
Any thoughts, ideas, suggestions, or tips are appreciated.
My automated functional tests usually follow one of two patters:
Database Connected Tests
Mock Persistence Layer Tests
Database Connected Tests
When I have automated tests that are connected to the database, I usually make a single test database template that has enough data for all the tests. When the automated tests are run, a new test database is generated from the template for every test. The test database has to be constantly re-generated because test will often change the data. As tests are added, I usually append more data to the test database template.
There are some nice advantages to this testing method. The obvious advantage is that the tests also exercise your schema. Another advantage is that after setting up the initial tests, most new tests will be able to re-use the existing test data. This makes it easy to add more tests.
The downside is that the test database will become unwieldy. Because data will usually be added one test at time, it will be inconsistent and maybe even unrealistic. You will also end up cursing the person who setup the test database when there is a significant database schema change (which for me usually means I end up cursing myself).
This style of testing obviously doesn't work if you can't generate new test databases at will.
Mock Persistence Layer Tests
For this pattern, you create mock objects that live with the test cases. These mock objects intercept the calls to the database so that you can programmatically provide the appropriate results. Basically, when the code you're testing calls the findCustomerByName() method, your mock object is called instead of the persistence layer.
The nice thing about using mock object tests is that you can get very specific. Often times, there are execution paths that you simply can't reach in automated tests w/o mock objects. They also free you from maintaining a large, monolithic set of test data.
Another benefit is the lack of external dependencies. Because the mock objects simulate the persistence layer, your tests are no longer dependent on the database. This is often the deciding factor when choosing which pattern to choose. Mock objects seem to get more traction when dealing with legacy database systems or databases with stringent licensing terms.
The downside of mock objects is that they often result in a lot of extra test code. This isn't horrible because almost any amount of testing code is cheap when amortized over the number of times you run the test, but it can be annoying to have more test code then production code.
I have to second the comment by #Phil Bennett as I try to approach these integration tests with a rollback solution.
I have a very detailed post about integration testing your data access layer here
I show not only the sample data access class, base class, and sample DB transaction fixture class, but a full CRUD integration test w/ sample data shown. With this approach you don't need multiple test databases as you can control the data going in with each test and after the test is complete the transactions are all rolledback so your DB is clean.
About unit testing business logic inside your app, I would also second the comments by #Phil and #Mark because if you mock out all the dependencies your business object has, it becomes very simple to test your application logic one entity at a time ;)
Edit: So are you looking for one huge integration test that will verify everything from logic pre-data base / stored procedure run w/ logic and finally a verification on the way back? If so you could break this out into 2 steps:
1 - Unit test the logic that happens before the data is pushed
into your data access code. For
example, if you have some code that
calculates some numbers based on
some properties -- write a test that
only checks to see if the logic for
this 1 function does what you asked
it to do. Mock out any dependancy
on the data access class so you can
ignore it for this test of the
application logic alone.
2 - Integration test the logic that happens once you take your
manipulated data (from the previous
method we unit tested) and call the
appropriate stored procedure. Do
this inside a data specific testing
class so you can rollback after it's
completed. After your stored
procedure has run, do a query
against the database to get your
object now that we have done some
logic against the data and verify it
has the values you expected
(post-stored procedure logic /etc )
If you need an entry in your database for the stored procedure to run, simply insert that data before you run the sproc that has your logic inside it. For example, if you have a product that you need to test, it might require a supplier and category entry to insert so before you insert your product do a quick and dirty insert for a supplier and category so your product insert works as planned.
It depends on what you're testing. If you're testing a business logic component -- then its immaterial where the data is coming from and you'd probably use a mock or a hand rolled stub class that simulates the data access routine the component would have called in the wild. The only time I mess with the data access is when I'm actually testing the data access components themselves.
Even then I tend to open a DB transaction in the TestFixtureSetUp method (obviously this depends on what unit testing framework you might be using) and rollback the transaction at the end of the test suite TestFixtureTeardown.
Mocking Frameworks enable you to test your business objects.
Data Driven tests often end up becoming more of a intergration test than a unit test, they also carry with them the burden of managing the state of a data store pre and post execution of the test and the time taken in connecting and executing queries.
In general i would avoid doing unit tests that touch the database from your business objects. As for Testing your database you need a different stratergy.
That being said you can never totally get away from data driven testing only limiting the amout of tests that actually need to invoke your back end systems.
It sounds like you might be testing message based systems, or systems with highly parameterised interfaces, where there are large numbers of permutations of input data.
In general all the rules of standard unti testing still hold:
Try to make the units being tested as small and discrete as possible.
Try to make tests independant.
Factor code to decouple dependencies.
Use mocks and stubs to replace dependencies (like dataaccess)
Once this is done you will have removed a lot of the complexity from the tests, hopefully revealing good sets of unit tests, and simplifying the sample data.
A good methodology for then compiling sample data for test that still require complex input data is Orthogonal testing, or see here.
I've used that sort of method for generating test plans for WCF and BizTalk solutions where the permutations of input messages can create multiple possible execution paths.
For lots of different runs over the same logic but with different data you can use CSV, as many columns as you like for the input and the last for the output etc.