We're developing in C++ under Linux and about to set up automated tests. We intend to use a testing framework like CppUnit oder CxxTest. We're using Ant to build the software and we will also use it to run the tests.
As some tests are going to involve database access, we are looking for a tool or framework which facilitates the tasks of preparing and cleaning up test data in the database - just like DbUnit (a JUnit extension) in the Java world.
Another option may be to employ the actual DbUnit - a Java VM is available. Making use of DbUnit's Ant task seems to be most promising. Any related field reports are welcome!
I would recommend boost unit testing. You would probably have to use the setup and teardown to manually clean up the database. Of course, you could build your own C++ DbUnit in ODBC. IF you do let me know because I could use this as well!
I suppose you have your own C++ api to work with DB.
If it is true, you'd better to do all your DB preparation by your own. In that case you will test your DB API as well.
As there seems to be no DbUnit-like tool for C++ development, we've built a little framework of our own. Basically it's an adaptor for calling actual DbUnit operations from within C/C++ testrunners. It makes use of the Ant tasks provided by DbUnit.
We defined some macros like TS_DB_INSERT(filename) which call system("ant -Ddb.dataset=filename db.insert") and the like.
In this case, db.insert is an Ant target which executes a DbUnit task performing an INSERT operation on the database. The filename references an XML dataset containing the data to insert.
There's also an assertion macro which wraps a DbUnit compare.
The test case might look like this:
void testDatabaseStuff
{
TS_DB_INSERT("input.xml");
TestedClass::doSomething();
TS_DB_ASSERT("expected.xml");
}
Related
With code targeting the full .net Framework I could mock up an IDbConnection and point it at a mocked DataSet in order to test that my queries are executing correctly. Similarly if I were using EntityFramework 6 I could have a mocked DbSet return IQueryables and test my data layer logic against that.
However .net core doesn't support DataSets (though that may change in the future?).
In the meantime, is there a way to create a collection of objects which dapper can query using an IDbConnection in order to test the query logic?
No, all dapper is, are extension methods on top of the IDbConnection class.
There is no InMemory implementation for this (IDbConnection) (that understands SQL strings).
Your best bet however, if you want to run it completely autonomous, would be to spin up a new sql server for each time you run unit tests. This can easily be done with the docker image that Microsoft has made for sqlserver: https://hub.docker.com/r/microsoft/mssql-server-linux/
or...
Or migrate to Entity framework, they allow you to unit test against an in-memory backing store.
why?
Dapper just contains some useful features to generate SQL. It by no means abstracts away from SQL. And sql is just plain text for C# code. it does not parse it, nor execute it. Thus you cant unit test your sql/dapper code without using a database behind it.
Entity framework does it differently. it tries to make, everything that you would want to do in a database into C# code/abstraction (eg the IDbCollection). Then they make 1 implementation that generates sql code and one implementation that uses in-memory backing store. this way you can unit test your code.
Microsofts solution
Microsoft often advertises using the Repository Pattern. This is basically an expensive word for abstracting all your database calls/commands into a separate class and interfacing these classes, and use the interfaces everywhere in code (using dependency injection). Now you can write unit tests that test all your code expect for the sql queries, for this interface you make a mock to test if the method is actually called.
Another option to test you database access code (queries etc.) is use a local SQL database instance but instead recreate it every time you can start a database transaction as part of your unit-test setup and rollback the transaction in tear down. Depending on the isolation level you have chosen this also addresses concurrency issues when tests / fixtures are executed in parallel.
We're using MVC, Entity Framework 4.1 Code First, SQL Server in our project.
Please share your experience: how do you unit test your data service layer? By data service layer I mean services supposed to be run by MVC controllers that have some kind of DbContext derived class declaration inside, so that they depend on this EF DbContext, and encapsulate some business\data logic to fetch and store the data.
After reading few articles and posts, I incline to use separate database to build unit/integration tests on, and I prefer to use in-memory (like SQLite) rather than SQL Compact. However I'm not even sure if it's possible, if you have such an experience, could be please share few lines of code to show how you achieve this.
Unit testing means testing unit = no database, no external dependency, just testing single testable unit. Once you want to involve database you don't unit test any more - you are doing integration testing.
I wrote multiple answers about unit testing / integration testing of code dependent on EF. The last one is here. So if your service layer creates linq queries on context you cannot reliably unit test them. You need integration tests.
I would use the same database as you expect to use in your real code. Why? Because mapping and behaviour between database provides can differ as well as implementation of LINQ. Also in case of SQL server you can use special EF features which don't have to be available in SQLite. Another reason is that last time I checked it, SQLite's provider didn't support database deletion, recreation, etc. which is something people usually want to use for integration tests. Solution for that can be Devart provider.
I don't use a separate database at all. In fact, my Unit Tests don't use a database at all.
My strategy is to create IEnityRepository interfaces for the DB Entities (replace Entity with the actual name). I then pass those to the constructor for my controllers.
During Unit Testing, I simply use a Mocking library to pass mock implementations of the repositories that I need and have the return some set of known data that I can use in the Unit Tests.
since the dao layer methods will be dependent on data in the database,
in complex systems some operations will depend on lots of tables,
in this way unit test are not repeatable and independent,
i'm wondering how good TDD layers do this? thx.
IMHO you are on to something... If you are communicating with your Sql Server, then you are not doing unit test but integration tests. If you do TDD, then you realise this and start to put server communication into wrappers, so you can stub and mock any test data, instead of using a framework like DbUnit to control database state. I think that Your business logic should not be directly in touch with databases -- or webservices or other external resources. If it is, odds are that you will never write anything but integration tests.
A testing framework like DbUnit is exactly what you want. From their site:
DbUnit is a JUnit extension (also usable with Ant) targeted at database-driven projects that, among other things, puts your database into a known state between test runs. This is an excellent way to avoid the myriad of problems that can occur when one test case corrupts the database and causes subsequent tests to fail or exacerbate the damage.
DbUnit also supports a variety of RDBMS's but I might recommend something like HSQLDB, which can be embedded into your project/tests so your unit tests aren't dependent on being able to connect to a database somewhere in your company's basement. :) Although, on the other hand, you will be testing using a different RDBMS than you would be using in a production environment...
When unit testing, is it a must to use a database when testing CRUD operations?
Can sql lite help with this? Do you have to cre-create the db somehow in memory?
I am using mbunit.
No. Integrating an actual DB would be integration testing. Not unit testing.
Yes you could use any in-memory DB like SQLite or MS SQL Compact for this if you can't abstract (mock) your DAL/DAO in any other way.
With this in mind I have to point out, that unit testing is possible all the way to DAL, but not DAL itself. DAL will have to be tested with some sort of an actual DB in integration testing.
As with all complicated question, the answer is: It depends :)
In general you should hide your data access layer behind an interface so that you can test the rest of the application without using a database, but what if you would like to test the data access implementation itself?
In some cases, some people consider this redundant since they mostly use declarative data access technologies such as ORMs.
In other cases, the data access component itself may contain some logic that you may want to test. That can be an entirely relevant thing to do, but you will need the database to do that.
Some people consider this to be Integration Tests instead of Unit Tests, but in my book, it doesn't matter too much what you call it - the most important thing is that you get value out of your fully automated tests, and you can definitely use a unit testing framework to drive those tests.
A while back I wrote about how to do this on SQL Server. The most important thing to keep in mind is to avoid the temptation to create a General Fixture with some 'representative data' and attempt to reuse this across all tests. Instead, you should fill in data as part of each test and clean it up after.
When unit testing, is it a must to use a database when testing CRUD operations?
Assuming for a moment that you have extracted interfaces round said CRUD operations and have tested everything that uses said interface via mocks or stubs. You are now left with a chunk of code that is a save method containing a bit of code to bind objects and some SQL.
If so then I would declare that a "Unit" and say you do need a database, and ideally one that is at least a good representation of your database, lest you be caught out with vender specific SQL.
I'd also make light use of mocks in order to force error conditions, but I would not test the save method itself with just mocks. So while technically this may be an integration test I'd still do it as part of my unit tests.
Edit: Missed 2/3s of your question. Sorry.
Can sql lite help with this?
I have in the past used in memory databases and have been bitten as either the database I used and the live system did something different or they took quite some time to start up. I would recommend that every developer have a developer local database anyway.
Do you have to cre-create the db somehow in memory?
In the database yes. I use DbUnit to splatter data and manually keep the schema up to date with SQL scripts but you could use just SQL scripts. Having a developer local database does add some additional maintenance as you have both the schema and the datasets to keep up to data but personally I find is worth while as you can be sure that database layer is working as expected.
As others already pointed out, what you are trying to achieve isn't unit testing but integration testing.
Having that said, and even if I prefer unit testing in isolation with mocks, there is nothing really wrong with integration testing. So if you think it makes sense in your context, just include integration testing in your testing strategy.
Now, regarding your question, I'd check out DbUnit.NET. I don't know the .NET version of this tool but I can tell you that the Java version is great for tests interacting with a database. In a few words, DbUnit allows you to put the database in a known state before a test is run and to perform assert on the content of tables. Really handy. BTW, I'd recommend reading the Best Practices page, even if you decide to not use this tool.
Really, if you are writing a test that connects to a database, you are doing integration testing, not unit testing.
For unit testing such operations, consider using some typed of mock-database object. For instance, if you have a class that encapsulates your database interaction, extract an interface from it and then create an inheriting class that uses simple in-memory objects instead of actually connecting to the database.
As mentioned above, the key here is to have your test database in a known state before the tests are run. In one real-world example, I have a couple of SQL scripts that are run prior to the tests that recreate a known set of test data. From this, I can test CRUD operations and verify that the new row(s) are inserted/updated/deleted.
I wrote a utility called DBSnapshot to help integration test sqlserver databases.
If your database schema is changing frequently it will be helpful to actually test your code against a real db instance. People use SqlLite for speedy tests (because the database runs in memory), but this isn't helpful when you want to verify that your code works against an actual build of your database.
When testing your database you want to follow a pattern similar to: backup the database, setup the database for a test, exercise the code, verify results, restore database to the starting state.
The above will ensure that you can run each test in isolation. My DBSnapshot utility will simplify your code if your writing it .net. I think its easier to use than DbUnit.NET.
I am introducing automated integration testing to a mature application that until now has only been manually tested.
The app is Windows based and talks to a MySQL database.
What is the best way (including details of any tools recommended) to keep tests independent of each other in terms of the database transactions that will occur?
(Modifications to the app source for this particular purpose are not an option.)
How are you verifying the results?
If you need to query the DB (and it sounds like you probably do) for results then I agree with Kris K, except I would endeavor to rebuild the DB after every test case, not just every suite.
This helps avoid dangerous interacting tests
As for tools, I would recommend CppUnit. You aren't really doing unit tests, but it shouldn't matter as the xUnit framework should give you the set up and teardown framework you'll need to automatically set up your test fixture
Obviously this can result in slow-running tests, depending on your database size, population etc. You may be able to attach/detach databases rather than dropping/rebuilding.
If you're interested in further research, check out XUnit Test Patterns. It's a fine book and a good website for this kind of thing.
And thanks for automating :)
Nick
You can dump/restore the database for each test suite, etc. Since you are automating this, it may be something in the setup/teardown functionality.
I used to restore the database in the SetUp function of the database related unit test class. This way it was ensured that each test runs under the same conditions.
You may consider to prepare special database content for the tests, i.e. with less data than the current production version (to keep the restore times reasonable).
The best environment for such testing, I believe, is VMWare or an equivalent. Set up your database, transaction log and so on, then record the whole lot - database as well as configuration. Then to re-test, reload the image and database and kick off the tests. This still requires maintenance of the tests as the system changes, but at least the tests are repeatable, which is one of your greatest challenges in integration testing.
For test automation, many people use Perl, but we've found that Perl programs grow like Topsy and become convoluted. The use of Python as a scripting language (we run C++ tests) is worthwhile if you're trying to build a series of structured tests.
As #Kris K. says dumping and restoring the database between each test will probably be the way to go.
Since you are looking at doing testing external to the App I would look to build the testing framework in a language where you can take advantage of better testing tools.
If you built the testing framework in Java you could take advantage of JUnit and potentially even something like FitNesse.
Don't think that just because the application under test is C++ that means you are stuck using C++ for your automated testing.
Please try AnyDbTest, I think it is the very tool you are finding. (www.anydbtest.com).
Features:
1.Writing test case with Xml, not Java/C++/C#/VB code. Not need those expensive programming tools.
2.Supports all popular databases, such as Oracle/SQL Server/My SQL
3.So many kinds of assertion supported, such as StrictEqual, SetEqual, IsSupersetOf, Overlaps, and RecordCountEqual etc. Plus, most of assertions can prefix logic not operator.
4.Allows using an Excel spreadsheet/Xml as the source of the data for the tests. As you know, Excel spreadsheet is to easily create/edit and maintain the test data.
5.Supports sandbox test model, if one test will be done in sandbox, all database operations on each DB will be rolled back meaning any changes will be undone.
6.Allows performing data pump from one database/Excel into target database in testing initialization and finalization phase. This is easy way to prepare the test data for testing.
7.Unique cross-different-type-database testing, which means target and reference result set can come from two databases, even one is SQL Server, another is Oracle.
8.Set style comparison for recordset. AnyDbTest will tell you what is the intersection, or surplus or absence between the two record sets.
9.Sequential style comparison for recordset or scalar values. It means the two result set will be compared in their original sequence.
10.Allow to export result set of SQL statement into Xml/Excel file.