I am wondering if you guys have any good reading to consider what to classify as unit testing / acceptance / integration testing. I have the following scenario and we're having a bit of a debate at work if it should be in unit tests:
In our data access layer, some statements use sql such as "select * from people where id IN ('x', 'y'), where the IN statement is dynamically generated according to the input. Recently we found out that our Oracle db have a limit of 1000 variables within the IN statement.
I personally think this is not a unit testing scenario. We test if the sql work against the database in unit tests and if the logic is right. However, stress testing should be done at higher level.
If we are to do testing with 1000s of records in unit tests, we need to fill the database each time with large number of records, which might be inefficient.
Any advice?
Regarding your particuliar example, you should in fact consider to do 2 tests for it:
The first one is a Unit Test, and will check that your function can accept the maximum number of input requested by the requirements. If nothing is specified there, ask for clarification to the analyst. Dynamically generated requests such as this are a pain to debug afterward.
The second one is a Stress test. But it should not be performed onthis particular part of your code, but rather on the integrated part tha will use it. If you start stress testing unit block, you'll end up doing premature optimisation because you'll loose the big picture and start making assumptions on how this will work togetheter rather than observing how it does really.
I believe that stress tests should be implemented as part of unit testing. Generally your unit tests should contain
Accuracy tests
Failure tests
Stress tests
If you don't want to run stress tests each time when you running other tests, you can consider to group stress tests in separate text fixture.
Unit tests should test, and specify, the functionality of the unit under test. In this case you are testing the database, not the unit, so I think this test is really a unit test.
A unit should be independent of the database it is using, if you are testing the way a unit interacts with a particular database then it seems like an integration test to me
Related
Let's assume we are not doing TDD (for which unit tests are obviously part and parcel), and have integration tests for all the use cases.
The integration tests assume assume a certain input and validate the output is as expected.
My thinking is that adding a unit test for a method that is traversed in an integration test, using the same data as would exist in the method in the integration test, would not expose any additional bugs.
That would lead to the conclusion that provided you have suffcient integration tests you do not then need to unit test the same code.
So, can someone give a concrete example where a unit test could expose a bug in the above scenario?
Integration tests can be seen as a form of Acceptance Testing. They ensure that the software is doing what it is supposed to be doing.
Unit tests, on the other hand, aren't particularly useful for customers. A customer is not concerned that the InitializeServerConnection is failing, but they are concerned that they're unable to send internal messages to their co-workers as a result.
So what good are unit tests for? They are a development tool, full stop. A unit test verifies that a cog in the machine is working properly. And if it is not, it is very easy to see it failing.
Arialdo Martini offers a great explanation:
Oversimplifying, a software system can be seen as a network of cooperating modules. Since they cooperate, some of them depend on other.
[...]
With integration and end-to-end tests you would be able to find all the broken features.
Yet, this is not of any help in guessing where the bug is. The same system, with the same bug, would result in these unit test failures:
So, even though a unit test doesn't add any business value, it does add value in the form of reducing the amount of time spent manually testing, debugging, and sifting through code looking for the root cause of an issue.
I have often seen tests where canned inputs are fed into a program, one checks the outputs generated against canned (expected) outputs usually via diff. If the diff is accepted, the code is deemed to pass the test.
Questions:
1) Is this an acceptable unit test?
2) Usually the unit test inputs are read in from the file system and are big
xml files (maybe they represent a very large system). Are unit tests supposed
to touch the file system? Or would a unit test create a small input on the fly
and feed that to the code to be tested?
3) How can one refactor existing code to be unit testable?
Output differences
If your requirement is to produce output with certain degree of accuracy, then such tests are absolutely fine. It's you who makes the final decision - "Is this output good enough, or not?".
Talking to file system
You don't want your tests to talk to file system in terms of relying on some files to exists somewhere in order for your tests to work (for example, reading values from configuration files). It's a bit different with tests input resources - you can usually embed them in your tests (or at least test project), treat them as part of codebase, and on top of that they usually should be loaded before test executes. For example, when testing rather large XMLs it's reasonable to have them stored as separete files, rather than strings in code files (which sometimes can be done instead).
Point is - you want to keep your tests isolated and repeatable. If you can achieve that with file being loaded at runtime - it's probably fine. However it's still better to have them as part of codebase/resources than standard system file lying somewhere.
Refactoring
This question is fairly broad, but to put you in the right direction - you want to introduce more solid design, decouple objects and separate responsibilities. Better design will make testing easier and, what's most important - possible. Like I said, it's broad and complex topic, with entire books dedicated to it.
1) is this an acceptable unit test?
This is not a unit test by definition. A unit test focuses on the smallest possible amount of code. Your test can still be a useful test, regression test, self-documenting test, TDD test, etc. It is just not a unit test, although it may be equally useful.
2) Are unit tests supposed to touch the file system?
Typically not, unless you need to unit test something explicitly related to the filesystem. One reason is, if you have several hundred unit tests it is nice to have them run in a couple seconds rather than minutes.
3) How can one refactor existing code to be unit testable?
A better question is why do you want the code to be unit testable? If you are trying to learn TDD it is better to start with a new project. If you have bugs then try to write tests for the bugs. If the design is slowing you down then you can refactor towards testability over time.
Addressing only the 3rd question. It is extremely difficult. You really need to write tests at the same time you write the code, or before. It is a nightmare to try to slap tests onto an existing code base, and it is often more productive to throw away the code and start over.
This is an acceptable unit test.
The files being read should be part of the test project so anyone that checks out the project from repository will have the same files at the same relative location.
Having black box tests is a great start, you can refactor the existing code and use the current tests to verify that it is still working (depending on the quality of the tests). Here is a short blog about refactoring for testability: http://www.beletsky.net/2011/02/refactoring-to-testability.html
A diff test can be acceptable as a Unit Tests, especially when your using test data that is shared between Unit Tests.
If you don't know how many items there are in the SUT you could use the following:
int itemsBeforeTest = SUT.Items.Count;
SUT.AddItem();
Assert.AreEqual(itemsBeforeTest + 1, SUT.Items.Count);
If a Unit Tests requires so much data that it needs to be read from a big XML file, it's not a real Unit Test. A Unit Test should test a class in complete isolation and mock out all dependencies.
Using a pattern like the Builder pattern, can also help in creating test data for your unit test. The biggest problem with having your test data in a separate file, is that it's hard to understand what the test does exactly. If you create your test data in the arrange part of your unit test, it's immediately clear what is important for your test.
For example, let's say you have the following arrange code to test if the price of an invoice is correct:
Address billingAddress = new Address("Stationsweg 9F",
"Groningen", "Nederland", "9726AE"); shippingAddress = new Address("Aweg 1",
"Groningen", "Nederland", "9726AB");
Customer customer = new Customer(99, "Piet", "Klaassens",
30,
billingAddress,
shippingAddress);
Product product = new Product(88, "Tafel", 19.99);
Invoice invoice = new Invoice(customer);
Can be changed to the following when using a Builder
Invoice invoice = Builder<Invoice>.CreateNew()
.With(i => i.Product = Builder<Product>.CreateNew()
.With(p => p.Price = 19.99)
.Build())
.Build();
When using a Builder its much easier to see what is important and your code is also more maintainable.
Refactoring code to become more testable is a broad topic. It comes down to thinking about 'how would I test this code?' while you are writing the code.
Take the following example:
public class AlarmClock
{
public AlarmClock()
{
SatelliteSyncService = new SatelliteSyncService();
HardwareClient = new HardwareClient();
}
}
This is hard to test. You need to make sure that both the SatteliteSyncService and the HardwareClient are functional when testing the AlarmClock.
This change tot the constructor makes it much easier to test:
public AlarmClock(IHardwareClient hardwareClient, ISatelliteSyncService satelliteSyncService)
{
SatelliteSyncService = satelliteSyncService;
HardwareClient = hardwareClient;
}
Techniques like Dependency Injection help with refactoring your code to be more testable. Also watch out for static values like DateTime.Now or the use of a Singleton because they are hard to test.
A really good introduction to writing testable code can be found here.
You should not require the code to be refactored to be able to perform unit tests. Unit tests, as the name implies, are testing a unit of code for the system. The best unit tests are small, quick to execute and exercise only a very small subset of the piece of code being tested (e.g. class).
The reason for having small, compact unit tests that only exercise one part of the code is that the objective of unit tests is to find bugs in that unit of code. If the unit test takes a long time to execute and tests lots of things it makes the identification of a bug in the code that much harder.
As to accessing the file system, I see no problem. Some unit tests may require a database to be constructed before the test is carried out, the output to be checked that would be difficult or time expensive to write the checks in code.
The files for unit testing should be treated like the rest of the code - put under version control. If you are paranoid you could implement a check within the unit test such as perform a MD5 on it and check against a hard coded value so future reruns of the test can verify that the test data has not inadvertenly changed.
Just my humble thoughts.
Is there anyone that can clearly define these levels of testing as I find it difficult to differentiate when doing TDD or unit testing. Please if anyone can elaborate how, when to implement these?
Briefly:
Unit testing - You unit test each individual piece of code. Think each file or class.
Integration testing - When putting several units together that interact you need to conduct Integration testing to make sure that integrating these units together has not introduced any errors.
Regression testing - after integrating (and maybe fixing) you should run your unit tests again. This is regression testing to ensure that further changes have not broken any units that were already tested. The unit testing you already did has produced the unit tests that can be run again and again for regression testing.
Acceptance tests - when a user/customer/business receive the functionality they (or your test department) will conduct Acceptance tests to ensure that the functionality meets their requirements.
You might also like to investigate white box and black box testing. There are also performance and load testing, and testing of the "'ilities" to consider.
Unit test: when it fails, it tells you what piece of your code needs to be fixed.
Integration test: when it fails, it tells you that the pieces of your application are not working together as expected.
Acceptance test: when it fails, it tells you that the application is not doing what the customer expects it to do.
Regression test: when it fails, it tells you that the application no longer behaves the way it used to.
Here's a simple explanation for each of the mentioned tests and when they are applicable:
Unit Test
A unit test is performed on a self-contained unit (usually a class or method) and should be performed whenever a unit has been implemented or updating of a unit has been completed.
This means it's run whenever you've written a class/method, fixed a bug, changed functionality...
Integration Test
Integration test aims to test how well several units interact with each other. This type of test should be performed Whenever a new form of communications has been established between units or the nature of their interaction have changed.
This means it's run whenever a recently written unit is integrated into the rest of the system or whenever a unit which is interacts with other systems has been updated (and successfully completed its unit tests).
Regression Test
Regression tests are performed whenever anything has been changed in the system, in order to check that no new bugs have been introduced.
This means it's run after all patches, upgrades, bug fixes. Regression testing can be seen as a special case of combined unit test and integration test.
Acceptance Test
Acceptance tests are performed whenever it is relevant to check that a subsystem (possibly the entire system) fulfils its entire specifications.
This means it's mainly run before finishing a new deliverable or announcing completion of a larger task. See this as your final check to see that you've really completed your goals before running to the client/boss and announcing victory.
This is at least the way I learned, though I'm sure there are other opposing views. Either way, I hope that helps.
I'll try:
Unit test: a developer would write one to test an individual component or class.
Integration test: a more extensive test that would involve several components or packages that need to collaborate
Regression test: Making a single change to an application forces you to re-run ALL the tests and check out ALL the functionality.
Acceptance test: End users or QA do these prior to signing off to accept delivery of an application. It says "The app met my requirements."
Unit Test: is my single method working correctly? (NO dependencies, or dependencies mocked)
Integration Test: are my two separately developed modules working corectly when put together?
Regression Test: Did I break anything by changing/writing new code? (running unit/integration tests with every commit is technically (automated) regression testing). More often used in context of QA - manual or automated.
Acceptance Test: testing done by client, that he "accepts" the delivered SW
Can't comment (reputation to low :-| ) so...
#Andrejs makes a good point around differences between environments associated with each type of testing.
Unit tests are run typically on developers machine (and possibly during CI build) with mocked out dependencies to other resources/systems.
Integration tests by definition have to have (some degree) of availability of dependencies; the other resources and systems being called so the environment is more representative. Data for testing may be mocked or a small obfuscated subset of real production data.
UAT/Acceptance testing has to represent the real world experience to the QA and business teams accepting the software. So needs full integration and realistic data volumes and full masked/obfuscated data sets to deliver realistic performance and end user experience.
Other "ilities" are also likely to need the environment to be as close as possible to reality to simulate the production experience e.g. performance testing, security, ...
(Leaving aside hair-splitting about if this is integration-testing or unit-testing.)
I would rather first test at the large scale. If my app writes a VRML file that is the same as the reference one then the VRML exporter works, I don't then have to run unit tests on every single statement in the code. I would also like to use this do some level of poor-man gui testing by comparing screenshots.
Is there a unit test framework (for C++ ideally) that integrates this sort of testing - or at least makes it easy to integrate with unit tests?
edit. It seems a better term is approval testing. So are there any other unit test frameworks that incorporate Approval Testing ?
Have a look at Approval Tests, written by a couple of friends of mine. Not C++, but it's the general idea of what you're after, also known as Golden Master tests. Good stuff, whether it's unit tests or not.
Kitware, for VTK, uses CDash to do most of its testing. Many of its tests are similar in nature to this - they write out an image of the rendered model, and compare it to a reference image.
In addition, they have code in there to specifically handle very subtle differences to the reference image due to different graphics card drivers/manufacturers/etc. The tests can be written in a way to compare the reference image with some tolerance.
Okay, I think you're making an incorrect assumption about the nature of unit test code; your statement that
If my app writes a VRML file that is
the same as the reference one then the
VRML exporter works, I don't then have
to run unit tests on every single
statement in the code.
is strictly correct if you're looking to do a validation test on your code, but note that this type of test is strictly different than what a unit test actually is. Unit tests are for testing individual units of code; they do not exist for verification purposes. Depending on your environment, you may not need unit tests at all, but please keep in mind that validation tests (testing the validity of the overall program output) and unit tests (testing that the individual code units work as expected) are completely different things.
(Note that I'm really not trying to be nitpicky about this; also, you can use plenty of unit test frameworks to achieve this result; keep in mind, though, that what you're writing aren't really "Unit Tests", despite running them in a Unit Test framework.)
It seems to me that most people write their tests against in-memory, in-process databases like SQLite when working with NHibernate. I have this up and running but my first test (that uses NHibernate) always takes between 3-4 seconds to execute. The next test runs much faster.
I am using FluentNhibernate to do the mapping but get roughly the same timings with XML mapping files. For me the 3-4 second delay seriously disrupts my flow.
What is the recomended way of working with TDD and NHibernate?
Is it possible to mock ISession to unit test the actual queries or can this only be done with in memory databases?
I am using the Repository Pattern to perform Database operations, and whenever I run my Tests I just run the higher-level tests that simply Mock the Repository (with RhinoMocks).
I have a seperate suite of tests that explicitly tests the Repository layer and the NHibernate mappings. And those usually don't change as much as the business and gui logic above them.
That way I get very fast UnitTests that never hit the DB, and still a well tested DB Layer
Unit testing data access is not possible, but you can integration test it.
I create integration test for my data access in a seperate project from my unit tests. I only run the (slow) integration tests when I change something in the repositories, mapping or database schema.
Because the integration tests are not mixed with the unit tests, I can still run the unit tests about 100 times a day without getting annoyed.
See http://www.autumnofagile.net and http://www.summerofnhibernate.com
Have you tried changing some of the defaults in the optional configuration properties? The slowdown is most likely related to certain optimizations nhibernate does with code generation.
http://nhibernate.info/doc/nh/en/index.html#configuration-optional
It seems like an in memory db is going to be the fastest way to test a your data layer. It also seems once you start testing your data layer you're moving a little beyond the realm of a unit test.