(Leaving aside hair-splitting about if this is integration-testing or unit-testing.)
I would rather first test at the large scale. If my app writes a VRML file that is the same as the reference one then the VRML exporter works, I don't then have to run unit tests on every single statement in the code. I would also like to use this do some level of poor-man gui testing by comparing screenshots.
Is there a unit test framework (for C++ ideally) that integrates this sort of testing - or at least makes it easy to integrate with unit tests?
edit. It seems a better term is approval testing. So are there any other unit test frameworks that incorporate Approval Testing ?
Have a look at Approval Tests, written by a couple of friends of mine. Not C++, but it's the general idea of what you're after, also known as Golden Master tests. Good stuff, whether it's unit tests or not.
Kitware, for VTK, uses CDash to do most of its testing. Many of its tests are similar in nature to this - they write out an image of the rendered model, and compare it to a reference image.
In addition, they have code in there to specifically handle very subtle differences to the reference image due to different graphics card drivers/manufacturers/etc. The tests can be written in a way to compare the reference image with some tolerance.
Okay, I think you're making an incorrect assumption about the nature of unit test code; your statement that
If my app writes a VRML file that is
the same as the reference one then the
VRML exporter works, I don't then have
to run unit tests on every single
statement in the code.
is strictly correct if you're looking to do a validation test on your code, but note that this type of test is strictly different than what a unit test actually is. Unit tests are for testing individual units of code; they do not exist for verification purposes. Depending on your environment, you may not need unit tests at all, but please keep in mind that validation tests (testing the validity of the overall program output) and unit tests (testing that the individual code units work as expected) are completely different things.
(Note that I'm really not trying to be nitpicky about this; also, you can use plenty of unit test frameworks to achieve this result; keep in mind, though, that what you're writing aren't really "Unit Tests", despite running them in a Unit Test framework.)
Related
We use SWTBot for writing of functional tests. To test some cases is very difficult and some programmers use classes and their methods directly from implementation (for example call methods from class AddUserDialog etc.). Is this good approach? And why?
And next qustion please. It is SWTBot enough for testing of eclipse RCP based application? Is is necessary to write unit tests please?
Note: We are scrum team.
SWTBot and JUnit serve two different purposes.
JUnit
As the name implies, JUnit is meant for unit testing. Unit tests should be small and fast to execute. They test only a single unit of code and the above mentioned attributes allow them to be executed often while developing the unit under test.
But there is more to (good) units tests. You may want to read one of the following posts for further attributes of unit tests:
Key qualities of a good unit test
What attribute should a good Unit-Test have?
I would go one step further and say that unit tests only make sense in TDD, that is you write the test before the production code. Otherwise you neglect the tests. Who want's to do the extra effort of writing tests for something that already works. And even if you have the discipline to write the tests afterwards, they merely manifest the state of your production code. Whereas, like in TDD, writing tests beforehand leads to lean production code that only does what is required by the tests.
But I guess that's something not everyone will agree on.
In an RCP setting, unit tests would ideally be able to run without starting the platform (which takes rather long). I.e. they would not require to be run as PDE JUnit Tests but as plain JUnit Tests instead. Therefore the unit under test should be isolated from the RCP APIs.
On a related note, see also this question: How to efficiently JUnit test Eclipse RCP Plugins
SWTBot
While SWTBot uses the JUnit runtime to execute the tests, it is rather meant as a utility to create integration or functional tests. SWTBot, when used with RCP, starts the entire workbench and runs all tests within the same instance. Therefore great care should be taken to ensure that each test leaves the environment in the same state as it was before the test started. Specialized Rules may help here to set up and tear down a particular recurring scenario.
It is perfectly valid in order to setup an SWTBot test to call methods from your application. For example, you could programmatically open the wizard and then use SWTBot to simulate a user that enters data and presses the OK button. There is no need to use SWTBot to laboriously open the wizard itself.
In my experience, SWTBot is even too much for simple use cases. Consider a test that should enter some data into a dialog and then press OK. If you already have the dialog opened programmatically you can as well continue without SWTBot:
dialog.textField.setText( "data" );
dialog.okButton.notifyListeners( SWT.Selection, null );
assertThat( dialog.getEnteredData() ).isEqualTo( "data" );
Use Both
The best bet is to have both, unit tests that ensure the behavior of the respective units and functional tests that make sure that the particular units play together as desired.
Not sure if that answers the question, if you have further concerns please leave a comment.
Let's assume we are not doing TDD (for which unit tests are obviously part and parcel), and have integration tests for all the use cases.
The integration tests assume assume a certain input and validate the output is as expected.
My thinking is that adding a unit test for a method that is traversed in an integration test, using the same data as would exist in the method in the integration test, would not expose any additional bugs.
That would lead to the conclusion that provided you have suffcient integration tests you do not then need to unit test the same code.
So, can someone give a concrete example where a unit test could expose a bug in the above scenario?
Integration tests can be seen as a form of Acceptance Testing. They ensure that the software is doing what it is supposed to be doing.
Unit tests, on the other hand, aren't particularly useful for customers. A customer is not concerned that the InitializeServerConnection is failing, but they are concerned that they're unable to send internal messages to their co-workers as a result.
So what good are unit tests for? They are a development tool, full stop. A unit test verifies that a cog in the machine is working properly. And if it is not, it is very easy to see it failing.
Arialdo Martini offers a great explanation:
Oversimplifying, a software system can be seen as a network of cooperating modules. Since they cooperate, some of them depend on other.
[...]
With integration and end-to-end tests you would be able to find all the broken features.
Yet, this is not of any help in guessing where the bug is. The same system, with the same bug, would result in these unit test failures:
So, even though a unit test doesn't add any business value, it does add value in the form of reducing the amount of time spent manually testing, debugging, and sifting through code looking for the root cause of an issue.
I have often seen tests where canned inputs are fed into a program, one checks the outputs generated against canned (expected) outputs usually via diff. If the diff is accepted, the code is deemed to pass the test.
Questions:
1) Is this an acceptable unit test?
2) Usually the unit test inputs are read in from the file system and are big
xml files (maybe they represent a very large system). Are unit tests supposed
to touch the file system? Or would a unit test create a small input on the fly
and feed that to the code to be tested?
3) How can one refactor existing code to be unit testable?
Output differences
If your requirement is to produce output with certain degree of accuracy, then such tests are absolutely fine. It's you who makes the final decision - "Is this output good enough, or not?".
Talking to file system
You don't want your tests to talk to file system in terms of relying on some files to exists somewhere in order for your tests to work (for example, reading values from configuration files). It's a bit different with tests input resources - you can usually embed them in your tests (or at least test project), treat them as part of codebase, and on top of that they usually should be loaded before test executes. For example, when testing rather large XMLs it's reasonable to have them stored as separete files, rather than strings in code files (which sometimes can be done instead).
Point is - you want to keep your tests isolated and repeatable. If you can achieve that with file being loaded at runtime - it's probably fine. However it's still better to have them as part of codebase/resources than standard system file lying somewhere.
Refactoring
This question is fairly broad, but to put you in the right direction - you want to introduce more solid design, decouple objects and separate responsibilities. Better design will make testing easier and, what's most important - possible. Like I said, it's broad and complex topic, with entire books dedicated to it.
1) is this an acceptable unit test?
This is not a unit test by definition. A unit test focuses on the smallest possible amount of code. Your test can still be a useful test, regression test, self-documenting test, TDD test, etc. It is just not a unit test, although it may be equally useful.
2) Are unit tests supposed to touch the file system?
Typically not, unless you need to unit test something explicitly related to the filesystem. One reason is, if you have several hundred unit tests it is nice to have them run in a couple seconds rather than minutes.
3) How can one refactor existing code to be unit testable?
A better question is why do you want the code to be unit testable? If you are trying to learn TDD it is better to start with a new project. If you have bugs then try to write tests for the bugs. If the design is slowing you down then you can refactor towards testability over time.
Addressing only the 3rd question. It is extremely difficult. You really need to write tests at the same time you write the code, or before. It is a nightmare to try to slap tests onto an existing code base, and it is often more productive to throw away the code and start over.
This is an acceptable unit test.
The files being read should be part of the test project so anyone that checks out the project from repository will have the same files at the same relative location.
Having black box tests is a great start, you can refactor the existing code and use the current tests to verify that it is still working (depending on the quality of the tests). Here is a short blog about refactoring for testability: http://www.beletsky.net/2011/02/refactoring-to-testability.html
A diff test can be acceptable as a Unit Tests, especially when your using test data that is shared between Unit Tests.
If you don't know how many items there are in the SUT you could use the following:
int itemsBeforeTest = SUT.Items.Count;
SUT.AddItem();
Assert.AreEqual(itemsBeforeTest + 1, SUT.Items.Count);
If a Unit Tests requires so much data that it needs to be read from a big XML file, it's not a real Unit Test. A Unit Test should test a class in complete isolation and mock out all dependencies.
Using a pattern like the Builder pattern, can also help in creating test data for your unit test. The biggest problem with having your test data in a separate file, is that it's hard to understand what the test does exactly. If you create your test data in the arrange part of your unit test, it's immediately clear what is important for your test.
For example, let's say you have the following arrange code to test if the price of an invoice is correct:
Address billingAddress = new Address("Stationsweg 9F",
"Groningen", "Nederland", "9726AE"); shippingAddress = new Address("Aweg 1",
"Groningen", "Nederland", "9726AB");
Customer customer = new Customer(99, "Piet", "Klaassens",
30,
billingAddress,
shippingAddress);
Product product = new Product(88, "Tafel", 19.99);
Invoice invoice = new Invoice(customer);
Can be changed to the following when using a Builder
Invoice invoice = Builder<Invoice>.CreateNew()
.With(i => i.Product = Builder<Product>.CreateNew()
.With(p => p.Price = 19.99)
.Build())
.Build();
When using a Builder its much easier to see what is important and your code is also more maintainable.
Refactoring code to become more testable is a broad topic. It comes down to thinking about 'how would I test this code?' while you are writing the code.
Take the following example:
public class AlarmClock
{
public AlarmClock()
{
SatelliteSyncService = new SatelliteSyncService();
HardwareClient = new HardwareClient();
}
}
This is hard to test. You need to make sure that both the SatteliteSyncService and the HardwareClient are functional when testing the AlarmClock.
This change tot the constructor makes it much easier to test:
public AlarmClock(IHardwareClient hardwareClient, ISatelliteSyncService satelliteSyncService)
{
SatelliteSyncService = satelliteSyncService;
HardwareClient = hardwareClient;
}
Techniques like Dependency Injection help with refactoring your code to be more testable. Also watch out for static values like DateTime.Now or the use of a Singleton because they are hard to test.
A really good introduction to writing testable code can be found here.
You should not require the code to be refactored to be able to perform unit tests. Unit tests, as the name implies, are testing a unit of code for the system. The best unit tests are small, quick to execute and exercise only a very small subset of the piece of code being tested (e.g. class).
The reason for having small, compact unit tests that only exercise one part of the code is that the objective of unit tests is to find bugs in that unit of code. If the unit test takes a long time to execute and tests lots of things it makes the identification of a bug in the code that much harder.
As to accessing the file system, I see no problem. Some unit tests may require a database to be constructed before the test is carried out, the output to be checked that would be difficult or time expensive to write the checks in code.
The files for unit testing should be treated like the rest of the code - put under version control. If you are paranoid you could implement a check within the unit test such as perform a MD5 on it and check against a hard coded value so future reruns of the test can verify that the test data has not inadvertenly changed.
Just my humble thoughts.
Is there anyone that can clearly define these levels of testing as I find it difficult to differentiate when doing TDD or unit testing. Please if anyone can elaborate how, when to implement these?
Briefly:
Unit testing - You unit test each individual piece of code. Think each file or class.
Integration testing - When putting several units together that interact you need to conduct Integration testing to make sure that integrating these units together has not introduced any errors.
Regression testing - after integrating (and maybe fixing) you should run your unit tests again. This is regression testing to ensure that further changes have not broken any units that were already tested. The unit testing you already did has produced the unit tests that can be run again and again for regression testing.
Acceptance tests - when a user/customer/business receive the functionality they (or your test department) will conduct Acceptance tests to ensure that the functionality meets their requirements.
You might also like to investigate white box and black box testing. There are also performance and load testing, and testing of the "'ilities" to consider.
Unit test: when it fails, it tells you what piece of your code needs to be fixed.
Integration test: when it fails, it tells you that the pieces of your application are not working together as expected.
Acceptance test: when it fails, it tells you that the application is not doing what the customer expects it to do.
Regression test: when it fails, it tells you that the application no longer behaves the way it used to.
Here's a simple explanation for each of the mentioned tests and when they are applicable:
Unit Test
A unit test is performed on a self-contained unit (usually a class or method) and should be performed whenever a unit has been implemented or updating of a unit has been completed.
This means it's run whenever you've written a class/method, fixed a bug, changed functionality...
Integration Test
Integration test aims to test how well several units interact with each other. This type of test should be performed Whenever a new form of communications has been established between units or the nature of their interaction have changed.
This means it's run whenever a recently written unit is integrated into the rest of the system or whenever a unit which is interacts with other systems has been updated (and successfully completed its unit tests).
Regression Test
Regression tests are performed whenever anything has been changed in the system, in order to check that no new bugs have been introduced.
This means it's run after all patches, upgrades, bug fixes. Regression testing can be seen as a special case of combined unit test and integration test.
Acceptance Test
Acceptance tests are performed whenever it is relevant to check that a subsystem (possibly the entire system) fulfils its entire specifications.
This means it's mainly run before finishing a new deliverable or announcing completion of a larger task. See this as your final check to see that you've really completed your goals before running to the client/boss and announcing victory.
This is at least the way I learned, though I'm sure there are other opposing views. Either way, I hope that helps.
I'll try:
Unit test: a developer would write one to test an individual component or class.
Integration test: a more extensive test that would involve several components or packages that need to collaborate
Regression test: Making a single change to an application forces you to re-run ALL the tests and check out ALL the functionality.
Acceptance test: End users or QA do these prior to signing off to accept delivery of an application. It says "The app met my requirements."
Unit Test: is my single method working correctly? (NO dependencies, or dependencies mocked)
Integration Test: are my two separately developed modules working corectly when put together?
Regression Test: Did I break anything by changing/writing new code? (running unit/integration tests with every commit is technically (automated) regression testing). More often used in context of QA - manual or automated.
Acceptance Test: testing done by client, that he "accepts" the delivered SW
Can't comment (reputation to low :-| ) so...
#Andrejs makes a good point around differences between environments associated with each type of testing.
Unit tests are run typically on developers machine (and possibly during CI build) with mocked out dependencies to other resources/systems.
Integration tests by definition have to have (some degree) of availability of dependencies; the other resources and systems being called so the environment is more representative. Data for testing may be mocked or a small obfuscated subset of real production data.
UAT/Acceptance testing has to represent the real world experience to the QA and business teams accepting the software. So needs full integration and realistic data volumes and full masked/obfuscated data sets to deliver realistic performance and end user experience.
Other "ilities" are also likely to need the environment to be as close as possible to reality to simulate the production experience e.g. performance testing, security, ...
What is the difference between unit tests and functional tests? Can a unit test also test a function?
Unit tests tell a developer that the code is doing things right; functional tests tell a developer that the code is doing the right things.
You can read more at Unit Testing versus Functional Testing
A well explained real-life analogy of unit testing and functional testing can be described as follows,
Many times the development of a system is likened to the building of a house. While this analogy isn't quite correct, we can extend it for the purposes of understanding the difference between unit and functional tests.
Unit testing is analogous to a building inspector visiting a house's construction site. He is focused on the various internal systems of the house, the foundation, framing, electrical, plumbing, and so on. He ensures (tests) that the parts of the house will work correctly and safely, that is, meet the building code.
Functional tests in this scenario are analogous to the homeowner visiting this same construction site. He assumes that the internal systems will behave appropriately, that the building inspector is performing his task. The homeowner is focused on what it will be like to live in this house. He is concerned with how the house looks, are the various rooms a comfortable size, does the house fit the family's needs, are the windows in a good spot to catch the morning sun.
The homeowner is performing functional tests on the house. He has the user's perspective.
The building inspector is performing unit tests on the house. He has the builder's perspective.
As a summary,
Unit Tests are written from a programmers perspective. They are made to ensure that a particular method (or a unit) of a class performs a set of specific tasks.
Functional Tests are written from the user's perspective. They ensure that the system is functioning as users are expecting it to.
Unit Test - testing an individual unit, such as a method (function) in a class, with all dependencies mocked up.
Functional Test - AKA Integration Test, testing a slice of functionality in a system. This will test many methods and may interact with dependencies like Databases or Web Services.
A unit test tests an independent unit of behavior. What is a unit of behavior? It's the smallest piece of the system that can be independently unit tested. (This definition is actually circular, IOW it's really not a definition at all, but it seems to work quite well in practice, because you can sort-of understand it intuitively.)
A functional test tests an independent piece of functionality.
A unit of behavior is very small: while I absolutely dislike this stupid "one unit test per method" mantra, from a size perspective it is about right. A unit of behavior is something between a part of a method and maybe a couple of methods. At most an object, but not more than one.
A piece of functionality usually comprises many methods and cuts across several objects and often through multiple architectural layers.
A unit test would be something like: when I call the validate_country_code() function and pass it the country code 'ZZ' it should return false.
A functional test would be: when I fill out the shipping form with a country code of ZZ, I should be redirected to a help page which allows me to pick my country code out of a menu.
Unit tests are written by developers, for developers, from the developer's perspective.
Functional tests may be user facing, in which case they are written by developers together with users (or maybe with the right tools and right users even by the users themselves), for users, from the user's perspective. Or they may be developer facing (e.g. when they describe some internal piece of functionality that the user doesn't care about), in which case they are written by developers, for developers, but still from the user's perspective.
In the former case, the functional tests may also serve as acceptance tests and as an executable encoding of functional requirements or a functional specification, in the latter case, they may also serve as integration tests.
Unit tests change frequently, functional tests should never change within a major release.
TLDR:
To answer the question: Unit Testing is a subtype of Functional Testing.
There are two big groups: Functional and Non-Functional Testing. The best (non-exhaustive) illustration that I found is this one (source: www.inflectra.com):
(1) Unit Testing: testing of small snippets of code (functions/methods). It may be considered as (white-box) functional testing.
When functions are put together, you create a module = a standalone piece, possibly with a User Interface that can be tested (Module Testing). Once you have at least two separate modules, then you glue them together and then comes:
(2) Integration Testing: when you put two or more pieces of (sub)modules or (sub)systems together and see if they play nicely together.
Then you integrate the 3rd module, then the 4th and 5th in whatever order you or your team see fit, and once all the jigsaw pieces are placed together, comes
(3) System Testing: testing SW as a whole. This is pretty much "Integration testing of all pieces together".
If that's OK, then comes
(4) Acceptance Testing: did we build what the customer asked for actually? Of course, Acceptance Testing should be done throughout the lifecycle, not just at the last stage, where you realise that the customer wanted a sportscar and you built a van.
"Functional test" does not mean you are testing a function (method) in your code. It means, generally, that you are testing system functionality -- when I run foo file.txt at the command line, the lines in file.txt become reversed, perhaps. In contrast, a single unit test generally covers a single case of a single method -- length("hello") should return 5, and length("hi") should return 2.
See also IBM's take on the line between unit testing and functional testing.
According to ISTQB those two are not comparable. Functional testing is not integration testing.
Unit test is one of tests level and functional testing is type of testing.
Basically:
The function of a system (or component) is 'what it does'. This is
typically described in a requirements specification, a functional
specification, or in use cases.
while
Component testing, also known as unit, module and program testing,
searches for defects in, and verifies the functioning of software
(e.g. modules, programs, objects, classes, etc.) that are separately
testable.
According to ISTQB component/unit test can be functional or not-functional:
Component testing may include testing of functionality and specific non-functional characteristics such as resource-behavior (e.g. memory leaks), performance or robustness testing, as well as structural testing (e.g. decision coverage).
Quotes from Foundations of software testing - ISTQB certification
In Rails, the unit folder is meant to hold tests for your models, the functional folder is meant to hold tests for your controllers, and the integration folder is meant to hold tests that involve any number of controllers interacting. Fixtures are a way of organizing test data; they reside in the fixtures folder. The test_helper.rb file holds the default configuration for your tests.
u can visit this.
very simply we can say:
black box: user interface test like functional test
white box: code test like unit test
read more here.
AFAIK, unit testing is NOT functional testing. Let me explain with a small example. You want to test if the login functionality of an email web app is working or not, just as a user would. For that, your functional tests should be like this.
1- existing email, wrong password -> login page should show error "wrong password"!
2- non-existing email, any password -> login page should show error "no such email".
3- existing email, right password -> user should be taken to his inbox page.
4- no #symbol in email, right password -> login page should say "errors in form, please fix them!"
Should our functional tests check if we can login with invalid inputs ? Eg. Email has no # symbol, username has more than one dot (only one dot is permitted), .com appears before # etc. ? Generally, no ! That kind of testing goes into your unit tests.
You can check if invalid inputs are rejected inside unit tests as shown in the tests below.
class LoginInputsValidator
method validate_inputs_values(email, password)
1-If email is not like string.string#myapp.com, then throw error.
2-If email contains abusive words, then throw error.
3-If password is less than 10 chars, throw error.
Notice that the functional test 4 is actually doing what unit test 1 is doing. Sometimes, functional tests can repeat some (not all) of the testing done by unit tests, for different reasons. In our example, we use functional test 4 to check if a particular error message appears on entering invalid input. We don't want to test if all bad inputs are rejected or not. That is the job of unit tests.
The way I think of it is like this: A unit test establishes that the code does what you intended the code to do (e.g. you wanted to add parameter a and b, you in fact add them, and don't subtract them), functional tests test that all of the code works together to get a correct result, so that what you intended the code to do in fact gets the right result in the system.
UNIT TESTING
Unit testing includes testing of smallest unit of code which usually are functions or methods. Unit testing is mostly done by developer of unit/method/function, because they understand the core of a function. The main goal of the developer is to cover code by unit tests.
It has a limitation that some functions cannot be tested through unit tests. Even after the successful completion of all the unit tests; it does not guarantee correct operation of the product. The same function can be used in few parts of the system while the unit test was written only for one usage.
FUNCTIONAL TESTING
It is a type of Black Box testing where testing will be done on the functional aspects of a product without looking into the code. Functional testing is mostly done by a dedicated Software tester. It will include positive, negative and BVA techniques using un standardized data for testing the specified functionality of product. Test coverage is conducted in an improved manner by functional tests than by unit tests. It uses application GUI for testing, so it’s easier to determine what exactly a specific part of the interface is responsible for rather to determine what a code is function responsible for.
Test types
Unit testing - In Procedural programming unit is a procedure, in Object oriented programming unit is a class. Unit is isolated and reflects a developer perspective
Functional testing - more than Unit. User perspective, which describes a feature, use case, story...
Integration testing - check if all separately developed components work together. It can be other application, service, library, database, network etc.
Narrow integration test - double[About] is used. The main purpose is to check if component is configured in a right way
Broad integration test (End to End test, System test) - live version. The main purpose is to check if all components are configured in a right way
UI testing - checks if user input triggers a correct action and the UI is changed when some actions are happened
...
Non functional testing - other cases
Performance testing - calculate a speed and other metrics
Usability testing - UX
...
[iOS tests]
[Android tests]
Unit Test:-
Unit testing is particularly used to test the product component by component specially while the product is under development.
Junit and Nunit type of tools will also help you to test the product as per the Unit.
**Rather than solving the issues after the Integration it is always comfortable to get it resolved early in the development.
Functional Testing:-
As for as the Testing is concerned there are two main types of Testing as
1.Functional Test
2.Non-Functional Test.
Non-Functional Test is a test where a Tester will test that The product will perform all those quality attributes that customer doesn't mention but those quality attributes should be there.
Like:-Performance,Usability,Security,Load,Stress etc.
but in the Functional Test:- The customer is already present with his requirements and those are properly documented,The testers task is to Cross check that whether the Application Functionality is performing according to the Proposed System or not.
For that purpose Tester should test for the Implemented functionality with the proposed System.
Unit testing is usually done by developers. The objective of doing the same is to make sure their code works properly. General rule of thumb is to cover all the paths in code using unit testing.
Functional Testing: This is a good reference. Functional Testing Explanation