we're currently working on testing JTA failure behaviour, on a system that receives messages using JMS, persists them, and sends results using another class.
The whole thing is tied together using Spring. Current unit tests use HSQLDB, Apache ActiveMQ and Bitronix for transaction management. Success with this has been limited, mostly because HSQLDB does not implement XA transactions.
So here is the question: how to best simulate database failures in a transaction unit test? Is there any way to make a standard JDBC driver (for Oracle, say) fail in the middle of a test?
n.b. pressing the power button is not a repeatable test :)
You need to decide what exactly do you want to test - for example if you want to test how Oracle would behave in XA transaction with Bitronix then mocking DAOs, as suggested by duffymo, is not going to help you. In such case you need to find a way to break connectivity in the middle of transaction and then see how Bitronix/Oracle would handle recovery - e.g. heuristic outcomes and so on.
Note that in quite a few cases there are ways to get the same functionality without actually using XA transactions. It could be simpler, faster and more testable. For example, in very common case when messages are consumed from MOM and DML executed in database there is a common pattern of how to get away without XA even so two resource managers are getting updated.
Write a mock object for the test whose implementation throws an exception in the middle of the transaction.
Since you're using Spring, it's an easy matter to write a new, test-only implementation of the DAO interface that behaves in a repeatable, predictable manner. Inject the 'wonky DAO' only for the test.
Of course you're using the XA driver to connect to the database. Two phase commit won't work otherwise.
Related
I have some doubts regarding the best approach to perform unit tests in Quarkus.
One option is using mocks, but I have a feeling that using mocks I just "make happy" the test coverage plugins, but on real I don't test anything with this approach.
Another option is using a real database, like embedded H2 database, but for this, I need to put the unit test in order (Insert, Get, Update, Delete), sometimes I'll need the inserted ID from another test to perform a delete operation, for example. There are some complex entities that create some difficulty to insert or delete. So, with this approach, I'll lose the concept of a unit test, because I'll lose the tests interdependency.
Does someone have a suggestion for this scenario?
Extra information: I'm using liquidbase, panache entity, junity.
Looks like you're looking for integration tests. I would probably go with the next options:
Depending on database type, you can either go with in-memory h2, or use testcontainers for external services.
Have test startup scripts for sql for general data, insert data directly for small test-cases.
Use tools like http://rest-assured.io/ (usually included in initial setup) to do real api calls
In case you really want unit tests, in this case 90% of the time you don't need to have the database to test functionality. Because of decoupling you probably have controllers(resources) separate from services. So in case of unit tests I would probably go with:
As stated above split logic from the networking layer, so that any Request handling and wrapping the output to satisfy clients is out of the logic layer. Split the logic itself into smaller chunks if possible, this will eliminate your main concern about complex data
For any dependent dependencies, either override them with test beans, or mock them using Mockito. This is especially important for persistency, you need to mock or fake any connection to database, you don't need to test this part because it's handled by libraries usually.
You can read JSON objects from files for really complex entities
If you really need to have persistency, look at the very first point in this answer.
So it feels like you want to check if your service is doing the job correctly on a higher level, which might not be a unit test, hope the answer will help
Testing the results of the endpoints is straight forward.
However, shouldn't I also have a look at the db layer, if (i.e.) the data I POSTed is saved correctly? I'm unclear how/if the business logic behind the REST call should be tested.
In my perfect world (someday I'd like to live there...) there are several different tests involved in fully validating a RESTful service I am providing.
There are unit tests for each of the logical layers of the application, that use dependency injection to mock the next lower layer and validate that each unit works correctly. This might be a unit that marshals/unmarshals the parameters and response, a unit that executes business logic, and a unit that manages persistence. (There could be multiple units, each with their own tests at each layer.) These tests have no dependencies outside the testing framework, with the exception of the persistence layer which has to exercise your persistence implementation.
There are also integration tests that require a running system. This is where you call the running service, and verify that you got the expected response. You may also inspect side effects of the call. On my team, we often do that by making a different service call (or calls) that relies on the result of the first call. That exercises more of the system. We find that direct inspection of the persistence layer for side effects seldom tells us much that we can't get by using a different service call.
Yes, you should test the db layer at some point but probably not in the same unit test than the one that test the result of the endpoint.
At the Unit Test level you would like to have small/quick/independant tests so you might end up with a test/suite for the REST part and another one focussing on db layer.
Then you might have some more integration/end-to-end tests that check the whole system : doing the request and checking the db is correctly updated.
My own experience, working as a QA, is that the tests involving DB are done with integration/end-to-end tests with a functional perspective by the QA using a third-party tool (we use Robot Framework for that).
One way of doing it is use the restful service itself to test that the data correctly persisted,
For example you can use the PUT message to persist an entity and use the GET operation to retrieve it, and make sure all properties are equal. This is more of an integration test which covers end to end.
Especially your application is doing more CRUD operations, and if you use this approach you could avoid creating tests for each and every layer (such as repositories)
I have a DAO service which retrieves and saves data to Hibernate.
Could anybody point me to testing Methodology using JUnit for such service.
What is the best practice? When should I have passed test failed test and should I test exception?
You should test every possible path in your DAO. You would definitely use an in-memory database for your tests, such as HSQLDB.
One strategy would be to give the database some initial data in your setUp() method and clean it up on tearDown(), so all tests have a consistent working environment.
With these setup you can basically do everything... For example, if you want to test a save() method in your DAO, just add the new element and make sure that your table now has one more element. Moreover, fetch that element and compare it to the one you've inserted, it should obviously be the same.
Remember that you should always test every possible path.
I would also look into using DBUnit to ensure that your database is consistent and test independent. They have articles on how to get started and a tutorial. There a quite a few articles on IBM's DeveloperWorks blog as well that you might want to read.
I think using an in-memory database is often a good idea - for unit test. But the ideal is to combine fast unit tests with such a database with integration tests hitting the real database. There are subtle differences between, e.g., Hypersonic SQL and most production servers.
Consider using mocks for testing your error handling (i.e. mock the Hibernate session object and make it throw exceptions for certain operations)
I have a REST web-service interface that calls-down to a service layer which orchestrates the creation, deletion, etc. of various objects in an entity-layer. These entity-layer objects ultimately map to database records. I have a number of unit tests (in nunit, it a c# application) that test this interface by sending http requests.
Consider my testing of a web service request that creates a some entity-layer object. I obviously want to verify that the web service considers the request to have been processed correctly, by checking the http status that it returns to me plus some data in the response body. I also want to independently verify that the correct database records have been created. I have a couple of ways (that I can think of) to do this:
The easiest way is to use existing 'reader' classes in the entity layer to read and validate the database entries. This is easy because they incorporate the validation and consistency logic for the entities they deal with, and using them is simple. I am uneasy about this, though, because I would be using the code I'm testing as part of the test. This seems to violate some principle of separation of concerns, and also introduce the possibility of an entity-layer bug causing the object creation to fail but appear to the unit test to have succeeded.
Alternatively, the test code could go straight to the database and do the checks itself. But then I'm embedding details about object storage and consistency rules in the test - which makes the test brittle if those details change, and also effectively means re-implementing, in the unit tests, the code I've already written in the entity layer.
I wonder what people think of the trade-offs involved with these (and maybe other) options, and what (if any) is the best practice? I'm not sure if there is a right or wrong answer, but I've wondered about it for a while and interested in other opinions.
EDIT
To clarify, I save separate test suites for the service-layer and the entity-layer. The concerns I have expressed -- using tested code in a test -- also apply to these tests.
We see two different tests, a test of the service methods and a test of the webclient.
For testing the service methods (like a reader), you may want to create a database with predefined values (test data), call the reader, and test, if the readers output matches the test data in the required way.
Once you've tested the service methods, you can move to the next test level and test the web client, again using the same test data but now testing if data shown on the web client matches the test data in the required way. On this test level, you can "trust" the readers (because they have been tested before).
Maybe you feel more comforable if you separate between "unit testing" and "integration testing". For unit testing, verify that a compilation unit works as required. This could be testing the reader: you populate the database with defined data, call findAll() (or something else), and assert that the test data and only the test data is in the result.
The other test is an integration test - there you verify, that service layer and entity layer work together as expected. Same with testing the web client: you verify that the client / service layer works as required.
And for integration tests I don't see any reason to not use (tested) service layer methods.
How are people unit testing their business applications? I've seen a lot of examples of unit testing with "simple to test" examples. Ex. a calculator. How are people unit testing data-heavy applications? How are you putting together your sample data? In many cases, data for one test may not work at all for another test which makes it hard to just have one test database?
Testing the data access portion of the code is fairly straightforward. It's testing out all the methods that work against the data that seem to be hard to test. For example, imagine a posting process where there is heavy data access to determine what is posted, numbers are adjusted, etc. There are a number of interim steps that occur (and need to be tested) along with tests afterwards that ensure the posting was successful. Some of those steps may actually be stored procedures.
In the past I've tried inserting the test data in a test database, then running the test, but honestly it's pretty painful to write this kind of code (and error prone). I've also tried just building a test database up front and rolling back the changes. That works OK but in a number of places you can't easily do this either (and many people would say that's integration testing; so be it, I still need to be able to test this somehow).
If the answer is that there isn't a nice way of handling this and it currently just sort of sucks, that would be useful to know as well.
Any thoughts, ideas, suggestions, or tips are appreciated.
My automated functional tests usually follow one of two patters:
Database Connected Tests
Mock Persistence Layer Tests
Database Connected Tests
When I have automated tests that are connected to the database, I usually make a single test database template that has enough data for all the tests. When the automated tests are run, a new test database is generated from the template for every test. The test database has to be constantly re-generated because test will often change the data. As tests are added, I usually append more data to the test database template.
There are some nice advantages to this testing method. The obvious advantage is that the tests also exercise your schema. Another advantage is that after setting up the initial tests, most new tests will be able to re-use the existing test data. This makes it easy to add more tests.
The downside is that the test database will become unwieldy. Because data will usually be added one test at time, it will be inconsistent and maybe even unrealistic. You will also end up cursing the person who setup the test database when there is a significant database schema change (which for me usually means I end up cursing myself).
This style of testing obviously doesn't work if you can't generate new test databases at will.
Mock Persistence Layer Tests
For this pattern, you create mock objects that live with the test cases. These mock objects intercept the calls to the database so that you can programmatically provide the appropriate results. Basically, when the code you're testing calls the findCustomerByName() method, your mock object is called instead of the persistence layer.
The nice thing about using mock object tests is that you can get very specific. Often times, there are execution paths that you simply can't reach in automated tests w/o mock objects. They also free you from maintaining a large, monolithic set of test data.
Another benefit is the lack of external dependencies. Because the mock objects simulate the persistence layer, your tests are no longer dependent on the database. This is often the deciding factor when choosing which pattern to choose. Mock objects seem to get more traction when dealing with legacy database systems or databases with stringent licensing terms.
The downside of mock objects is that they often result in a lot of extra test code. This isn't horrible because almost any amount of testing code is cheap when amortized over the number of times you run the test, but it can be annoying to have more test code then production code.
I have to second the comment by #Phil Bennett as I try to approach these integration tests with a rollback solution.
I have a very detailed post about integration testing your data access layer here
I show not only the sample data access class, base class, and sample DB transaction fixture class, but a full CRUD integration test w/ sample data shown. With this approach you don't need multiple test databases as you can control the data going in with each test and after the test is complete the transactions are all rolledback so your DB is clean.
About unit testing business logic inside your app, I would also second the comments by #Phil and #Mark because if you mock out all the dependencies your business object has, it becomes very simple to test your application logic one entity at a time ;)
Edit: So are you looking for one huge integration test that will verify everything from logic pre-data base / stored procedure run w/ logic and finally a verification on the way back? If so you could break this out into 2 steps:
1 - Unit test the logic that happens before the data is pushed
into your data access code. For
example, if you have some code that
calculates some numbers based on
some properties -- write a test that
only checks to see if the logic for
this 1 function does what you asked
it to do. Mock out any dependancy
on the data access class so you can
ignore it for this test of the
application logic alone.
2 - Integration test the logic that happens once you take your
manipulated data (from the previous
method we unit tested) and call the
appropriate stored procedure. Do
this inside a data specific testing
class so you can rollback after it's
completed. After your stored
procedure has run, do a query
against the database to get your
object now that we have done some
logic against the data and verify it
has the values you expected
(post-stored procedure logic /etc )
If you need an entry in your database for the stored procedure to run, simply insert that data before you run the sproc that has your logic inside it. For example, if you have a product that you need to test, it might require a supplier and category entry to insert so before you insert your product do a quick and dirty insert for a supplier and category so your product insert works as planned.
It depends on what you're testing. If you're testing a business logic component -- then its immaterial where the data is coming from and you'd probably use a mock or a hand rolled stub class that simulates the data access routine the component would have called in the wild. The only time I mess with the data access is when I'm actually testing the data access components themselves.
Even then I tend to open a DB transaction in the TestFixtureSetUp method (obviously this depends on what unit testing framework you might be using) and rollback the transaction at the end of the test suite TestFixtureTeardown.
Mocking Frameworks enable you to test your business objects.
Data Driven tests often end up becoming more of a intergration test than a unit test, they also carry with them the burden of managing the state of a data store pre and post execution of the test and the time taken in connecting and executing queries.
In general i would avoid doing unit tests that touch the database from your business objects. As for Testing your database you need a different stratergy.
That being said you can never totally get away from data driven testing only limiting the amout of tests that actually need to invoke your back end systems.
It sounds like you might be testing message based systems, or systems with highly parameterised interfaces, where there are large numbers of permutations of input data.
In general all the rules of standard unti testing still hold:
Try to make the units being tested as small and discrete as possible.
Try to make tests independant.
Factor code to decouple dependencies.
Use mocks and stubs to replace dependencies (like dataaccess)
Once this is done you will have removed a lot of the complexity from the tests, hopefully revealing good sets of unit tests, and simplifying the sample data.
A good methodology for then compiling sample data for test that still require complex input data is Orthogonal testing, or see here.
I've used that sort of method for generating test plans for WCF and BizTalk solutions where the permutations of input messages can create multiple possible execution paths.
For lots of different runs over the same logic but with different data you can use CSV, as many columns as you like for the input and the last for the output etc.