I'm setting up some unit tests in Codeception and need to create an instance of an Object. I can do this in _before but this then creates a new instance before every test. I have tried to use _beforeSuite but the constructor for the Object requires an environment variable, from my understanding this won't work as beforeSuite is ran before bootstrap? When I try this out I seem to be getting null instead of the variable.
I am new to testing so I am curious if it is okay to create the Object in _before or if I should be using something else?
What you should strive for in testing in general is, that when executing your system-under-test (SUT) this happens in a clearly defined context. During test execution you want all the aspects that could influence the execution of your SUT under control. Re-using objects between tests is therefore (normally) not advisable, because a previous test might have made modifications to the objects. Which may then have an impact on the results of the later tests. The advice against sharing objects between tests holds even if you know the exact order in which tests will be executed (tests should be independent - lots of information about this on the web, e.g. Why in unit testing tests should not depend on the order of execution?).
Therefore, unless in exceptional circumstances, prefer that you have a fresh object for every single test. You can have it created in _before, but it might even be better (for readability purposes) to create it directly within each test case that needs it.
I have a application which is using two different database for replication. I want to write test cases for this application.
I can not used Mock framework as it is not using interfaces. We need to check that If there is record in one database table and application run then it should move records to other database. We also want that after execution values should not be saved in database. Operation should be rolled back after test case is complete.
From how you phrase your question it is not quite clear if the tests you want to perform are unit-tests or integration-tests. The distinction is not so much on whether or not you isolate your code, but rather what the goal of the test is.
Assumption A: Your test goal is to test the logic of your own code. The dependency on the data bases is a nuisance, and you would rather have your code isolated from the data bases. The access to the data bases, however, is not via interfaces (in the sense of a programming language construct), but goes to the data base APIs directly, possibly even to global instances of the data bases. And, you have not the possibility to change the code.
Depending on the programming language and environment you use there may be other ways to replace the data bases with mocks, stubs or whatever: For example, you don't need to link against the original data base lib. Instead, you can create a mocked data base lib to link against ("linker seam") which you can control from your test code. Or, you could replace calls to the db with the help of the preprocessor (in C or C++) to, again, a mocked implementation.
Assumption B: Your test goal is to ensure that the interaction between your code and the two data bases works as expected. For example, you want to be sure that the elements are truly transferred from the one data base to the other.
This would be an integration test scenario, and mocking the data bases would not be a useful step here, for the following reason: If you made wrong assumptions how to interact with the data base, the mocks you create only again reflect your wrong assumptions.
Your problem under assumption B is more how you can ensure that each test case starts and exits with a clean state. How to achieve this is hard to judge from the little information you provided. You could work with backups of the data base to do the rollback etc., but the options depend a lot on your specific situation.
I have been tasked with testing an existing application, and I've done so with my own in the past, but I am a little unsure of whether I am using the right approach.
Let's say I have the following example:
Class users has a function called getAll() which returns the list of users by querying the database. There are (as far as I can tell) two straightforward ways to test this functionality:
Use the same query code that the getAll() function uses, independently in the unit test. Then, perform the getAll() call and assert the results.
Have a data provider give the unit test data that is of the format that is expected, hard-coded to values we know are contained in the database. For example, we would pass a hard-coded array containing all users in the format the the getAll() function expects, and assert the results.
Although the second approach seems more in line with testing, it has the added results of adding a layer of security risk. Anyone with access to the test suite would also have plaintext access to a lot of potentially sensitive data. On top of that, depending on database size queries like the above could make the tests ridiculously long for very little reason.
Which would be the preferred approach?
In the both approaches you suggest to work with the real DB and compare the results. I think that the proper unit test should be able to verify the function behaviour without connection to the real resource. I would suggest to change the connection string (that may be the function parameter) to some test database with the similar users table and verify the function behaviour. In this way you will not meet any security risk and the unit test (or tests) will be independent.
I'm just getting stated with boost-test and unit testing in general with a new application, and I am not sure how to handle the applications initialisation (eg loading config files, connecting to a database, starting an embedded python interpretor, etc).
I want to test this initialisation process, and also most of the other modules in the application require that the initialisation occurred successfully.
Some way to run some shut down code would also be appreciated.
How should I go about doing this?
It seems what you intent to do is more integration test than unit-test. It's not to pinpoint on wording, but it makes a difference. Unit testing mean testing methods in isolation, in an environment called a fixture, created just for one test, end then deleted. Another instance of the fixture will be re-created if the next case require the same fixture. This is done to isolate the tests so that an error in one test does not affect the outcome of the subsequent tests.
Usually, one test has three steps:
Arrange - prepare the fixture : instantiate the class to be tested, possibly other objects needed
Act - call the method to be tested
Assert - verify the expectations
Unit tests typically stays away of external resources such as files and databases. Instead mock objects are used to satisfy the dependencies of the class to be tested.
However, depending on the type of your application, you can try to run tests from the application itself. This is not "pure" unit testing, but can be valuable anyway, especially if the code has not been written with unit testing in mind, it might not be "flexible" enough to be unit tested.
This need a special execution mode, with a "-test" parameter for instance, which will initialize the application normally, and then invoke tests that will simulate inputs and use assertions to verify the application reacted as expected. Likewise, it might be possible to invoke the shutdown code and verify with assertions if the database connection has be closed (if the objects are not deleted).
This approach has several drawbacks compared to unit tests: it depends on the config files (the software may behave differently depending on the parameters), on the database (on its content and on the ability to connect to it), the tests are not isolated ... The two first can be overcome using default values for the configuration and connecting to a test database in test mode.
Are you defining BOOST_TEST_MAIN? If so, and you have no main function of your own (where you would otherwise put initialization code) you could feasibly use some form of singleton object that exposes an init function that you can call before each test if required.
I have a .NET application with a web front-end, WCF Windows service back-end. The application is fairly simple - it takes some user input, sending it to the service. The service does this - takes the input (Excel spreadsheet), extracts the data items, checks SQL DB to make sure the items are not already existing - if they do not exist, we make a real-time request to a third party data vendor and retrieve the results, inserting them into the database. It does some logging along the way.
I have a Job class with a single public ctor and public Run() method. The ctor takes all the params, and the Run() method does all of the above logic. Each logical piece of functionality is split into a separate class - IParser does file parsing, IConnection does the interaction with the data vendor, IDataAccess does the data access, etc. The Job class has private instances of these interfaces, and uses DI to construct the actual implementations by default, but allows the class user to inject any interface.
In the real code, I use the default ctor. In my unit tests for the Run() method, I use all mock objects creating via NMock2.0. This Run() method is essentially the 'top level' function of this application.
Now here's my issue / question: the unit tests for this Run() method are crazy. I have three mock objects I'm sending into the ctor, and each mock object sets expectations on themselves. At the end I verify. I have a few different flows that the Run method can take, each flow having its own test - it could find everything is already in the database and not make a request to vendor... or an exception could be thrown and the job status could be set to 'failed'... OR we can have the case where we didn't have the data and needed to make the vendor request (so all those function calls would need to be made).
Now - before you yell at me and say 'your Run() method is too complicated!' - this Run method is only a mere 50 lines of code! (It does make calls to some private function; but the entire class is only 160 lines). Since all the 'real' logic is being done in the interfaces that are declared on this class. however, the biggest unit test on this function is 80 lines of code, with 13 calls to Expect.BLAH().. _
This makes re-factoring a huge pain. If I want to change this Run() method around, I have to go edit my three unit tests and add/remove/update Expect() calls. When I need to refactor, I end up spending more time creating my mock calls than I did actually writing the new code. And doing real TDD on this function makes it even more difficult if not impossible. It's making me think that it's not even worth unit testing this top level function at all, since really this class isn't doing much logic, it's just passing around data to its composite objects (which are all fully unit tested and don't require mocking).
So - should I even bother testing this high level function? And what am I gaining by doing this? Or am I completely misusing mock/stub objects here? Perhaps I should scrap the unit tests on this class, and instead just make an automated integration test, which uses the real implementations of the objects and Asserts() against SQL Queries to make sure the right end-state data exists? What am I missing here?
EDIT: Here's the code - the first function is the actual Run() method - then my five tests which test all five possible code paths. I changed it some for NDA reasons but the general concept is still there. Anything you see wrong with how I'm testing this function, any suggestions on what to change to make it better? Thanks.
I guess my advice echos most of what is posted here.
It sounds as if your Run method needs to be broken down more. If its design is forcing you into tests that are more complicated than it is, something is wrong. Remember this is TDD we're talking about, so your tests should dictate the design of your routine. If that means testing private functions, so be it. No technological philosophy or methodology should be so rigid that you can't do what feels right.
Additionally, I agree with some of the other posters, that your tests should be broken down into smaller segments. Ask yourself this, if you were going to be writting this app for the first time and your Run function didn't yet exist, what would your tests look like? That response is probably not what you have currently (otherwise you wouldn't be asking the question). :)
The one benefit you do have is that there isn't a lot of code in the class, so refactoring it shouldn't be very painful.
EDIT
Just saw you posted the code and had some thoughts (no particular order).
Way too much code (IMO) inside your SyncLock block. The general rule is to keep the code to a minimal inside a SyncLock. Does it ALL have to be locked?
Start breaking code out into functions that can be tested independently. Example: The ForLoop that removes ID's from the List(String) if they exist in the DB. Some might argue that the m_dao.BeginJob call should be in some sort of GetID function that can be tested.
Can any of the m_dao procedures be turned into functions that can tested on their own? I would assume that the m_dao class has it's own tests somewhere, but by looking at the code it appears that that might not be the case. They should, along with the functionality in the m_Parser class. That will relieve some of the burden of the Run tests.
If this were my code, my goal would be to get the code to a place where all the individual procedure calls inside Run are tested on their own and that the Run tests just test the final out come. Given input A, B, C: expect outcome X. Give input E, F, G: expect Y. The detail of how Run gets to X or Y is already tested in the other procedures' tests.
These were just my intial thoughts. I'm sure there are a bunch of different approaches one could take.
Two thoughts: first you should have an integration test anyway to make sure everything hangs together. Second, it sounds to me like you're missing intermediate objects. In my world, 50 lines is a long method. It's hard to say anything more precise without seeing the code.
The first thing I would try would be refactroing your unit tests to share the set up code between tests by refactoring to a method that sets up the mocks and expectations. Parameterize the method so your expectations are configurable. You may need one or perhaps more of these set up methods depending on how much alike the set up is from test to test.
So - should I even bother testing this
high level function?
Yes. If there are different code-paths, you should be.
And what am I gainging by doing this?
Or am I completely mis-using mock/stub
objects here?
As J.B. pointed out (Nice seeing you at AgileIndia2010!), Fowler's article is recommended read. As a gross simplification: Use Stubs, when you don't care about the values returned by the collaborators. If you the return values from the collaborator.call_method() changes the behavior(or you need non trivial checks on args, computation for return values), you need mocks.
Suggested refactorings:
Try moving the creation and injection of mocks into a common Setup method. Most unit testing frameworks support this; will be called before each test
Your LogMessage calls are beacons - calling out once again for intention revealing methods. e.g. SubmitBARRequest(). This will shorten your production code.
Try n move each Expect.Blah1(..) into intention revealing methods.
This will shorten your test code and make it immensely readable and easier to modify. e.g.
Replace all instances of
.
Expect.Once.On(mockDao) _
.Method("BeginJob") _
.With(New Object() {submittedBy, clientID, runDate, "Sent For Baring"}) _
.Will([Return].Value(0));
with
ExpectBeginJobOnDAO_AndReturnZero(); // you can name it better
on whether to test such function: you said in a comment
" the tests read just like the actual
function, and since im using mocks,
its only asserting the functions are
called and sent params (i can check
this by eyeballing the 50 line
function)"
imho eyeballing the function isn't enough, haven't you heard: "I can't believe I missed that!" ... you have a fair amount of scenarios that could go wrong in that Run method, covering that logic is a good idea.
on tests being brittle: try having some shared methods that you can use in the test class for the common scenarios. If you are concerned about a later change breaking all the tests, put the pieces that concerned you in specific methods that can be changed if needed.
on tests being too long / hard to know what's in there: don't test single scenarios with every single assertion that its related to it. Break it up, test stuff like it should log x messages when y happens (1 test), it should save to the db when y happens (another separate test), it should send a request to a third party when z happens (yet another test), etc.
on doing integration/system tests instead of these unit tests: you can see from your current situation that there are plenty of scenarios & little variations involved in that part of your system. That's with the shield of replacing yet more logic with those mocks & the ease of simulating different conditions. Doing the same with the whole thing will add a whole new level of complexity to your scenario, something that is surely unmanageable if you want to cover a wide set of scenarios.
imho you should minimize the combinations that you are leaving for your system tests, exercising some main scenarios should already tell you that a Lot of the system is working correctly - it should be a lot about everything being hooked correctly.
The above said, I do recommend adding focused integration tests for all the integration code you have that might not be currently covered by your tests / since by definition unit tests don't get there. This exercises specifically the integration code with all the variations you expect from it - the corresponding tests are much simpler than trying to reach those behaviors in the context of the whole system & tell you very quickly if any assumptions in those pieces is causing trouble.
If you think unit-tests are too hard, do this instead: add post-conditions to the Run method. Post-conditions are when you make an assertion about the code. For example, at the end of that method, you may want some variable to hold a particular value or one value out of some possible choices.
After, you can derive your pre-conditions for the method. This is basically the data type of each parameter and the limits and constraints on each of those parameters (and on any other variable initialized at the beginning of the method).
In this way, you can be sure both the input and output are what is desired.
That probably still won't be enough so you will have to look at the code of the method line by line and look for large sections that you want to make assertions about. If you have an If statement, you should check for some conditions before and after it.
You won't need any mock objects if you know how to check if the arguments to the object are valid and you know what range of outputs are desired.
Your tests are too complicated.
You should test aspects of your class rather than writing a unittest for each member of yor class. A unittest should not cover the entire functionality of a member.
I'm going to guess that each test for Run() set expectations on every method they call on the mocks, even if that test doesn't focus on checking every such method invocation. I strongly recommend you Google "mocks aren't stubs" and read Fowler's article.
Also, 50 lines of code is pretty complex. How many codepaths through the method? 20+? You might benefit from a higher level of abstraction. I'd need to see code to judge more certainly.