I'm fairly new to elixir and functional programming in general and I'm struggling to properly unit test functions that are composed of other functions. The general question is: when I have a function f that uses other functions g, h... internally, which approach should I take to test the whole?
Coming from the OOP world the first approach that comes to mind involves injecting the functions f depends of. I could unit test g, h... and inject all of those as arguments to f. Then, unit tests for f would just make sure it calls the injected functions as expected. This feels like overfitting though, and as an overall cumbersome approach that is against the functional mindset for which function composition should be a cheap thing to do and you should not be concerning yourself on passing all those arguments around the whole codebase.
I can also unit test g, h... as well as f by treating each of those as black boxes, which feels like the appropriate thing to do, but then the complexity of f's tests increases dramatically. Having simple tests that scale is one of the main purposes of unit testing.
To make the argument more concrete I'll put an example of a function that composes other functions inside and that I don't know how to unit test properly. This in particular is code for a plug that handles the creation of a resource in a RESTful fashion. Note that some of the "dependencies" are pure functions (such as validate_account_admin) but others are not (Providers.create):
def call(conn, _opts) do
account_uuid = conn.assigns.current_user.account["uuid"]
with {:ok, conn} <- Http.Authorization.validate_account_admin(conn),
{:ok, form_data} <- Http.coerce_form_data(conn, FormData),
{:ok, provider} <- Providers.create(FormData.to_provider(form_data), account_uuid: account_uuid) do
Http.respond_create(conn, Http.provider_path(provider))
else
{:error, reason, messages} -> Http.handle_error(conn, reason, messages)
end
end
Thanks!
Maybe this will be quite subjective answer, because there might be no perfect and ultimate one for such question.
Your assumption for me is wrong in terms of using public functions inside other public function. You shouldn't do that at all in business logic areas, because they should be separated and the only place where you can do that and - in fact - you has to is in controllers, but you test controllers with integration tests, not with unit tests, so all you care in such tests are proper and valid responses.
I like Erlang's explicit approach to declare which functions should be public by using export clause. In Elixir you should also follow this approach and whatever should be hidden in the module, should be declared with defp and defmacrop respectively for private functions and private macros.
Your unit tests should follow the rule of black box - you care about the output based on the input. That's all. Test is dumb and doesn't know at all how function under test looks like and what it contains.
In your example you're using some functions in the Plug callfunction and I'm pretty sure that this plug makes more than it should - remember about single responsible principle. This makes this one function almost impossible to test without mocking... I would rewrite this plug into 3 or 4 four separated plugs, because with clause is redundant - plugs check the outcom of previous plug to proceed - it's case inside case, just what with does.
Considering you have new plugs you can use some extra functions inside the plug except call and init that do the real work defined as private functions and this action would propably help you organize your code and avoid creating chained modules in terms of usage and responsibility.
Then, unit tests would be much easier, because you would test isolated plugs.
Assuming that you have this plug called like this:
plug MyPlug
you would rewrite into:
plug :validate_is_admin
plug :coerce_form_data
plug :create_from_form_data
Maybe it's simplified, but I hope you get what I meant here.
TL; DR: Split functions into smaller ones and test them in isolation. Hide internal computations in private functions and test only public API.
Related
I am wondering if I should unit test functions that have no control flow. This functions take some input, call a sequence of 5/6 other functions, then return some output.
Testing them seems a waste of time, since I don't see what I would be testing exactly. The other functions called already have unit test for them.
The main drawback for me is that I don't know what the output should be a priori, I would need to call the same functions in the test scripit to see if the results coincide; and then what am I testing? That the test function and the actual function have the same lines in the same order?
Thanks for any insight
Note: Same as last question, if you think it's primarily opinion based reformulate as "According to the principles advocated in Art of Unit Testing, should i unit test functions with no control flow?"
Short answer: yes, of course you do!
Long answer: how a method does something is in the end "implementation" detail. In that sense: you should not care at all if a method is using a switch, some if/elses, a loop, or just calls other methods in sequence.
Instead, you should understand the contract that your method provides: which input it takes; and what comes out of it (depending on the inputs maybe).
That is what you focus on: creating a setup where your method can run; to then check if the method upholds that contract.
Example:
public void foo(Bar bar) {
FooBar fooBar = bar.wobbel();
fooBar.throttle();
fooBar.rattle(this.someField);
}
that code above doesn't contain any control flow statements. But still, there are various points in there where things could go wrong (for example NullPointerExceptions). Don't you think it would be better to catch those using unit tests?
I'm looking to better understand I should test functions that have many substeps or subfunctions.
Let's say I have the functions
// Modify the state of class somehow
public void DoSomething(){
DoSomethingA();
DoSomethingB();
DoSomethingC();
}
Every function here is public. Each subfunction has 2 paths. So to test every path for DoSomething() I'd have 2*2*2 = 8 tests. By writing 8 tests for DoSomething() I will have indirectly tested the subfunctions too.
So should I be testing like this, or instead write unit tests for each of the subfunctions and then only write 1 test case that measures the final state of the class after DoSomething() and ignore all the possible paths? A total of 2+2+2+1 = 7 tests. But is it bad then that the DoSomething() test case will depend on the other unit test cases to have complete coverage?
There appears to be a very prevalent religious belief that testing should be unit testing. While I do not intend to underestimate the usefulness of unit testing, I would like to point out that it is just one possible flavor of testing, and its extensive (or even exclusive) use is indicative of people (or environments) that are somewhat insecure about what they are doing.
In my experience knowledge of the inner workings of a system is useful as a hint for testing, but not as an instrument for testing. Therefore, black box testing is far more useful in most cases, though that's admittedly in part because I do not happen to be insecure about what I am doing. (And that is in turn because I use assertions extensively, so essentially all of my code is constantly testing itself.)
Without knowing the specifics of your case, I would say that in general, the fact that DoSomething() works by invoking DoSomethingA() and then DoSomethingB() and then DoSomethingC() is an implementation detail that your black-box test should best be unaware of. So, I would definitely not test that DoSomething() invokes DoSomethingA(), DoSomethingB(), and DoSomethingC(), I would only test to make sure that it returns the right results, and using the knowledge that it does in fact invoke those three functions as a hint I would implement precisely those 7 tests that you were planning to use.
On the other hand, it should be noted that if DoSomethingA() and DoSomethingB() and DoSomethingC() are also public functions, then you should also test them individually, too.
Definitely test every subfunction seperately (because they're public).
It would help you find the problem if one pops up.
If DoSomething only uses other functions, I wouldn't bother writing additional tests for it. If it has some other logic, I would test it, but assume all functions inside work properly (if they're in a different class, mock them).
The point is finding what the function does that is not covered in other tests and testing that.
Indirect testing should be avoided. You should write unit tests for each function explicitly. After that You should mock submethods and test your main function. For example :
You have a method which inserts a user to DB and method is like this :
void InsertUser(User user){
var exists = SomeExternal.UserExists(user);
if(exists)
throw new Exception("bla bla bla");
//Insert codes here
}
If you want to test InsertUser function, you should mock external/sub/nested methods and test behaviour of InsertUser function.
This example creates two tests: 1 - "When user exists then Should throw Exception" 2 - "When user does not exist then Should insert user"
If a function just calls another function or performs actions. How do I test it? Currently, I enforce all the functions should return a value so that I could assert the function return values. However, I think this approach mass up the API because in the production code. I don't need those functions to return value. Any good solutions?
I think mock object might be a possible solution. I want to know when should I use assert and when should I use mock objects? Is there any general guide line?
Thank you
Let's use BufferedStream.Flush() as an example method that doesn't return anything; how would we test this method if we had written it ourselves?
There is always some observable effect, otherwise the method would not exist. So the answer can be to test for the effect:
[Test]
public void FlushWritesToUnderlyingStream()
{
var memory = new byte[10];
var memoryStream = new MemoryStream(memory);
var buffered = new BufferedStream(memoryStream);
buffered.Write(0xFF);
Assert.AreEqual(0x00, memory[0]); // not yet flushed, memory unchanged
buffered.Flush();
Assert.AreEqual(0xFF, memory[0]); // now it has changed
}
The trick is to structure your code so that these effects aren't too hard to observe in a test:
explicitly pass collaborator objects,
just like how the memoryStream is passed
to the BufferedStream in the constructor.
This is called dependency
injection.
program against an interface, just
like how BufferedStream is programmed
against the Stream interface. This enables
you to pass simpler, test-friendly implementations (like MemoryStream in this case) or use a mocking framework (like MoQ or RhinoMocks), which is all great for unit testing.
Sorry for not answering straight but ... are you sure you have the exact balance in your testing?
I wonder if you are not testing too much ?
Do you really need to test a function that merely delegates to another?
Returns only for the tests
I agree with you when you write you don't want to add return values that are useful only for the tests, not for production. This clutters your API, making it less clear, which is a huge cost in the end.
Also, your return value could seem correct to the test, but nothing says that the implementation is returning the return value that corresponds to the implementation, so the test is probably not proving anything anyway...
Costs
Note that testing has an initial cost, the cost of writing the test.
If the implementation is very easy, the risk of failure is ridiculously low, but the time spend testing still accumulates (over hundred or thousands cases, it ends up being pretty serious).
But more than that, each time you refactor your production code, you will probably have to refactor your tests also. So the maintenance cost of your tests will be high.
Testing the implementation
Testing what a method does (what other methods it calls, etc) is critized, just like testing a private method... There are several points made:
this is fragile and costly : any code refactoring will break the tests, so this increases the maintenance cost
Testing a private method does not bring much safety to your production code, because your production code is not making that call. It's like verifying something you won't actually need.
When a code delegates effectively to another, the implementation is so simple that the risk of mistakes is very low, and the code almost never changes, so what works once (when you write it) will never break...
Yes, mock is generally the way to go, if you want to test that a certain function is called and that certain parameters are passed in.
Here's how to do it in Typemock (C#):
Isolate.Verify.WasCalledWithAnyArguments(()=> myInstance.WeatherService("","", null,0));
Isolate.Verify.WasCalledWithExactArguments(()=> myInstance. StockQuote("","", null,0));
In general, you should use Assert as much as possible, until when you can't have it ( For example, when you have to test whether you call an external Web service API properly, in this case you can't/ don't want to communicate with the web service directly). In this case you use mock to verify that a certain web service method is correctly called with correct parameters.
"I want to know when should I use assert and when should I use mock objects? Is there any general guide line?"
There's an absolute, fixed and important rule.
Your tests must contain assert. The presence of assert is what you use to see if the test passed or failed. A test is a method that calls the "component under test" (a function, an object, whatever) in a specific fixture, and makes specific assertions about the component's behavior.
A test asserts something about the component being tested. Every test must have an assert, or it isn't a test. If it doesn't have assert, it's not clear what you're doing.
A mock is a replacement for a component to simplify the test configuration. It is a "mock" or "imitation" or "false" component that replaces a real component. You use mocks to replace something and simplify your testing.
Let's say you're going to test function a. And function a calls function b.
The tests for function a must have an assert (or it's not a test).
The tests for a may need a mock for function b. To isolate the two functions, you test a with a mock for function b.
The tests for function b must have an assert (or it's not a test).
The tests for b may not need anything mocked. Or, perhaps b makes an OS API call. This may need to be mocked. Or perhaps b writes to a file. This may need to be mocked.
For example this article introduces them.
What is the benefit?
Static analysis seems cool but at the same time it would prevent the ability to pass null as a parameter in unit test. (if you followed the example in the article that is)
While on the topic of unit testing - given how things are now surely there is no point for code contracts if you already practice automated testing?
Update
Having played with Code Contracts I'm a little disappointed. For example, based on the code in the accepted answer:
public double CalculateTotal(Order order)
{
Contract.Requires(order != null);
Contract.Ensures(Contract.Result<double>() >= 0);
return 2.0;
}
For unit testing, you still have to write tests to ensure that null cannot be passed, and the result is greater than or equal to zero if the contracts are business logic. In other words, if I was to remove the first contract, no tests would break, unless I had specifically had a test for this feature. This is based on not using the static analysis built into the better (ultimate etc...) editions of Visual Studio however.
Essentially they all boil down to an alternate way of writing traditional if statements. My experience actually using TDD, with Code Contracts shows why, and how I went about it.
I don't think unit testing and contracts interfere with each other that much, and if anything contracts should help unit testing since it removes the need to add tedious repetitive tests for invalid arguments. Contracts specify the minimum you can expect from the function, whereas unit tests attempt to validate the actual behaviour for a particular set of inputs. Consider this contrived example:
public class Order
{
public IEnumerable Items { get; }
}
public class OrderCalculator
{
public double CalculateTotal(Order order)
{
Contract.Requires(order != null);
Contract.Ensures(Contract.Result<double>() >= 0);
return 2.0;
}
}
Clearly the code satisfies the contract, but you'd still need unit testing to validate it actually behaves as you'd expect.
What is the benefit?
Let's say that you want to make sure that a method never returns null. Now with unit tests, you have to write a bunch of test cases where you call the method with varying inputs and verify that the output is not null. Trouble is, you can't test all possible inputs.
With code contracts, you just declare that the method never returns null. The static analyzer will then complain if it is not possible to prove that. If it doesn't complain, you know that your assertion is correct for all possible inputs.
Less work, perfect correctness guarantees. What's not to like?
Contracts allow you say what the actual purpose of the code is, as opposed to letting whatever the code does with whatever random arguments are handed it standing as the definition from the point of view of the compiler, or the next reader of the code. This allows significantly better static analysis and code optimization.
For instance, if I declare an integer parameter (using the contract notation) to be in the range of 1 to 10, and I have a local array in my function declared the same size, that is indexed by the parameter, the compiler can tell that there is no possibility of subscript error, thus producing better code.
You can state that null is valid value in a contract.
The purpose of unit testing is to verify dynamically that the code achieves whatever stated purpose it has. Just because you've written a contract for a function, doesn't mean the code does that, or that static analysis can verify the code does that. Unit testing won't go away.
Well it will not interfere with unit-testing in general. But as I saw you mentioned something about TDD.
If I think about it from that perspective I guess it could/may change the procedure from the standard one
create method (just signature)
create Unit test -> implement the test
run the test: let it fail
implement the method, hack it to the end just to make it working
run the test: see it pass
refactor your (possibly messy) method body
(re-run the test just to see you've not broken anything)
This would be the really hard-full-featured unit-testing procedure. In such a context I guess you could insert code contracts between the 1st and 2nd point like
create method (just signature)
insert code contracts for the methods input parameters
create Unit test -> implement the test
...
The advantage I see at the moment is that you can write easier unit tests in the sense that you wouldn't have to check every possible path since some is already taken into account by your defined contracts. It just gives you additional checking, but it wouldn't replace unit testing since there will always be more logic within the code, more path that have to be tested with unit tests as usual.
Edit
Another possibility I didn't consider before would be to add the code contracts in the refactoring part. Basically as additional way of assuring things. But that would somehow be redundant and since people don't like to do redundant stuff...
So, I'm starting to write some logic for a simple program (toy game on the side). You have a specific ship (called a setup) that is a ship + modules. You start with an empty setup based off a ship and then add modules to that setup. Ships also have a numbered array of module positions.
var setup = new Setup(ship); // ship is a stub (IShip) defined someplace else
var module = new Mock<IModule>().Object;
setup.AddModule(module, 1); // 1 = which position
So, this is the code in my test method. I now need to assert on this code. Well, I need a getter method right?
Assert.AreEqual(module, setup.GetModule(1));
This might sound really dumb and I'm worrying about nothing, but for some stupid reason I'm concerned with adding a method just to assert that a test passed.
Is this fine and is in fact part of the design process that TDD is pushing out? For instance I know I need an AddModule method because I want to test it, and the fact that this requires a GetModule method to test is simply an evolution of my design via TDD.
Or is this kind of a smell because I don't even know if I'll really need GetModule in my code and it will only be used in a test?
For example, adding a module is going to ultimately affect different stats of a setup (armor, shield, firepower, etc). The thing is those are going to be complex, and I wanted to start with a simple test. But in the end, those are the public attributes I care about -- a setup is defined by its stats, not by a list of modules.
Interesting question. I'm glad to hear you're writing the tests first.
If you let the design manifest itself through the tests, you're more likely to build only the parts you'll need. But is this the best design? Maybe not, but don't let that discourage you -- your add method works!
It may be too early to tell if you'll need the GetModule method later. For now, build up the functionality you need and go green, then slowly refactor it (going from red to green again) to get the design you want.
Part of evolving the design is to start with baby steps like a simple method and then grow into the complex stats (eventually dropping this method and changing the test) when enough supports it. When doing TDD, don't expect that the first test you write is targeting the ideal interface. It is OK to have some messiness that will get dropped as you evolve the design.
That being said, if you see no public purpose to the method, try to limit its visibility as much as is reasonable to the test code. Although even that should eventually go away as you get to build out the rest of the system and have something real to test as a side effect of the set method.
I would be wary of introducing a public method in my class that is only used for testing.
There are various ways how you could test this:
Reflection: The GetModule method is a private method in your class (this could also work if your 'stats' are private) and you can access it in your test method via reflection. This will work well, the only trouble is you will not get any compiler errors if you change the name of the private method or add / delete some variables (but, of course, your test will fail and you will know early)
Inheritance: The GetModule method could be protected (only inheritance visible) and your test class could inherit from the main class. This way your test class gets access to this method, but this is not really exposed to the outside world.
Assert the side-effect: This is where you really think about what it means to add a module to the system. If it is going to affect some 'stats' as you put it, you could write tests which assert that the stats are appropriately modified.