I’d like to know if there is a way to really link the test of a build, with a test case WorkItem.
I've seen that you can attach the result of a test to a test case, but I want the test (method) to be linked to the test case, not the result.
I’m using Report Builder, where I show the list of test result of a build. The thing is, the name of each tests comes from the methods name. I’d prefer to use the name of the test case.
You cannot link a part of code (in this case a test method) to an work item (in this case a test case).
Related
Best way of handling many test cases needing to navigate to a particular place before they run their asserts? For example a process has 5 steps and a test case needs to test a part of step 5, how can I set it up? Call the test case methods of the previous steps inside this test case, and do that for all test cases that test step 5?
Similarly, if a test case goes deep into the website - through many pages - should that navigation be re-written for every test case, or just call some test that already does that?
Any tips on this situations?
Best way of handling many test cases needing to navigate to a particular place before they run their asserts? For example a process has 5 steps and a test case needs to test a part
of step 5, how can I set it up?
I would create a Transporter class / pattern that the test case can call to get to that state. That will make the navigation code reusable to other tests and not make the test too big/complicated. You can also use the setUp() methods in your xUnit testing frameworks which will be called before each test and place the navigator code there, if you need to use it for other tests.
Similarly, if a test case goes deep into the website - through many
pages - should that navigation be re-written for every test case, or
just call some test that already does that?
I would extract that code into a helper class called Transporter and have the tests call it to easily navigate to the deep page in one method call.
I wrote about this and other test design patterns in a conference paper at the Pacific Northwest Software Quality Conference. Look for the Transporter pattern in that paper.
Here's an example using a Transporter class where you have to login and navigate to the checkout page:
public class Transporter {
public static void login() {
//App specific code to navigate to login screen and login to the application
}
public static void gotoCheckout() {
//App specific code to navigate to the checkout page
}
}
Now your tests can just call this Transporter class to do the navigation for them.
If you are using BDD, such as JBehave (not sure if Cucumber has the same feature) where you have the Given, When, Then story (feature) structure in Gherkin syntax, you can actually use "GivenStories" feature which are like prequel test cases to set you up for your specific test case, exactly as you are describing.
There's nothing wrong, however, when using BDD to simply make multiple step scenarios leading up to the particular test case, i.e. first scenario logs-in, second scenario navigates to certain page, third scenario performs your actual test.
By writing it as a separate story (feature), however, you can re-use those as "GivenStories" in JBehave as a shortcut to get where you need to be without duplicating the steps.
I have a concern with "honesty" of test when doing TDD. TDD is
Write red test
Write just enough code to make it green
Refactor and let the test green
So far so good. Now here is an example of applying the principle above, such kind of example were already met in tutorial & real life :
I want to check that the current user email is displayed on the default page of my webapp.
Write a red test : "example#user.com" is displayed inside default_page.html
Write just enough code to make it green : hardcode "example#user.com" inside default_page.html
Refactor by implementing get_current_user(), some other code in some others layers etc, letting the test green.
I'm "shocked" by step 2. There is something wrong here : the test is green even if nothing is actually working. There a test smell here, it means that maybe at some point someone could break the production code without breaking the test suite.
What I am missing here ?
Your assertion that "nothing is working" is false. The code functions correctly for the case that the email address is example#user.com. And you do not need that final refactoring. Your next failing test might be to make it fail for the case that the user has a different email address.
I would say that what you have is only partially complete. You said:
I want to check that the current user email is displayed on the default page of my webapp.
The test doesn't check the current users email address on the default page, it checks that the fixed email address "example#user.com" is in the page.
To address this you either need to provide more examples (ie have multiple tests with different email addresses) or to randomly generate the email address in the test setup.
So I would say what you have is something like this is pseudo code:
Given current user has email address "example#user.com"
When they visit the default page
The page should contain the email address "example#user.com"
This is the first test you can write in TDD and you can indeed hardcode this to avoid implementing unnecessary stuff. You can now add another test which will force you to implement the correct behavior
Given current user has email address "example2#user.com"
When they visit the default page
The page should contain the email address "example2#user.com"
Now you have to remove the hardcoding as you cannot satisfy both of these tests with a hardcoded solution.So this will force you to get the actual email address from the current user and display this.
Often it makes sense to end up with 3 examples in your tests. These don't need to be 3 separate tests, you can use data driven tests to reuse the same test method with different values. You don't say what test framework you are using, so I can't give a specific example.
This approach is common in TDD and is called triangualtion.
You are correct about
step 2. There is something wrong here
but it's not in the TDD approach. IMHO it's in the test logic. After all this (step 2) validates that the test harness is working correctly. That the new test does not mistakenly pass without requiring any new code, and that the required feature does not already exist.
What I am missing here ?
This step also should tests the test itself, in the negative: it rules out the possibility that the new test always passes, and therefore is worthless. The new test should also fail for the expected reason. It's vital that this step increases the developer's confidence that it is testing the right thing, and passes only in intended cases.
For example, a test case named ExampleTest
ExampleTest {
testA{};
testB{};
testC();
}
I could run all this class with TestSuite.addTestSuite(ExampleTest.class);
but, how to select testA and testB run into TestSuite?
2 Ideas:
See if it makes sense to split ExampleTest into 2 test classes based on your partition
Otherwise use Categories to tag your tests and then run tests which belong to a specific category. Its simpler in NUnit, JUnit seems to require you to create an empty/ marker interface to get this done. A sample link with code.
Let's say in my user model I have a ChangePassword method. Given an already initialised user model, it takes the new password as a parameter and does the database work to make the magic happen. The front end to this is a web form, where the user enters their current password and their desired new password. The controller then checks to see if the user's current password is correct. If so, it invokes the user model's ChangePassword method. If not, it displays an error to the user.
From what I hear you're supposed to unit test the smallest piece of code possible, but doing that in this case completely ignores the check to make sure the user entered the correct current password. So what should I do?
Should I:
A) Unit test only from the controller, effectively testing the model function too?
OR
B) Create 2 different tests; one for the controller and one for the model?
When in doubt, test both. If you only test the controller and the test fails, you don't know whether the issue is in the controller or the model. If you test both, then you know where the problem lies by looking at the model's test result - if it passes, the controller is at fault, if it fails, then the model is at fault.
A)
The test fails. You have a problem in either the model or the controller, or both and spend time searching through the model and controller.
B)
The model and controller tests fail... chances are you have a problem in the model.
Only the controller test fails... chances are better that the problem is not in the model, only in the controller.
Only the model test fails... hard to see this happening, but if it does somehow then you know the problem is in the model, not in the controller.
It's good to test both layers. It'll make finding the problem later that much easier.
There should be multiple tests here:
Verify the correct password was entered.
Validate the new password, e.g. doesn't match existing one, has minimum length, sufficient complexity, tests for errors thrown, etc.
Updating the database to the new password.
Don't forget that the tests can also help act as documentation of the code in a sense so that it becomes clear for what each part of the code is there.
You might want to consider another option: Mock objects. Using these, you can test the controller without the model, which can result in faster test execution and increased test robustness (if the model fails, you know that the controller still works). Now you have two proper unit tests (both testing only a single piece of code each), and you can still add an integration test if required.
Unit testing means to test every unit on its own, so in this case you would need to build two unit tests, one for the frontend and one for the backend.
To test the combination of both an integration test is needed, at least the ITSQB calls it like that.
If you code object oriented you usually build unit tests for every class as that is the smallest independent unit testable.
A) is not a unit test in my opinion since it uses more than one class (or layer). So you should really be unit-testing the model only.
Suppose I have class Car with following methods:
LoadGasoline(IFuel gas)
InsertKey(IKey key)
StartEngine()
IDrivingSession Go()
the purpose of Car is to configure and return an IDrivingSession which the rest of the application uses to drive the car. How do I unit-test my Car?
It looks like it requires a sequence of operations done before I can call Go() method. But I want to test each method separately as they all have some important logic. I don't want to have bunch of unit-tests like
Test1: LoadGasoline, Assert
Test2: LoadGasoline, InsertKey, Assert
Test3: LoadGasoline, InsertKey, StartEngine, Assert
Test4: LoadGasoline, InsertKey, StartEngine, Go, Assert
Isn't there a better way to unit-test sequential logic or is this a problem with my Car design?
--- EDIT ----
Thanks for all the answers. As many noticed, I should also have tests for invalid scenarios and I have those too, but this question is focused on how to test the valid sequence.
I think each method should be tested separately and independently.
IMHO, you should prepare the environment for each case, so only the LoadGasoline test will break if you change the LoadGasoline method, and you won't need to see all the tests break because of a single bug.
I don't know how the state of your Car looks like, but, before the InsertKey, you should prepare with a method like, car.SetTotalGasoline(20); or whatever variable is set in this method, but not depend on a complex logic of the method LoadGasoline.
You will later need a test (in this case, not a unit test) to test all the sequence.
Some unit testing frameworks let you specify set-up code which runs before the actual test starts.
This allows you to get the target object into the proper state before running your test. That way your test can pass or fail based on the specific code you're testing rather than on the code needed before you can run a test.
As a result, your test sequence will wind up something like this:
Test1:
LoadGasoline, Assert
Test2 Setup:
LoadGasoline
Test2:
InsertKey, Assert
Test3 Setup:
LoadGasoline, InsertKey
Test3:
StartEngine, Assert
Test4 Setup:
LoadGasoline, InsertKey, StartEngine
Test4:
Go, Assert
Realistically speaking, since the tests are all run in sequence there's no chance of Test's Setup failing if the previous test passes.
With that said, you should also test failure cases that aren't expected to work but that's a different issue.
Why don't you want all those tests?
Go has very different behavior if you call it before or after, say, InsertKey, right? So you ought to be testing both behaviors, in my opinion.
Its a fair reluctance but sometimes thats what you need to do. If you can't fake out the system under test so it thinks its in a later state, then you need to go through the same steps to get it into that state. Without knowing more about what your testing its not clear how you could fake out the different states.
One way you can make this tolerable is use an extract method refactoring on tests for one state so that same code can be used to prepare the next test.
I would probaly have
Test 1: LoadGasoline, Assert, InsertKey Assert, StartEngine Assert, Go Assert
Test 2: LoadGasoline, Go, Assert
Test 3: Go, Assert
Test 4: StartEngine, Go, Assert
Depending on the actual object, I would probally not try and do all permutations, but I would have a single test that hits the success track, then I would tests that hit my fringe cases.
Edit:
After some thought I might have tests like:
Start a car key that has no gas
Start a car with gas, and wrong key
Start a car with gas and right key (Test 1 above)
Push Peddle before starting car.
Technically you should use the following tests at least:
testLoadGasoline
testInsertKeyGasolineNotLoaded
testStartEngineKeyNotInserted
testGoEngineNotStarted
testGo
If you can directly view intermediate steps you can add
testInsertKeyGasolineLoaded
testStartEngineKeyInserted
Note that if you can directly set the state (which is language and design dependent), then
testInsertKeyGasolineLoaded might not actually call LoadGasoline.