I already check this however I would like to know what the common practice is for the next situation. I have several roles in a web I'm creating, only one role can create the others, now I wonder if for the others I can test in just 1 test function. Something like:
def test_cannot_create_user(self):
for role in ['role_1','role_2','role_3']: #roles that can't create
for user in ['role_1','role_2','role_3','role_4']: # all roles
# assign user to a role
# user with role fails on creating user
# do assertions
# remove user from group
Or should I create a function for each role and for each role to fail to be created?
Design is what we do to get more of what we want than we would get by just doing it.
When we are refactoring (as in red green refactor), the structure of our tests doesn't matter very much, because any test failure is always a consequence of the change that we just made - we don't need fine grain reporting, because we already know where the problem is.
But when we are doing a merge with someone else's code, having a clear report of a failure means that we can more clearly guess where we need to look for the problem that has been introduced.
When we are trying to inspect our test reports, then one of the things we care about in an audit is "what did we test" - having a clear enumeration of the cases in that report is better aligned with that goal than is expecting people to dig into the test details.
Horses for courses
What you can do in some frameworks is create a "parameterized" test, where you have a single test function, that accepts input arguments, with those arguments also being copied into the test report.
In your case, that might look something like
def test_cannot_create_user(self, role):
# your test code here
A good video to review would be Kevlin Henney's 2016 talk, where he discusses some test design considerations for a simple Gregorian leap year calculation; how should you design your tests if you want them to accurately describe your requirements.
Related
I have a concern with "honesty" of test when doing TDD. TDD is
Write red test
Write just enough code to make it green
Refactor and let the test green
So far so good. Now here is an example of applying the principle above, such kind of example were already met in tutorial & real life :
I want to check that the current user email is displayed on the default page of my webapp.
Write a red test : "example#user.com" is displayed inside default_page.html
Write just enough code to make it green : hardcode "example#user.com" inside default_page.html
Refactor by implementing get_current_user(), some other code in some others layers etc, letting the test green.
I'm "shocked" by step 2. There is something wrong here : the test is green even if nothing is actually working. There a test smell here, it means that maybe at some point someone could break the production code without breaking the test suite.
What I am missing here ?
Your assertion that "nothing is working" is false. The code functions correctly for the case that the email address is example#user.com. And you do not need that final refactoring. Your next failing test might be to make it fail for the case that the user has a different email address.
I would say that what you have is only partially complete. You said:
I want to check that the current user email is displayed on the default page of my webapp.
The test doesn't check the current users email address on the default page, it checks that the fixed email address "example#user.com" is in the page.
To address this you either need to provide more examples (ie have multiple tests with different email addresses) or to randomly generate the email address in the test setup.
So I would say what you have is something like this is pseudo code:
Given current user has email address "example#user.com"
When they visit the default page
The page should contain the email address "example#user.com"
This is the first test you can write in TDD and you can indeed hardcode this to avoid implementing unnecessary stuff. You can now add another test which will force you to implement the correct behavior
Given current user has email address "example2#user.com"
When they visit the default page
The page should contain the email address "example2#user.com"
Now you have to remove the hardcoding as you cannot satisfy both of these tests with a hardcoded solution.So this will force you to get the actual email address from the current user and display this.
Often it makes sense to end up with 3 examples in your tests. These don't need to be 3 separate tests, you can use data driven tests to reuse the same test method with different values. You don't say what test framework you are using, so I can't give a specific example.
This approach is common in TDD and is called triangualtion.
You are correct about
step 2. There is something wrong here
but it's not in the TDD approach. IMHO it's in the test logic. After all this (step 2) validates that the test harness is working correctly. That the new test does not mistakenly pass without requiring any new code, and that the required feature does not already exist.
What I am missing here ?
This step also should tests the test itself, in the negative: it rules out the possibility that the new test always passes, and therefore is worthless. The new test should also fail for the expected reason. It's vital that this step increases the developer's confidence that it is testing the right thing, and passes only in intended cases.
I want to validate an opinion with you.
I have to design a web service that searches into a database of restaurants affiliated to a discount program in a specific country around a given address.
The REST call to such a webservice will look like http://server/search?country=<countryCode>&language=<languageCode>&address=<address>&zipcode=<zipcode>
The problem is that some countries do not have zipcodes or do not have them in the entire country.
Now, what would you do if the user passes such a parameter for a country that does not have zipcodes, but he/she passes a valid address?
Return 400 Bad request.
Simply igonre the zipcode parameter and return results based on the valid address
Return an error message in a specific format (e.g. JSON) stating that zipcodes are not supported for that country
Some colleagues are also favoring the following option
4. Simply return no results. And state in the documentation that the zipcode parameter is not supported. Also we have to create a webservice method which returns what fields should be displayed in the user interface.
What option do you think is best and why?
Thanks!
Well the OpenStreetMap Nomination Server returns results even if you dont know the ZIP Code and you can look at the results anyway. What if the user doesnt know the zip code but wants to find hist object?
I would try to search for that specific object anyway, especially because you said that some countries have zip codes partially.
If you simply return nothing te user doesnt know what went wrong and he wont know what to do.
That would depend on the use case. How easy is it for a user of the API to trigger that case? Is it a severe error which the user really should know how to avoid? Or is it something that is not entirely clear, where a user may know (or think he knows) a zipcode where officially there shouldn't be one? Does it come down to trial and error for the user how to retrieve correct results from your API? Is it a bad enough error that the user needs to be informed about it and that he needs to handle this on his side?
If you place this restriction in your API, consider that it will have to be clearly documented when this case is triggered, every user of the API will have to read and understand that documentation, it needs to be clear how to avoid the problem, it needs to be possible for the user to avoid the problem and every user will have to correctly implement extra code on his side to avoid this problem. Is it possible for the user to easily know which areas have zipcodes and which don't?
I think the mantra of "be flexible in what you accept, strict in what you output" applies...
I've started writing some tests for my Django app and I'm unsure how best to structure the code.
Say I have a register page and a page for logged in users only.
My first plan was to have an earlier method perform the register and a later method use that login to test the page:
def test_register_page(self):
//send request to register page and check user has been registered correctly
def test_restricted_page(self):
c = Client();
c.login("someUser","pass");
c.post("/someRestrictedPage/");
//Test response
However this means that now one of my tests rely on the other.
The alternatives I see are calling register in setUp() but this still means that the restricted page test relies on the register page working.
I could try creating a new user manually in setup which I also don't like because this isn't testing a user created by the system.
What is the usual pattern for testing this kind of situation?
You are trying to mix together a lot of different functionalities in one test case. A clean design would be having one test case
for user registration and
one for the view.
Having them depend on each other will introduce a lot of dependencies between them - and - if the test fails the error will be even harder to debug. The success of the registration test should be determined through the correct creation of the user instance (so check necessary attributes etc of the user) and not through being able to login on a certain page. Therefore you will need to set up a "correct" user instance for the view test case. This may seem a bit more complicated than necessary, but it will make future maintainance a lot easier.
What you are trying to do is more something like an integration test, which tests a whole system, but before that you should split up your system in functional units and do unit tests on this units!
The smaller and well-defined the single tests are, the easier will be their maintainance and debugging.
I am using mTurk for surveys, and I need a way of making sure that people who have participated in a previous survey / HIT do not participate in certain future surveys / HITs. I am not sure whether I should do this as a qualification or in some other way.
I know there is some way to do this, but I have no idea how. I have very limited programming experience and would greatly, greatly appreciate specific instructions on how I might do this. My understanding is that I might need to use AWS? Many thanks!
Mass rejections as suggested above are a really, really bad idea in terms of your reputation as a requester. You are much better off creating a Qualification for the new HIT, which automatically grants a score of 100 (or whatever) to anyone who takes it, and assigning scores of zero to everyone who has done the previous surveys. This prevents repeats but doesn't annoy any of your workers.
The easiest way to create a Qualification is at https://requester.mturk.com/qualification_types.
If you download the csv of workers from here https://requester.mturk.com/workers, you can assign scores to workers who have done the previous HIT(s).
To make the qualification grant scores to new workers automatically requires the API, though.
Here's a hacky way to do it:
When you accept HITs for surveys, save every participating worker's ID.
In the writeup, note that "if you've done previous surveys w/ us, then you can't do this one (IE, you can, but we won't approve it)".
When you approve HITs, cross-reference the worker ids with anybody who participated in a previous survey, and reject the hits of any that match.
If you're doing enough surveys, then yes, you probably want to use AWS API for at least the approval part. otherwise, most things appear to be do-able from the requester interface.
Amazon Mechanical Turk service has this option for requesters to grant their workers by Qualification_Type. In this way by connecting your HITs to a qualification_type naming "A", then granting workers exactly the same qualification_type, only workers who have that qualification can see and work on HITs.
First, creating desired qualification types through mturk web UI.(it is only name and description) requester.mturk.com > manage > QualificationTypes. It will give you a qualification id after generating it. (you will need it soon)
Second, in HIT creation loop, you have to use QualificationRequirement class. (I am using java code and it looks like the below-mentioned code):
QualificationRequirement[] qualReq = new QualificationRequirement[1];
qualReq[0] = new QualificationRequirement();
qualReq[0].setQualificationTypeId(qualID);
qualReq[0].setComparator(Comparator.EqualTo);
qualReq[0].setIntegerValue(100);
qualReq[0].setRequiredToPreview(false);
then in HIT creation loop, I will use this:
try {
hit = this.service.createHIT(null,
props.getTitle(),
props.getDescription(),
props.getKeywords(),
question.getQuestion(),
new Double(props.getRewardAmount()),
new Long(props.getAssignmentDuration()),
new Long(props.getAutoApprovalDelay()),
new Long(props.getLifetime()),
new Integer(props.getMaxAssignments()),
props.getAnnotation(),
qualReq,
null);
Third is assigning the qualification type to the workers that you want them to work on your HITs. It is very straightforward, I usually use mturk UI to do it. https://requester.mturk.com/ > manage tab > Workers. You should download the CSV file if you want to assign this qualification to a bunch of workers.
(Workers are who worked with you in the past)
you could notify workers by sending them an email after qualifying them
Notice: Some workers are very slow in answering your new HITs after qualifying them; so keep in mind that you should have some backup plan and time if you will not receive enough response in a certain amount of time.
I am writing unit tests for a PHP class that maintains users in a database. I now want to test if creating a user works, but also if deleting a user works. I see multiple possibilities to do that:
I only write one method that creates a user and deletes it afterwards
I write two methods. The first one creates the user, saves it's ID. The second one deletes that user with the saved ID.
I write two methods. The first one only creates a user. The second method creates a user so that there is one that can afterwards be deleted.
I have read that every test method should be independent of the others, which means the third possibility is the way to go, but that also means every method has to set up its test data by itself (e.g. if you want to test if it's possible to add a user twice).
How would you do it? What is good unit testing style in this case?
Two different things = Two tests.
Test_DeleteUser() could be in a different test fixture as well because it has a different Setup() code of ensuring that a User already exists.
[SetUp]
public void SetUp()
{
CreateUser("Me");
Assert.IsTrue( User.Exists("Me"), "Setup failed!" );
}
[Test]
public void Test_DeleteUser()
{
DeleteUser("Me");
Assert.IsFalse( User.Exists("Me") );
}
This means that if Test_CreateUser() passes and Test_DeleteUser() doesn't - you know that there is a bug in the section of the code that is responsible for deleting users.
Update: Was just giving some thought to Charlie's comments on the dependency issue - by which i mean if Creation is broken, both tests fail even though Delete. The best I could do was to move a guard check so that Setup shows up in the Errors and Failures tab; to distinguish setup failures (In general cases, setup failures should be easy to spot by an entire test-fixture showing Red.)
How you do this codependent on how you utilize Mocks and stubs. I would go for the more granular approach so having 2 different tests.
Test A
CreateUser("testuser");
assertTrue(CheckUserInDatabase("testuser"))
Test B
LoadUserIntoDB("testuser2")
DeleteUser("testuser2")
assertFalse(CheckUserInDatabase("testuser2"))
TearDown
RemoveFromDB("testuser")
RemoveFromDB("testuser2")
CheckUserInDatabase(string user)
...//Access DAL and check item in DB
If you utilize mocks and stubs you don't need to access the DAL until you do your integration testing so won't need as much work done on the asserting and setting up the data
Usually, you should have two methods but reality still wins over text on paper in the following case:
You need a lot of expensive setup code to create the object to test. This is a code smell and should be fixed but sometimes, you really have no choice (think of some code that aggregates data from several places: You really need all those places). In this case, I write mega tests (where a test case can have thousands of lines of code spread over many methods). It creates the database, all tables, fills them with defined data, runs the code step by step, verifies each step.
This should be a rare case. If you need one, you must actively ignore the rule "Tests should be fast". This scenario is so complex that you want to check as many things as possible. I had a case where I would dump the contents of 7 database tables to files and compare them for each of the 15 SQL updates (which gave me 105 files to compare in a single test) plus about a million asserts that would run.
The goal here is to make the test fail in such a way that you notice the source of the problem right away. It's like pouring all the constraints into code and make them fail early so you know which line of app code to check. The main drawback is that these test cases are hell to maintain. Every change of the app code means that you'll have to update many of the 105 "expected data" files.