I have a requirement something like this:
As soon as the user signsup(and will be in the waiting state untill he confirms his email address), a session variable is set something like "FIRST_TIME_FREE_SESSION_EXPIRY_AGE_KEY"(sorry if the name sounds confusing!) which will be set to a datetime object adding 8hrs to the current time.
How this should impact the user is, the user gets 8hrs time to actually use all the features of our site, without confirming his signedup email address. After 8hrs, every view/page will show a big banner telling the user to confirm. (All this functionality is achieved using a single "ensure_confirmed_user" decorator for every view).
I want to test the same functionality using django's unittest addon(TestCase class). How do I do it?
Update: Do I need to manually update the mentioned session variable value(modified 8hrs to a few seconds) so as to get it done? Or any better way is there?
Update: This may sound insane, but I want to simulate a request from the future.
Generally, if unit testing is difficult because the product code depends on external resources that won't cooperate, you can abstract away those resources and replace them with dummies that do what you want.
In this case, the external resource is the time. Instead of using datetime.now(), refactor the code to accept an external time function. It can default to datetime.now. Then in your unit tests, you can change the time as the test progresses.
This is better than changing the session timeout to a few seconds, because even then, you have to sleep for a few seconds in the test to get the effect you want. Unit tests should run as fast as you can get them to, so that they will be run more often.
I can think of a couple of possibilities. During your test run, override the FIRST_TIME_FREE_SESSION_EXPIRY_AGE_KEY variable and set it to a smaller time limit. You can then wait until that time limit is over and verify if the feature is working as expected.
Alternately replace your own datetime functions (assuming your feature relies on datetime)
You can accomplish these by overriding setup_ and teardown_test_environment methods.
My settings.py differs slightly depending on whether django is run in a production environment or in a development environment. I have 2 settings modules: settings.py and settings_dev.py. The development version looks as follows:
from settings import *
DEBUG = True
INSTALLED_APPS = tuple(list(INSTALLED_APPS) + [
'dev_app',
])
Now you can solve your problem in different ways:
add the variable with different values to both settings modules;
Where you set the variable, choose between two values according to the value of the DEBUG setting. You can also use DEBUG to omit the unit test when on the production server, because the test will probably take too long there anyway.
You can use the active settings module like this:
from django.conf.project_template import settings
if settings.DEBUG:
...
Related
I am working on a django project that, along with having a database for its models and relations, writes to a log directory called activity_logs outside of the project directory to keep track of formatted user activities, one file for each user. This is an alternative file-structure-based solution to having a database table carry this information along, because this offloads some storage from the DB and is relatively easy to format and express such activities. Perhaps some of you may recommend storing this kind of data in the database, which is fine, but I still believe there is question from all of this that I need help answering.
This django project has multiple apps that have an extensive test suite, one for each app. Additionally, there is a logging.py file that encapsulates the logging functionality (writing/reading activities to/from log files), and so both the test cases within the test suites as well as the view functions (and various other utility functions) all utilize these logging functions in order to store these user activities and retrieve them based on model relationships to emulate a user notification system. Since the logging module takes care of this logging, it needs to know where to write to, and so we have a directory structure called activity_logs to which it writes user log files, creating one for a new user and deleting one for a user removed from the database. One of the newest changes we would like to make in this project is to create a separate logging directory for testing this logging functionality, something like test_activity_logs, so that it would never be confused when writing to the test directory for test users or the regular activity log directory for real users.
My problem is this: at runtime, how can I tell the system, at whichever startpoint of execution (whether it be from a view function call through the django test Client object, a test case, an actual HTTPrequest made via a URL, etc.), when to look inside the activity_logs or the test_activity_logs directory? It solely depends on whether I am generating new information for a real user or a test user, but a User is a User in our system, and I'm facing some trouble trying to tell these functions that call some logging functions to write to the test log directory vs. the regular one. For example, one approach I am trying is to pass a keyword argument (kwarg) to the logging functions so that they can be made aware of which directory to read/write to/from, like so:
self.assertTrue(activity_has_been_logged(ACTIVITY_ACCOUNT_CREATED, user.get_profile(), use_test_activity_log_directory=True) == True)
the kwarg called use_test_activity_log_directory=True will tell the logging function called activity_has_been_logged to read the test activity log directory. Unfortunately, apart from being a little inflexible (but tolerable), this doesn't solve the situation where the django test client object sends a GET or POST request via a URL to a view function that writes activities to log files:
response = client.post(propose_match_url, post) #Can't write to test_log_directory if by default it writes to regular directory!
How do I let the client pass on this kwarg to those view functions? I think that it should totally be possible to do this, but I'm not sure if fiddling with these kwargs is the best way, or maybe create a global variable in the project settings file, but maybe that might cause some trouble with race conditions with a shared mutable variable.
Your help would be great. Thanks in advance!
So I just solved this problem. The logging file hosting all logging functionality is really the only place that needs to know where to look (either test_activity_logs or activity_logs), since all other components will invoke functions from the logging module to write/read to/from these directories. I gave an additional field to the model class of the UserProfile class called is_test that is a boolean field to determine whether to look in the test_activity_logs if is_test=True, or activity_logs if is_test=False. That way, the logging module needs only to check the input parameter of type UserProfile and its new field to determine where to perform its logging functionalities. Problem solved!
Check out daemontools if you're on a *nix box or launchd on OS X. Both can make sure your Django instance stays running in whatever mode you prefer (daemontools has a few more options for that) and can isolate a directory for logging stdout/stderr.
You can set environment variables for each instance to help other log files and temporary files know where to be created, which you then get from os.environment or simply use the current working directory as a base if using daemontools.
The directory is automatically created for you using daemontools.
I've started writing some tests for my Django app and I'm unsure how best to structure the code.
Say I have a register page and a page for logged in users only.
My first plan was to have an earlier method perform the register and a later method use that login to test the page:
def test_register_page(self):
//send request to register page and check user has been registered correctly
def test_restricted_page(self):
c = Client();
c.login("someUser","pass");
c.post("/someRestrictedPage/");
//Test response
However this means that now one of my tests rely on the other.
The alternatives I see are calling register in setUp() but this still means that the restricted page test relies on the register page working.
I could try creating a new user manually in setup which I also don't like because this isn't testing a user created by the system.
What is the usual pattern for testing this kind of situation?
You are trying to mix together a lot of different functionalities in one test case. A clean design would be having one test case
for user registration and
one for the view.
Having them depend on each other will introduce a lot of dependencies between them - and - if the test fails the error will be even harder to debug. The success of the registration test should be determined through the correct creation of the user instance (so check necessary attributes etc of the user) and not through being able to login on a certain page. Therefore you will need to set up a "correct" user instance for the view test case. This may seem a bit more complicated than necessary, but it will make future maintainance a lot easier.
What you are trying to do is more something like an integration test, which tests a whole system, but before that you should split up your system in functional units and do unit tests on this units!
The smaller and well-defined the single tests are, the easier will be their maintainance and debugging.
I have a project that uses a SOLR search engine through django-haystack. The search engine is on the different live server and touching it during the test run is undesirable (actually, it's impossible, since the access to that host is firewalled)
I'm using standard django testrunner. Luckily, it gives me the object test-settings I can modify to my liking, but turns out it's not the end of the story.
A lot of stuff in django-haystack is instantiated at the import-time, so by the time I change test-settings in my test runner it is too late, and despite the fact that I change the SEARCH_BACKEND to dummy, tests still make call to SOLR. The problem is not specific to HAYSTACK - same issue happens with mongoengine. Any class-level statements (eg CharField(default=Blah.objects.find(...))) are executed at the instantiation-time before django has a chance to change settings.
Of course the root of the problem is the fact that Django settings is a scary globally mutable mess and that Django provides no centralized place for the instantiation code. Given that, are there any suggestions on what testing solution will be the easiest? At the moment I'm thinking about a shell script which will change DJANGO_SETTINGS environment variable to test_settings and run ./manage.py test afterwards. It would be nicer if I could still do things via ./manage.py though.
Any better ideas? People with similar problems?
I took the answer from here and modified it slightly. This works great for me:
from contextlib import contextmanager
#contextmanager
def connection(**kwargs):
from haystack import connections
for key, new_value in kwargs.items():
setattr(connections, key, new_value)
connections['default'].options['URL'] = connections.connections_info['default']['URL']
yield
My test, then, looks like:
def test_job_detail_by_title_slug_job_id(self):
with connection(connections_info=solr_settings.HAYSTACK_CONNECTIONS):
resp = self.client.get('/0/rts-crb-unix-production-engineer/27216666/job/')
self.assertEqual(resp.status_code, 404)
resp = self.client.get('/indianapolis/indiana/usa/jobs/')
self.assertEqual(resp.status_code, 200)
In our company we make news portals for a pretty big number of local newspapers (currently 13, going to 30 next month and more in the future), each with 2k to 100k page views/day. Since we are evolving from a situation where each site was heavily customized to one where each difference is a matter of configuration or custom template, our software is already pretty much the same for all sites. Right now our deployment strategy is one gunicorn instance for each site (with 1-17 workers each, depending on the site traffic), on a 16-core server and 12GB RAM. The problem with this setup is that each worker (regular pre-forked gunicorn) takes 110MB, whether its being used or not. Now with the new sites we would need to add more RAM to serve not that much many requests, so basically it doesn't scale. Also, since we are moving from this model where each site is independent, each site has its own database and I quite like it that way, especially since we are using relational databases (mysql, but migrating to pgsql), so its much easier to shard this way.
I'm doing some research and experimenting with running all sites on one gunicorn instance, so I could use the servers fully and add more servers behind a load balancer when it came to it. The problem is that django assumes in a lot of places that only one site is running per process, so for what I've thought of so far I'd have to implement:
A middleware that takes the HTTP_HOST from the request and places an identifier on a threadlocal variable.
A template loader that uses that variable to load custom templates accordingly.
Monkey patch django.db.model.Model, probably adding a metaclass (not even sure that's possible, but I think I would need it because of the custom managers we sometimes need to use) that would overwrite the managers for one that would first call db_manager(identifier) on the original manager and then call the intended method. I would also need to overwrite the save and delete methods to always include the using=identifier parameter.
I guess I would need to stop using inclusion_tag decorators, not a big problem, but I need to think of other cases like this.
Heavy and ugly patching of urlresolvers if I need custom or extra urls for each site. I don't need them now, but probably will at some point.
And this is just is what I came up with without even implementing it and seeing where it breaks, I'm sure I'd need many more changes for it to work. So I really don't want to do it, especially with the extra maintenance effort I'll need, but I don't see any alternatives and would love to learn that someone already solved this in a better way. Of course I could also stop using django altogether (I already have many reasons to do so) but that would mean a major rewrite and having two maintain two incompatible branches of the software until the new one reached feature parity with the django version, so to me it seems even worse than all the ugly hacks.
I've recently developed an e-commerce system with similar requirements -- many instances running from the same project sharing almost everything. The previous version of the system was a bunch of independent installations (~30) so it was pretty unmaintainable. I'm sure the requirements still differ from yours (for example, all instances shared the same models in my case), but it still might be useful to share my experience.
You are right that Django doesn't help with scenarios like this out of the box, but it's actually surprisingly easy to work it around. Here is a brief description of what I did.
I could see a synergy between what I wanted to achieve and django.contrib.sites. Also because many third-party Django apps out there know how to work with it and use it, for example, to generate absolute URLs to the current site. The major problem with sites is that it wants you to specify the current site id in settings.SITE_ID, which a very naive approach to the multi host problem. What one naturally wants, and what you also mention, is to determine the current site from the Host request header. To fix this problem, I borrowed the hook idea from django-multisite: https://github.com/shestera/django-multisite/blob/master/multisite/threadlocals.py#L19
Next I created an app encapsulating all the functionality related to the multi host aspect of my project. In my case the app was called stores and among other things it featured two important classes: stores.middleware.StoreMiddleware and stores.models.Store.
The model class is a subclass of django.contrib.sites.models.Site. The good thing about subclassing Site is that you can pass a Store to any function where a Site is expected. So you are effectively still just using the old, well documented and tested sites framework. To the Store class I added all the fields needed to configure all the different stores. So it's got fields like urlconf, theme, robots_txt and whatnot.
The middleware class' function was to match the Host header with the corresponding Store instance in the database. Once the matching Store was retrieved, It would patch the SITE_ID in a way similar to https://github.com/shestera/django-multisite/blob/master/multisite/middleware.py. Also, it looked at the store's urlconf and if it was not None, it would set request.urlconf to apply its special URL requirements. After that, the current Store instance was stored in request.store. This has proven to be incredibly useful, because I was able to do things like this in my views:
def homepage(request):
featured = Product.objects.filter(featured=True, store=request.store)
...
request.store became a natural additional dimension of the request object throughout the project for me.
Another thing that was defined on the Store class was a function get_absolute_url whose implementation looked roughly like this:
def get_absolute_url(self, to='/'):
"""
Return an absolute url to this `Store` or to `to` on this store.
The URL includes http:// and the domain name of the store.
`to` can be an object with `get_absolute_url()` or an absolute path as string.
"""
if isinstance(to, basestring):
path = to
elif hasattr(to, 'get_absolute_url'):
path = to.get_absolute_url()
else:
raise ValueError(
'Invalid argument (need a string or an object with get_absolute_url): %s' % to
)
url = 'http://%s%s%s' % (
self.domain,
# This setting allowed for a sane development environment
# where I just set it to ".dev:8000" and configured `dnsmasq`.
# The same value was also removed from the `Host` value in the middleware
# before looking up the `Store` in database.
settings.DOMAIN_SUFFIX,
path
)
return url
So I could easily generate URLs to objects on other than the current store, e.g.:
# Redirect to `product` on `store`.
redirect(store.get_absolute_url(product))
This was basically all I needed to be able to implement a system allowing users to create a new e-shop living on its own domain via the Django admin.
So I'm working on a website where there are a couple important values that get used in various places throughout the site. For example, certain important dates, like the start and end dates for registration.
One way I can do this is making a model that stores these values, but that sounds like overkill (since I'd only have one instance). Another way is to store these values in the settings.py file, but if I wanted to change them, it seems like I would need to restart the webserver for them to take effect. I was wondering what would be the best practice in Django to handle this kind of stuff.
You can store them in settings.py. While there is nothing wrong with this (you can even organize your settings into multiple different files, if you have to many custom settings), you're right that you cannot change these at runtime.
We were solving the same problem where I work and came up with a simple app called django-constance (you can get it from github at https://github.com/comoga/django-constance). What this lets is store your settings in a settings.py, but once you need to turn them into settings configurable at runtime, you can switch to a Redis data store with django admin frontend. You can even use the value from settings as your default. I suggest you try this app out.
The changes to your code are pretty minimal, as pasted from docs you initialize your dynamic settings like this:
CONSTANCE_CONFIG = {
'MY_SETTINGS_KEY': (42, 'the answer to everything'),
}
And then instead of importing settings from django conf, you do this:
from constance import config
if config.MY_SETTINGS_KEY == 42:
answer_the_question()
If you want a specific set of variables available to all of your template, what you are looking for is Context Processors.
http://docs.djangoproject.com/en/dev/ref/templates/api/#writing-your-own-context-processors
More links
http://www.b-list.org/weblog/2006/jun/14/django-tips-template-context-processors/
http://blog.madpython.com/2010/04/07/django-context-processors-best-practice/
The code for your context processors, can live anywhere in your project. You just have to add it to your settings.py under:
TEMPLATE_CONTEXT_PROCESSORS =
You could keep the define your constants in your settings.py or even under a constants.py and just
from constants import *
However as you mentioned, you would need to reload your server each time the settings are updated. I think you first need to figure out how often will you be changing these settings? Is it worth the extra effort to be able to reload the settings automatically?
If you wanted to automatically enable the settings, each time they are updated you could do the following:
Store settings in the DB
Upon save/change, write output to a file
settings.py / constants.py reads files
reload server
In addition, you have a look at the mezzanine project which allows you to update settings from the django admin interface and will reload as well.
See: http://mezzanine.jupo.org/docs/configuration.html
If the variables you need will be updated infrequently, i suggest just store them in settings.py and add a custom context processor.
If you are using source control such as GIT, updating will be quite easy, you can just update the file and push to your server. For really simple reloading of the server you could also create a post-recieve hook for git that will automatically reload the server when new code is pushed.
I would only suggest the other option if you are updating settings fairly regularly.