The django docs clearly states
You shouldn't alter settings in your applications at runtime.
Here's the link to that statement
My question is, why is this so? I want to add applications dynamically at runtime, and add databases at runtime, both of which involve editing the settings. Can someone explain why settings are not to be edited at runtime and if exceptions exist, which settings they are and why they are exceptional? I'm not so much interested in how to achieve my goal, but in the reason for why settings shouldn't be altered.
Most settings will not be re-read if you change them at runtime. So Django will not recognise the changes you make.
This is due to the fact that Django is just normal Python code. It isn't like a server that is monitoring your code - it is just part of your code.
In some cases, parts of Django code might respond to changes in settings, because they might do 'settings.DEFAULT_FROM_EMAIL' every time mail is sent, for example.
But if Django processes the setting in any way, like it has to do for INSTALLED_APPS, it isn't going to notice you changed something and re-do the processing.
Which settings are safe? Well, the docs are saying "none are safe", because it might change in the future. Django might save a copy of any setting for some reason, or do some processing.
Changing INSTALLED_APPS could never be made to work, because it alters which modules are imported. There is simply no way that Django could work around the way that Python works at this level - it would need to be able to 'unimport' modules, which is basically impossible (the only way is to restart the process), and there are other problems associated with cross-app links.
AFAIK there is no documentation on which settings are safely modifiable at runtime, but there is an open ticket asking that they be documented more clearly.
If you take a look under the hood at the settings object Django exposes to interface with your project's settings module, you'll notice that there's nothing preventing you to dynamically change the settings at runtime.
You should, however, appreciate that the framework's architecture is built around a request-response flow where a lot of global state is being shared between threads for memory optimization, based on the premise that the application is configured only once during initialization.
Related
I want my code to behave a tiny bit differently in development than in production; for example, don't actually post things on facebook when the dev profile is activated. Right now I'm thinking I can use robert-hooke to add hooks to functions I don't want run in development; however, how can I check which profiles are activated?
I've also checked out environ which looks great for development vs production configurations but doesn't seem to hit my problem.
I don't think this is a rare problem so if there's already some accepted ways to handle this; great.
If you take a look at the luminus guestbook example, it's actually using profiles to set an environment variable :dev, and then environ to read it back from within the application. Environ suggests using the 12 factor app as a model, which makes an argument against grouping configurations inside of the application. Leiningen let's us have the best of both by naming the configuration group external to the actual application. Unfortunately the variable passed to the application is named the same as the profile, and thus groups configurations in the app. Naming it cache.disable but leaving it in the dev profile could fix that.
You could also take a look at isolating dependencies for development. The article has an example near the end using System/getenv that could also use environ as a replacement.
I maintain an installable Django app that includes a regular test suite.
Naturally enough when project authors run manage.py test for their site, the tests for both their own apps and also any third party installed apps such as mine will all run.
The problem that I'm seeing is that in several different cases, the user's particular settings.py will contain configurations that cause my app's tests to fail.
A couple of examples:
Some of the tests need to check for returned error messages. These error messages use the internationalization framework, so if the site language is not english then these tests fail.
Some of the tests need to check for particular template output. If the site is using customized templates (which the app supports) then the tests will end up using their customized templates in preference to the defaults, and again the tests will fail.
I want to try to figure out a sensible approach to isolating the environment that my tests get run with in order to avoid this.
My plan at the moment is to have all my TestCase classes extend a base TestCase, which overrides the settings, and any other environment setup I may need to take care of.
My questions are:
Is this the best approach to app-level test-environment isolation? Is there an alternative I've missed?
It looks like I can only override a setting at a time, when ideally I'd probably like a completely clean configuration. Is there be a way to do this, and if not which are the main settings I need to make sure are set in order to have a basic clean setup?
I believe I'm correct in saying that overriding some settings such as INSTALLED_APPS may not actually affect the environment in the expected way due to implementation details, and global state issues. Is this correct? Which settings do I need to be aware of, and what globally cached environment information may not be affected as expected?
What other environment state other than settings might I need to ensure is clean?
More generally, I'd also be interested in any context regarding how much of an issue this is for other third party installable apps, or if there are any plans to further address any of this in core. I've seen conversation on IRC regarding similar issues with eg. some of Django's contrib apps running under unexpected settings configurations. I seem to also remember running into similar cases with both third party apps and django contrib apps a few times, so it feels like I'm not alone in facing these kind of problems, but it's not clear if there's a consensus on if this is something that needs more work or if the status quo is good enough.
Note that:
These are integration-level tests, so I want to address these environment issues at the global level.
I need to support Django 1.3, but can put in some compatibility wrappers so long as I'm not re-implementing massive amounts of Django code.
Obviously enough, since this is an installable app, I can't just specify my own DJANGO_SETTINGS_MODULE to be used for the tests.
A nice approach to isolation I've seen used by Jezdez is to have a submodule called my_app.tests which contains all the test code (example). This means that those tests are NOT run by default when someone installs your app, so they don't get random phantom test failures, but if they want to check that they haven't inadvertently broken something then it's as simple as adding myapp.tests to INSTALLED_APPS to get it to run.
Within the tests, you can do your best to ensure that the correct environment exists using override_settings (if this isn't in 1.4 then there's not that much code to it). Personally my feeling is that with integration type tests perhaps it doesn't matter if they fail. If you like, you can include a clean settings file (compressor.test_settings), which for a major project may be more appropriate.
An alternative is that you separate your tests out a bit - there are two separate bodies of tests for contrib.admin, those at django.contrib.admin.tests, and those at tests.regression_tests.contrib.admin (or some path like that). The ones to check public apis and core functionality (should) reside in the first, and anything likely to get broken by someone else's (reasonable) configuration resides in the second.
IMHO, the whole running external apps tests is totally broken. It certainly shouldn't happen by default (and there are discussions to that effect) and it shouldn't even be a thing - if someone's external app test suite is broken by my monkey patching (or whatever) I don't actually care - and I definitely don't want it to break the build of my site. That said, the above approaches allow those who disagree to run them fairly easily. Jezdez probably has as many major pluggable apps as anyone else, and even if there are some subtle issues with his approach at least there is consistency of behaviour.
Since you're releasing a reusable third-party application, I don't see any reason the developer using the application should be changing the code. If the code isn't changing, the developers shouldn't need to run your tests.
The best solution, IMO, is to have the tests sit outside of the installable package. When you install Django and run manage.py tests, you don't run the Django test suite, because you trust the version of Django you've installed is stable. This should be the same for developers using your third-party application.
If there are specific settings you want to ensure work your library, just write test cases that use those settings values.
Here's an example reusable django application that has the tests sit outside of the installed package:
https://github.com/Yipit/django-roughage/tree/master
It's a popular way to develop python modules as seen:
https://github.com/kennethreitz/requests
https://github.com/getsentry/sentry
I am using mongoengine and would like to run connect() after settings (not inside them as suggested in its docs). This is actually more like a general question how to run code right after all settings are loaded.
Update: I need a solution for management commands as well. Common approach is adding a middleware with exception MiddlewareNotUsed or adding code to root urls.py, but both don't work for commands.
The normal place for startup-like code is in a urls.py (when you need the settings to be already loaded). Django doesn't have a good spot yet for this.
(There is an "app refactor" branch that a gsoc student worked on in 2011, but it didn't get merged into core django yet. This "app refactor" includes a solution to your very problem, but that doesn't help you...)
You mention that a management command also needs it. Is that your own management command? Nothing stops you from importing the urls.py there, is it?
This is sadly one of the few Django weak points. Luckily there aren't that many :-)
In a large Django project, I have several evil monkey-patching hacks to execute at application startup time. However, I don't quite see a correct place for such hackery: neither urls.py nor settings.py nor manage.py seem fit for this. Where would you recommend I put those?
In python you will always come across initialization. That's why its always better to use init for initialization. Even in django when you create a project it must have an init.py in it.
I usually put all my initialization in __init__.py its a safe and clean way. You can do the same rather than create another initialization module.
There isn't really a good answer to this question at the moment. There's a Summer of Code project at the moment to rewrite the app loading process, which hopefully will include hooks for initialization code.
In the meantime, I think the best place for this is in urls.py. The admin application and Haystack both do it there, and it seems a good pattern.
I have a django project, and a problem - it eats a lot of memory and loads hosting too much.
How can I find the problem places in the project which eat a lot of memory?
If you're using Django with DEBUG = True then Django logs every database query which can quickly mount up and use a substantial amount of memory.
If you're not running in DEBUG mode, then take a look at gc module and in particular try adding gc.set_debug(gc.DEBUG_LEAK) to settings.py. This will show you a great deal of information about what objects are using memory.
In general for debugging/profiling, I suggest django-debug-toolbar as a starting location as well as the various tips in:
http://docs.djangoproject.com/en/dev/topics/db/optimization/
However this won't give memory usage info. If you really need that, you can try some middleware using pympler to log memory usage while debugging and run the development server.
http://www.rkblog.rk.edu.pl/w/p/profiling-django-object-size-and-memory-usage-pympler/?c=1
I've found that doing this grinds my webapps to a near-halt and then there are the problems from using the dev-webserver (e.g., media files not getting served).
But as others said your best bet is to set DEBUG=False:
http://docs.djangoproject.com/en/dev/faq/models/#why-is-django-leaking-memory
As Andrew Wilkinson stated this might have to do with the DEBUG = True setting. However it might also be important to know if you're running this project stand-alone or as a webserver.
A Django will automaticly cache querysets when opening a request and remove the references when the request returns. Since there are no requests in a stand-alone project the references are never delete and hence every queryset ever requested get saved.
To fix the stand-alone python issue simply call django.db.reset_queries() after you've done a bunch of request. This will allow the querysets to be garbage collected and fix your leak.