I'm trying to use django-cache-machine to cache queries within my application, but I want to use Redis as a backend. The docs don't really explain how to do this, yet the repository is filled with Redis references, so I'm pretty sure it's possible. I want to make sure I do it right though, so I'm wondering if anyone has any experience with configuring this and maybe more importantly, knows if there are any caveats?
In your settings set:
CACHE_MACHINE_USE_REDIS = True
REDIS_BACKEND = redis://127.0.0.1:6379?socket_timeout=0.1
https://github.com/jbalogh/django-cache-machine/blob/master/caching/invalidation.py#L187
https://github.com/jbalogh/django-cache-machine/blob/master/caching/invalidation.py#L213
I have a little experience in my project, a report system that generate tables from about 50 million records.
The database is Mysql and I could show my settings and models FYI.
settings:
# cache machine
CACHES = {
'default': {
'BACKEND': 'caching.backends.memcached.MemcachedCache',
'LOCATION': [
'127.0.0.1:11211',
],
'PREFIX': 'report:',
},
}
CACHE_COUNT_TIMEOUT = 60 * 24 # one day
CACHE_EMPTY_QUERYSETS = True
models:
class App(**CachingMixin**, models.Model):
**objects = CachingManager()**
name = models.CharField(max_length=64,
default='')
Note that cache-machine works fine for query_set.filter and count, not good for query_set.annotate or aggregate. Of course do not forget launch your memcache client first.
And when running you can see cache-machine logs in your django*.log to tell you hit or miss cache.
Related
Problem:
I want to figure out a way to disable throttling when running my tests with pytest -vv
Details:
I have this default throttling policy in my settings.py file:
'DEFAULT_THROTTLE_RATES': {
'anon': '100/day',
'user': '1000/day'
}
I also have this fixture in my confest.py which returns an error whenever I exceed the limit of requests:
def get_token(user, client):
response = client.post(
"/email-login",
{"email":user.email, "password": "B9vX95phJDi3C4"},
)
return {
"HTTP_AUTHORIZATION": f"Bearer {response.json()['token']['access']}"
}
What I have tried:
I have attempted to use the solution in this GitHub Issue: https://github.com/encode/django-rest-framework/issues/1336, but it doesn't work in my case.
First you need to create a way to differentiate between test env and otherwise. Like we do for PROD and DEV using settings.DEBUG config.
My recommendation is to create an env variable test=Trueand then in your settings.py write -
if os.environ.get("test", False):
REST_FRAMEWORK = {
'DEFAULT_THROTTLE_CLASSES': [
'rest_framework.throttling.AnonRateThrottle',
'rest_framework.throttling.UserRateThrottle'
],
'DEFAULT_THROTTLE_RATES': {
'anon': '100/day',
'user': '1000/day'
}
}
else it does nothing, and drf will not throttle.
I was able to resolve this problem with the following steps:
I created a new settings file which inherited from the base settings file. i.e from settings import *
Then I deleted the DEFAULT_THROTTLE_RATES key i.e del REST_FRAMEWORK["DEFAULT_THROTTLE_RATES"]
Next thing I did was to point to the new settings file in pytest.ini i.e DJANGO_SETTINGS_MODULE="new_settings.py"
Now the tests will use the new settings file
#ra123 has the right idea in general. As another approach, with all Django projects I add something like this to my settings/__init__.py (or just settings.py if you do a one file thing). It looks at argv to see if its in test mode
IS_TESTING = bool(set(sys.argv[:2]) & {"pytest", "test", "jenkins"})
REST_FRAMEWORK = { "YOUR_CONFIG": "..." }
# at the very very end, AFTER your settings are loaded:
if IS_TESTING:
# override your rest framework settings in test mode
REST_FRAMEWORK["DEFAULT_THROTTLE_CLASSES"] = []
# some other handy things, for making tests faster/easier
PASSWORD_HASHERS = ("django.contrib.auth.hashers.MD5PasswordHasher",)
EMAIL_BACKEND = "django.core.mail.backends.locmem.EmailBackend"
DEFAULT_FILE_STORAGE = "inmemorystorage.InMemoryStorage"
I ended up with it this way so we don't have to worry about it ever getting the wrong settings. It also helps keep things centralized, so (for example) you don't call sentry.init in testing mode, even if there is a sentry_url in the environment.
How this could be done?
For now I get only indexes that are addressed to the page titles. Like cms.title objects are indexed.
I have add the app like intended:
#settings.py
#TODO: It's not recommended for production use but it will return results.
HAYSTACK_CONNECTIONS = {
'default': {
'ENGINE': 'haystack.backends.simple_backend.SimpleEngine',
},
}
HAYSTACK_ROUTERS = ['aldryn_search.router.LanguageRouter', ]
ALDRYN_SEARCH_REGISTER_APPHOOK = True
Is there any trigger that I'm unaware of?
Though it's a old question, i hope this may help other people
I also added
INSTALLED_APPS = [ 'aldryn_search', 'spurl', 'haystack', 'standard_form' ]
see Mention that the aldryn search view requires standard_form #87
Well, I have a django project which works fine now.
I'd like to add a new app to it, in which I need to access multiple databases.
I know Django support multiple databases settings and know how to configure it. This is not the problem.
The issue is that, in the 90% of my project components, I don't need to maintain a multiple databases settings. The only usage for the second databases is in the new added app.
So I tried to alter the settings by calling:
django.conf.settings.configure(DATABASES = {....})
in the new app. And django said:
RuntimeError: Settings already configured.
Which makes sense, since I have the origin settings file and set the DJANGO_SETTINGS_MODULE up.
So I question is that what should be a good approach in this case.
I don't want to discard DJANGO_SETTINGS_MODULE variable.
I try to not include second database in the setting file initially, since the new app is also an independent module which should work independently outside the django project. So I want to have the similar config in new app to setup the database config.
Does anyone has any idea about this?
Thanks in advance!
Actually, I have the same issue in a current project. As you I have a totally independent app which uses an another database, and I could have other apps which could have the same behaviour.
The thing that I have done is to create a dir apps where I store my apps and then I add this at the end of my settings.py file :
DATABASE_ROUTERS = ['myproject.routers.MultiDBRouter']
import os
APPS_DIR = os.path.join(PROJECT_ROOT, 'apps')
for app_name in os.listdir(APPS_DIR):
print '\nLooking for settings in apps/%s :' % app_name
if os.path.exists(os.path.join(APPS_DIR, app_name, 'settings.py')):
print ' Settings file found...'
app = __import__('%s.settings' % app_name)
content = dir(app.settings)
if 'DATABASES' in content:
print ' Adding databases :'
for key, value in app.settings.DATABASES.iteritems():
if DATABASES.has_key(key):
print ' Can not add %s database config, because it already exists' % key
else:
DATABASES[key] = value
DATABASES[key]['APPS'] = [app_name]
print ' Added %s database config' % key
It will automatically look after settings.py file in all the apps/myapp/ directories. If it finds a new DATABASES variable in a app/myapp/settings.py file, it will add the other database configurations to your DATABASES variable (the true one).
I have also created a router to do not have to use the using command (the MultiDBRouter).
And then I add a settings.py file in all the app which requires another database :
DATABASES = {
'db': {
'ENGINE': 'django.db.backends.mysql',
'NAME': 'database',
'USER': 'username',
'PASSWORD': 'mysecretpassword',
}
}
So Im running into this CACHE error when I try to runserver or syncdb.
Here is the traceback: https://gist.github.com/1538051
I tried inserting this into the settings.py file:
CACHE_BACKEND = {
'default': {
'BACKEND': 'django.core.cache.backends.dummy.DummyCache',
}
}
But that gave another error which makes no sense to me.
if backend_uri.find(':') == -1:
AttributeError: 'dict' object has no attribute 'find'
Can someone help me figure what the problem is and how I can go about fixing it.
NB: I am working on the dev server
If you're using Django 1.2 or lower, CACHE_BACKEND (docs) accepts a string:
CACHE_BACKEND = 'dummy://'
If you're using Django 1.3+, CACHE_BACKEND has been deprecated in favour of CACHES (docs):
CACHES = {
'default': {
'BACKEND': 'django.core.cache.backends.dummy.DummyCache',
}
}
I'm not sure why your installation isn't creating the appropriate default - it could be that there's something else going on with your installation, but I don't really have enough information without knowing more about your settings.py etc.
First off, What version of django are you using? The dictionary- style backend config is new to django 1.3, and your traceback suggests that you are on something like 1.2.4.
If that's the case, you will need to use the older uri-style cache settings. Something like
CACHE_BACKEND = 'dummy://'
would match what you are trying to set in your question.
The full documentation on caching, relevant to Django 1.2, is available here: http://docs.djangoproject.com/en/1.2/topics/cache/
I've been evaluating django and wondered if the following is possible. I've already looked at the regular multiple database docs so please don't point me to that because this use case isn't mentioned as far as i can make out. If i'm wrong i take it back :)
I want one main database in which most of my app's models will reside, however one of the app's will need to dynamically create databases, these will be customer specific databases.
The database path (i plan to use sqlite) will be stored in primary database and so the cursor would need to be changed but the model will remain the same.
I would welcome any thoughts on ways to achieve this?
I will open with "You should not edit settings at runtime".
Having said that, I have exactly this same issue, where I want to create a unique database for each user. The reason for doing this is I am offering the ability for the user to save/access to/from a database not stored on my server, which entails having multiple databases, and thus one for each user.
This answer is NOT the recommended way to achieve the desired goal. I would love to hear from a django-guru how to best approach this problem. However, this is a solution I have been using and it has worked well so far. I am using sqlite however it can be easily modified for any of the databases.
In summary, this is the process:
Add the new database to settings (at runtime)
Create a file to store these settings for reloading when the server is restarted (at runtime)
Run a script which loads the saved settings files (whenever the server is restarted)
Now, how to achieve this:
1) Firstly, when a new user is created, I create a new database in the settings. This code lives in my view where new users are created.
from YOUR_PROJECT_NAME import settings
database_id = user.username #just something unique
newDatabase = {}
newDatabase["id"] = database_id
newDatabase['ENGINE'] = 'django.db.backends.sqlite3'
newDatabase['NAME'] = '/path/to/db_%s.sql' % database_id
newDatabase['USER'] = ''
newDatabase['PASSWORD'] = ''
newDatabase['HOST'] = ''
newDatabase['PORT'] = ''
settings.DATABASES[database_id] = newDatabase
save_db_settings_to_file(newDatabase) #this is for step 2)
This script loads the database settings 'at runtime' into the django project settings. However if the server is restarted, this database will no longer be in settings.
2) To facilitate reloading these settings automatically whenever the server is restarted, I create a file for each database which will be loaded whenever the server is started. Creating this file is performed by the function save_db_settings_to_file:
def save_db_settings_to_file(db_settings):
path_to_store_settings = "/path/to/your/project/YOUR_PROJECT_NAME/database_settings/"
newDbString = """
DATABASES['%(id)s'] = {
'ENGINE': '%(ENGINE)s', # Add 'postgresql_psycopg2', 'mysql', 'sqlite3' or 'oracle'.
'NAME': '%(NAME)s', # Or path to database file if using sqlite3.
'USER': '', # Not used with sqlite3.
'PASSWORD': '', # Not used with sqlite3.
'HOST': '', # Set to empty string for localhost. Not used with sqlite3.
'PORT': '', # Set to empty string for default. Not used with sqlite3.
}
""" % db_settings
file_to_store_settings = os.path.join(path_to_store_settings, db_settings['id'] + ".py")
write_file(file_to_store_settings, newDbString) #psuedocode for compactness
3) To actually load these settings when the server is started, I add a single line to the very bottom of /path/to/your/project/YOUR_PROJECT_NAME/settings.py, which loads each file in the settings folder and runs it, having the effect of loading the database details into the settings.
import settings_manager
Then, import settings_manager will load the file at /path/to/your/project/YOUR_PROJECT_NAME/settings_manager.py, which contains the following code:
from settings import DATABASES
import os
path_to_store_settings = "/path/to/your/project/YOUR_PROJECT_NAME/database_settings/"
for fname in os.listdir(path_to_settings):
full_path = os.path.join(path_to_settings, fname)
f = open(full_path)
content = f.read()
f.close()
exec(content) #you'd better be sure that the file doesn't contain anything malicious
Note that you could put this code directly at the bottom of settings.py instead of the import statement, but using the import statement keeps the abstraction level of settings.py consistent.
This is a convenient way to load each database setting because to remove a database from the settings all you have to do is delete the settings file, and the next time the server restarts it won't load those details into the settings, and the database will not be accessible.
As I said, this works and I have had success using it so far, but this is NOT the ideal solution. I would really appreciate if someone could post a better solution.
What's bad about it:
It explicitly defies advice from django team not to modify settings at runtime. I do not know the reason for why this advice is given.
It uses an exec statement to load the data into settings. This should be OK, but if you get some corrupt or malicious code in one of those files you will be a sad panda.
Note that I still use the default database for auth and sessions data, but all the data from my own apps is stored in the user-specific database.
To augment #thedawnrider's answer, in some cases editing settings.DATABASES may not be enough. It might be more reliable to edit django.db.connections.databases, which serves as a cache and wrapper around settings.DATABASES.
e.g.
from django.db import connections
database_id = user.username #just something unique
newDatabase = {}
newDatabase["id"] = database_id
newDatabase['ENGINE'] = 'django.db.backends.sqlite3'
newDatabase['NAME'] = '/path/to/db_%s.sql' % database_id
newDatabase['USER'] = ''
newDatabase['PASSWORD'] = ''
newDatabase['HOST'] = ''
newDatabase['PORT'] = ''
connections.databases[database_id] = newDatabase
This question is pretty outdated and should be updated.
Django support multiple databases out of the box
Here is many good tutorials [ 1, 2 ] and answers how to manage connections dynamically
Also you may simply use django-dynamic-db-router
Database connection settings may be configured on the fly by using configure method of settings object:
from django.conf import settings
dbs = settings.DATABASES.copy()
dbs['some_new_db'] = {'ENGINE': 'dummy'}
settings.configure(DATABASES=dbs)