I'm trying to cache access to the Django profile object. I'm using django-redis-cache to cache data in this project. I'm using a snippet for automatically creating a profile if one does not exist. Here is a simplified version of what I am doing (without caching):
User.profile = property(lambda u: UserProfile.objects.get_or_create(user=u)[0])
Whenever profile information is needed, the user.profile property is accessed. That works as expected, however, when I try to cache the profile property, such as in Exhibit 1, I still see SQL queries (in django-debug-toolbar) that are selecting the profile and are not taking advantage of the cache.
Specifically, the cache_object_list() function from Exhibit 2 is a bit of code that checks to see if a cached value is available. If it is, it calls the cache key. If not, it runs the query passed to it (via the "query" argument) and caches the results.
cache_object_list() prints "Hit" or "Miss" indicating a cache hit or miss. After refreshing twice, everything is reported as a hit (as expected). However, django-debug-toolbar still shows no reduction in query count and shows queries selecting the profile.
Does anyone have any advice as to how to ensure that the user.profile pulls a cached version of the profile when available? Thanks for reading.
Exhibit 1: myproject/myapp/models.py
def get_or_create_profile(u):
return cache_utils.cache_single_object(
"user_get_or_create_profile",
u.id, UserProfile.objects.get_or_create(user=u)[0])
User.profile = property(lambda u: cache_utils.cache_single_object(
"user_get_or_create_profile", u.id,
get_or_create_profile(u)))
Exhibit 2: myproject/cache_utils.py
def cache_single_object(key_prefix, id, query, timeout=500):
key = '%s_%s' % (key_prefix, id)
object_list = cache.get(key, None)
if object_list is None:
print "Miss %s" % (key)
object_list = query
cache.set(key, object_list, timeout)
else:
print "Hit %s" % (key)
return object_list
Exhibit 3: myproject/templates/mytemplate.py
<div>Example of what's in the template </div>
{{ myobject.owner.profile.bio }}
I think the problem is related to the way you defined your method....
User.profile = property(lambda u: cache_utils.cache_single_object(
"user_get_or_create_profile", u.id,
get_or_create_profile(u)))
when you access the profile property, you will always call the method get_or_create_profile(u) that calls:
return cache_utils.cache_single_object(
"user_get_or_create_profile",
u.id, UserProfile.objects.get_or_create(user=u)[0])
having UserProfile.objects.get_or_create(user=u) there is what is creating your query every single time even if you already have the data in the cache. I think you should try using a util method where you don't evaluate the query every time you call it. Maybe something like this: https://stackoverflow.com/a/2216326/234304
Related
I have a simple(I think) question, about Django context rendering.
I'll step right into it -
Basically what I need is, some temp table, which in my case, I called Locked. And when a user presses a button, which Is a form, that object goes straight to the table Locked(just a simple insert). And inside that table there is a field called is_locked, and if its True, that object needs to go gray, or to have some lock icon inside the html table.
Just some kind of a viewable sign, that an object is inside the table Locked, and that another user can't access it.
But, my problem is, since in my views.py, my lock function is not returning exact html where I want to render that locker icon, instead, it returns another html.
Is there any way, to render same context, on 2 html pages? Thank's.
This is my code :
views.py
def lock(request, pk):
# Linking by pk.
opp = get_object_or_404(OpportunityList, pk=pk)
opp_locked = get_object_or_404(Locked, pk=pk)
# Taking two parametters for 2 fields.
eluid = Elementiur.objects.get(eluid=pk)
user = User.objects.get(username=request.user)
# Dont bother with this one! Just pulling one field value.
field_name = 'optika_korisnik'
obj = OpportunityList.objects.get(pk=pk)
field_object = OpportunityList._meta.get_field(field_name)
field_value = getattr(obj, field_object.attname)
# This is the main part! This is where i'm inserting data into Locked table.
if opp_locked.DoesNotExist:
opp_locked.id = int(eluid.eluid)
opp_locked.locked_eluid = eluid
opp_locked.locked_comment = field_value
opp_locked.locked_user = user
opp_locked.locked_name = 'Zaključao korisnik - ' + request.user.username
opp_locked.is_locked = True
opp_locked.save()
# This is what has to be returned, but i need context on the other page.
return render(request, 'opportunity/detalji/poziv.html',
{'opp': opp, 'locked': opp_locked})
else:
# This return has the context that i need(from the first opp_locked variable)
return render(request, 'opportunity/opp_optika.html', {'locked_test': opp_locked})
I can provide more code, but i think that it's not important for this type of question, because all of the logic is happening inside the lock finction, and last two returns.
I just had a quick overview of your snippet sorry if this not help you but you need to review it a little bit.
You call DoesNotExist on an instance of a Locked model
if opp_locked.DoesNotExist: [...]
that's not how you should use this exception.
You have a method .exists() that is available but only for Querysets.
Also if your instance does not exists you are alredy returning an Http404 response when you use get_object_or_404() method.
And perhaps you should avoid sharing primary keys between instances and replace them with models.OneToOneField (OneToOnefield)
Since i got no answers, i added a new field, is_locked, into my Locked model and that solved it.
I am trying to find out if it is safe to do the following:
items = MyModel.objects.filter(q)
if items:
list(items)
I know that the QuerySet is being evaluated in the if statement, checking if the database returns an empty set. But I want to know if the value of that evaluation is reused when doing list(items). Is the QuerySet being evaluated here too, or is it using the previously evaluated one?
I know I could just do the following:
items = MyModel.objects.filter(q).all()
if items:
list(items)
And this would result in one evaluation, but I am just trying to find out the behavior the first variation has. I have gone throught these pieces of doc (1 2) but couldn't really find a straight answer to this matter.
No. Both will not execute twice (internally .filter(), .all() and .filter().all() are same). You can check it in django shell itself
from django.db import connection
print connection.queries
items = MyModel.objects.filter(q).all() #or MyModel.objects.filter(q)
if items:
list(items)
print connection.queries
Then here is the magic of .all()
queryset = MyModel.objects.all() #no db hit, hit=0
print list(queryset) #db hit, hit=1
print list(queryset) #no db hit, hit=1
print list(queryset.all()) #db hit, hit=2
print list(queryset.all()) #db hit, hit=3
That means .all() on an evaluated queryset will force db hit.
When a QuerySet is evaluated, it typically caches its results. If the data in the database might have changed since a QuerySet was evaluated, you can get updated results for the same query by calling all() on a previously evaluated QuerySet
It will reuse it's cache, because when you do
if items:
It will call __bool__ method
def __bool__(self):
self._fetch_all()
return bool(self._result_cache)
So as you see inside __bool__ it does call _fetch_all. Which caches data
def _fetch_all(self):
if self._result_cache is None:
self._result_cache = list(self.iterator())
if self._prefetch_related_lookups and not self._prefetch_done:
self._prefetch_related_objects()
For better perfomance do:
items = MyModel.objects.filter(q) # no evaluation
if items.exists(): # evaluates, hits db
# do stuff here # further actions evaluates, hits db
At a Django server, I have a page with a single button that starts a function.
This function takes a while to complete, and I tried writing updates on the process to a request.session variable with the intent of checking on its contents from a separate page.
It seems however that request.session variables are not updated until the function they are included in are done. At least the variable does not update until then.
Am I right, and if so, is there a way to write to request.session variable before the function's completion?
The sessions are set up properly, I can write and read variables with other examples. For now I'll also just make a temporary db record to store the status update info and read it from there, but I'm curious as of this request.session thing - is my guess correct, and is there a way around?
update:
views.py
#login_required
def autolink(request):
result, time = access_check(request, 'super')
if not result:
return redirect('index')
result = f_autolink(request)
if result is None:
result = request.session.get('technical', '')
return render(request, 'autolink.html', {'result': result, })
functions.py
def f_autolink(request):
if request.method == 'GET':
result = None
elif request.method == 'POST':
request.session['technical'] = 'starting the job'
result = f_kzd_autolink(request)
else:
result = None
return result
def f_kzd_autolink(request):
homeless = Kzd.objects.filter(payment__isnull=True, returned=False).distinct()
result = []
count = homeless.count()
counter = 0
request.session['technical'] = 'Starting link job - {c} records to process'.format(c=count)
for h in homeless:
counter += 1
request.session['technical'] = 'Checking record {c1} of {c}'.format(c1=counter, c=count)
/* long code that makes the h in homeless cycle run for about 3 minutes, unrelated to the question */
so basically, the view shows request.session.get('technical', ''), but neither function writes to it until they are done (it then writes about processing the last record).
The session is saved on a per-request basis when it was modified or when the setting settings.SESSION_SAVE_EVERY_REQUEST is set to True in your settings.
So the simple answer yes, the session is saved by the session middleware when processing the response created by a view. But you could do it manually by calling request.session.save() inside your view.
If you have code, that runs very long it would be better to imediately create a response and use tools like celery to asynchronously process your task.
And you should consider storing your data into an own database table/ own model if it is not really related to the user session.
Here is sample codes in django.
[Case 1]
views.py
from sampleapp.models import SampleModel
from django.core.cache import cache
def get_filtered_data():
result = cache.get("result")
# make cache if result not exists
if not result:
result = SampleModel.objects.filter(field_A="foo")
cache.set("result", result)
return render_to_response('template.html', locals(), context_instance=RequestContext(request))
template.html
{% for case in result %}
<p>{{ case.field_A}}</p>
{% endfor %}
In this case, there's no generated query after cache made. I checked it by django_debug_toolbar.
[Case 2]
views.py - added one line result = result.order_by('?')
from sampleapp.models import SampleModel
from django.core.cache import cache
def get_filtered_data():
result = cache.get("result")
# make cache if result not exists
if not result:
result = SampleModel.objects.filter(field_A="foo")
cache.set("result", result)
result = result.order_by('?')
return render_to_response('template.html', locals(), context_instance=RequestContext(request))
template.html - same as previous one
In this case, it generated new query even though I cached filtered query.
How can I adapt random ordering without additional queryset?
I can't put order_by('?') when making a cache.
(e.g. result = SampleModel.objects.filter(field_A="foo").order_by('?'))
Because it even caches random order.
Is it related with 'django queryset is lazy' ?
Thanks in advance.
.order_by performs sorting at database level.
Here is an example. We store lasy queryset in var results. No query has been made yet:
results = SampleModel.objects.filter(field_A="foo")
Touch the results, for example, by iterating it:
for r in results: # here query was send to database
# ...
Now, if we'll do it again, no attempt to database will be made, as we already have this exact query:
for r in results: # no query send to database
# ...
But, when you apply .order_by, the query will be different. So, django has to send new request to database:
for r in results.order_by('?'): # new query was send to database
# ...
Solution
When you do the query in django, and you know, that you will get all elements from that query (i.e., no OFFSET and LIMIT), then you can process those elements in python, after you get them from database.
results = list(SampleModel.objects.filter(field_A="foo")) # convert here queryset to list
At that line query was made and you have all elements in results.
If you need to get random order, do it in python now:
from random import shuffle
shuffle(results)
After that, results will have random order without additional query being send to database.
I am trying to implement a MultiValueField for IP Adress/Domain Name entries. It works as expected for entering data.
My Problem is that if I want to display the form bound to specific data, the IP Address/Domain Name field stays empty. All other fields are filled with the desired data. If I use a normal CharField, I get the data that I would expect. But it does not work with my custom field.
I have tracked it down to the fact that my custom MultiWidget does not call its decompress method.
Here is my Field:
class accessIPField(forms.MultiValueField):
"""
custom Field for access IP
"""
def __init__(self, *args, **kwargs):
self.fields=(
forms.IPAddressField(label='IP Adress'),
forms.CharField(max_length=50,label='Domain Name')
)
self.widget=accessIPWidget()
super(accessIPField,self).__init__(self.fields,self.widget, *args, **kwargs)
def compress(self,data_list):
if data_list:
return " ".join(data_list)
And here is my widget:
class accessIPWidget(forms.MultiWidget):
"""
Widget to display IP Adress / Domain name pairs
"""
def __init__(self,*args,**kwargs):
self.widgets=(forms.TextInput(),forms.TextInput())
super(accessIPWidget,self).__init__(self.widgets,*args,**kwargs)
def decompress(self,value):
print 'decompress called'
if value:
return value.rsplit()
return [None,None]
def format_output(self, rendered_widgets):
return u'\n'.join(rendered_widgets)
The whole thing is called (in a larger context) as
self.fields['access_IPs'] = accessIPField()
Now as you can see, I put a print statement in my compress method, and I never get to see that statement. Also, if I rename compress to something like foobar, I would expect (according to the django code for MultiWidget) to get the NotImplementedError, which is not the case. Any suggestions?
I am using python 2.6.5, django 1.1 on ubuntu server 10.04.
It turns out that the problem was with the value_from_datadict() method as implemented by MultiWidget. First of all, it allready returned a list, so that is why decompress() was not called in the first place. Secondly, it allways returen a [None,None] list, so that is why the bound form stayed empty.
I needed to implement my own (within my accessIPWidget class):
def value_from_datadict(self, data, files, name):
try:
return data.get(name,None).rsplit()
except AttributeError:
return [widget.value_from_datadict(data, files, name + '_%s' % i) for i, widget in enumerate(self.widgets)]
Now the last line is what the original method did. In order to get the data into the bound form, I needed to add data.get(name,None).rsplit().
As far as I understand, the original value_from_datadict method only works for unbound fields. Because it changes the name of the original field to name + '_%s', which is what you get when pressing the submit button. In order to fill in a bound method, the datadict needs to be queried for 'name' only.
Hm, not shure if there is a way around this, but it seems to me that this behaviour should at least be documented somewhere.
Maybe I misunderstood something?