Invalidating Memcached Keys on save() in Django - django

I've got a view in Django that uses memcached to cache data for the more highly trafficked views that rely on a relatively static set of data. The key word is relatively: I need invalidate the memcached key for that particular URL's data when it's changed in the database. To be as clear as possible, here's the meat an' potatoes of the view (Person is a model, cache is django.core.cache.cache):
def person_detail(request, slug):
if request.is_ajax():
cache_key = "%s_ABOUT_%s" % settings.SITE_PREFIX, slug
# Check the cache to see if we've already got this result made.
json_dict = cache.get(cache_key)
# Was it a cache hit?
if json_dict is None:
# That's a negative Ghost Rider
person = get_object_or_404(Person, display = True, slug = slug)
json_dict = {
'name' : person.name,
'bio' : person.bio_html,
'image' : person.image.extra_thumbnails['large'].absolute_url,
}
cache.set(cache_key)
# json_dict will now exist, whether it's from the cache or not
response = HttpResponse()
response['Content-Type'] = 'text/javascript'
response.write(simpljson.dumps(json_dict)) # Make sure it's all properly formatted for JS by using simplejson
return response
else:
# This is where the fully templated response is generated
What I want to do is get at that cache_key variable in it's "unformatted" form, but I'm not sure how to do this--if it can be done at all.
Just in case there's already something to do this, here's what I want to do with it (this is from the Person model's hypothetical save method)
def save(self):
# If this is an update, the key will be cached, otherwise it won't, let's see if we can't find me
try:
old_self = Person.objects.get(pk=self.id)
cache_key = # Voodoo magic to get that variable
old_key = cache_key.format(settings.SITE_PREFIX, old_self.slug) # Generate the key currently cached
cache.delete(old_key) # Hit it with both barrels of rock salt
# Turns out this doesn't already exist, let's make that first request even faster by making this cache right now
except DoesNotExist:
# I haven't gotten to this yet.
super(Person, self).save()
I'm thinking about making a view class for this sorta stuff, and having functions in it like remove_cache or generate_cache since I do this sorta stuff a lot. Would that be a better idea? If so, how would I call the views in the URLconf if they're in a class?

URLConf should point to any callable. There's no strict requirement to make it point to function exactly. You could implement base class with your cache methods then extend it:
class RealView(BaseViewWithCacheMethods):
def __call__(self, request):
if request.is_ajax():
return self.ajax_view()
return self.html_view()
URLConf definition would be something like that:
from django.conf.urls.defaults import *
from views import RealView
urlpattrens = patterns('',
(r'^$', RealView()),
)

Related

mongoengine know when to delete document

New to django. I'm doing my best to implement CRUD using Django, mongodb, and mongoengine. I'm able to query the database and render my page with the correct information from the database. I'm also able to change some document fields using javascript and do an Ajax POST back to the original Django View class with the correct csrf token.
The data payload I'm sending back and forth is a list of each Document Model (VirtualPageModel) serialized to json (each element contains ObjectId string along with the other specific fields from the Model.)
This is where it starts getting murky. In order to update the original document in my View Class post function I do an additional query using the object id and loop through the dictionary items, setting the respective fields each time. I then call save and any new data is pushed to the Mongo collection correctly.
I'm not sure if what I'm doing to update existing documents is correct or in the spirit of django's abstracted database operations. The deeper I get the more I feel like I'm not using some fundamental facility earlier on (provided by either django or mongoengine) and because of this I'm having to make things up further downstream.
The way my code is now I would not be able to create a new document (although that's easy enough to fix). However what I'm really curious about is how I would know when to delete a document which existed in the initial query, but was removed by the user/javascript code? Am I overthinking things and the contents of my POST should contain a list of ObjectIds to delete (sounds like a security risk although this would be an internal tool.)
I was assuming that my View Class might maintain either the original document objects (or simply ObjectIds) it queried and I could do my comparisions off of that set, but I can't seem to get that information to persist (as a class variable in VolumeSplitterView) from its inception to when I received the POST at the end.
I would appreciate if anyone could take a look at my code. It really seems like the "ease of use" facilities of Django start to break when paired with Mongo and/or a sufficiently complex Model schema which needs to be directly available to javascript as opposed to simple Forms.
I was going to use this dev work to become django battle-hardened in order to tackle a future app which will be much more complicated and important. I can hack on this thing all day and make it functional, but what I'm really interested in is anyone's experience in using Django + MongoDB + MongoEngine to implement CRUD on a Database Schema which is not vary Form-centric (think more nested metadata).
Thanks.
model.py: uses mongoengine Field types.
class MongoEncoder(JSONEncoder):
def default(self, o):
if isinstance(o, VirtualPageModel):
data_dict = (o.to_mongo()).to_dict()
if isinstance(data_dict.get('_id'), ObjectId):
data_dict.update({'_id': str(data_dict.get('_id'))})
return data_dict
else:
return JSONEncoder.default(self, o)
class SubTypeModel(EmbeddedDocument):
filename = StringField(max_length=200, required=True)
page_num = IntField(required=True)
class VirtualPageModel(Document):
volume = StringField(max_length=200, required=True)
start_physical_page_num = IntField()
physical_pages = ListField(EmbeddedDocumentField(SubTypeModel),
default=list)
error_msg = ListField(StringField(),
default=list)
def save(self, *args, **kwargs):
print('In save: {}'.format(kwargs))
for k, v in kwargs.items():
if k == 'physical_pages':
self.physical_pages = []
for a_page in v:
tmp_pp = SubTypeModel()
for p_k, p_v in a_page.items():
setattr(tmp_pp, p_k, p_v)
self.physical_pages.append(tmp_pp)
else:
setattr(self, k, v)
return super(VirtualPageModel, self).save(*args, **kwargs)
views.py: My attempt at a view
class VolumeSplitterView(View):
#initial = {'key': 'value'}
template_name = 'click_model/index.html'
vol = None
start = 0
end = 20
def get(self, request, *args, **kwargs):
self.vol = self.kwargs.get('vol', None)
records = self.get_records()
records = records[self.start:self.end]
vp_json_list = []
img_filepaths = []
for vp in records:
vp_json = json.dumps(vp, cls=MongoEncoder)
vp_json_list.append(vp_json)
for pp in vp.physical_pages:
filepath = get_file_path(vp, pp.filename)
img_filepaths.append(filepath)
data_dict = {
'img_filepaths': img_filepaths,
'vp_json_list': vp_json_list
}
return render_to_response(self.template_name,
{'data_dict': data_dict},
RequestContext(request))
def get_records(self):
return VirtualPageModel.objects(volume=self.vol)
def post(self, request, *args, **kwargs):
if request.is_ajax:
vp_dict_list = json.loads(request.POST.get('data', []))
for vp_dict in vp_dict_list:
o_id = vp_dict.pop('_id')
original_doc = VirtualPageModel.objects.get(id=o_id)
try:
original_doc.save(**vp_dict)
except Exception:
print(traceback.format_exc())

How can I access URL parameters from within a BasePermission?

I'm trying to write a custom rest_framework Permission to prevent users from querying information that's not of the same company as them. Unfortunately, I can't seem to access any of the URL's parameters from within has_permission() or has_object_permissions().
Here's the beginning of my router:
# Create a basic router
router = routers.SimpleRouter()
# Establish some variables to assist with nested routes
root_elem = 'companies'
root_elem_id = '/(?P<company_id>[0-9]+)'
loca_elem = '/locations'
loca_elem_id = '/(?P<location_id>[0-9]+)'
# Companies will be the root from which all other relations branch
router.register(r'' + root_elem, views.CompanyViewSet)
router.register(r'' + root_elem + root_elem_id + loca_elem,
views.LocationViewSet)
Here's my custom permission:
# Only permit actions originating from location managers or company admins
class IsLocationManagerOrHigher(BasePermission):
# Checked when displaying lists of records
def has_permission(self, request, *args, **kwargs):
is_correct_level = False
# Admins can see every location if their location_id
# matches a location that's a child of the company
# specified in the URL
if request.employee.is_admin:
is_correct_level = True
return request.user and is_correct_level
# Checked when viewing specific records
def has_object_permission(self, request, view, obj):
is_correct_level = False
# Admins can see location details if their location's company_id
# matches a Location's company_id
if request.employee.is_admin:
is_correct_level = True
# Managers can see location details if it's their location
elif obj.id == request.employee.location_id and request.employee.is_manager:
is_correct_level = True
return request.user and is_correct_level
Right now checking request.employee.is_admin is only half of what I need - I also need to access the company_id from the URL and make sure it matches the admin's location's company_id:
# Pseudocode
try:
user_location = Location.objects.get(id=request.employee.location_id)
return user_location.company_id == kwargs['company_id']
except ObjectDoesNotExist:
pass
I've yet to figure out how to pass these parameters into the Permission so that it can perform this extra step. Or perhaps there's a better way of accomplishing what I'm trying to do?
If you can't pass them in directly (which would be preferable), they are available on the request object:
company_id = request.resolver_match.kwargs.get('company_id')
request.resolver_match.args and request.resolver_match.kwargs contain the positional/keyword arguments captured in your url.
As an alternative to the correct response posted by knbk, you can also get the URL parameters using the view object passed to has_permission method. Like this:
company_id = view.kwargs.get('company_id')

Convert POST to PUT with Tastypie

Full Disclosure: Cross posted to Tastypie Google Group
I have a situation where I have limited control over what is being sent to my api. Essentially there are two webservices that I need to be able to accept POST data from. Both use plain POST actions with urlencoded data (basic form submission essentially).
Thinking about it in "curl" terms it's like:
curl --data "id=1&foo=2" http://path/to/api
My problem is that I can't update records using POST. So I need to adjust the model resource (I believe) such that if an ID is specified, the POST acts as a PUT instead of a POST.
api.py
class urlencodeSerializer(Serializer):
formats = ['json', 'jsonp', 'xml', 'yaml', 'html', 'plist', 'urlencoded']
content_types = {
'json': 'application/json',
'jsonp': 'text/javascript',
'xml': 'application/xml',
'yaml': 'text/yaml',
'html': 'text/html',
'plist': 'application/x-plist',
'urlencoded': 'application/x-www-form-urlencoded',
}
# cheating
def to_urlencoded(self,content):
pass
# this comes from an old patch on github, it was never implemented
def from_urlencoded(self, data,options=None):
""" handles basic formencoded url posts """
qs = dict((k, v if len(v)>1 else v[0] )
for k, v in urlparse.parse_qs(data).iteritems())
return qs
class FooResource(ModelResource):
class Meta:
queryset = Foo.objects.all() # "id" = models.AutoField(primary_key=True)
resource_name = 'foo'
authorization = Authorization() # only temporary, I know.
serializer = urlencodeSerializer()
urls.py
foo_resource = FooResource
...
url(r'^api/',include(foo_resource.urls)),
)
In #tastypie on Freenode, Ghost[], suggested that I overwrite post_list() by creating a function in the model resource like so, however, I have not been successful in using this as yet.
def post_list(self, request, **kwargs):
if request.POST.get('id'):
return self.put_detail(request,**kwargs)
else:
return super(YourResource, self).post_list(request,**kwargs)
Unfortunately this method isn't working for me. I'm hoping the larger community could provide some guidance or a solution for this problem.
Note: I cannot overwrite the headers that come from the client (as per: http://django-tastypie.readthedocs.org/en/latest/resources.html#using-put-delete-patch-in-unsupported-places)
I had a similar problem on user creation where I wasn't able to check if the record already existed. I ended up creating a custom validation method which validated if the user didn't exist in which case post would work fine. If the user did exist I updated the record from the validation method. The api still returns a 400 response but the record is updated. It feels a bit hacky but...
from tastypie.validation import Validation
class MyValidation(Validation):
def is_valid(self, bundle, request=None):
errors = {}
#if this dict is empty validation passes.
my_foo = foo.objects.filter(id=1)
if not len(my_foo) == 0: #if object exists
foo[0].foo = 'bar' #so existing object updated
errors['status'] = 'object updated' #this will be returned in the api response
return errors
#so errors is empty if object does not exist and validation passes. Otherwise object
#updated and response notifies you of this
class FooResource(ModelResource):
class Meta:
queryset = Foo.objects.all() # "id" = models.AutoField(primary_key=True)
validation = MyValidation()
With Cathal's recommendation I was able to utilize a validation function to update the records I needed. While this does not return a valid code... it works.
from tastypie.validation import Validation
import string # wrapping in int() doesn't work
class Validator(Validation):
def __init__(self,**kwargs):
pass
def is_valid(self,bundle,request=None):
if string.atoi(bundle.data['id']) in Foo.objects.values_list('id',flat=True):
# ... update code here
else:
return {}
Make sure you specify the validation = Validator() in the ModelResource meta.

Django: restrict access to a view, dependent upon referring url

I'm making a school records webapp. I want staff users to be able to view the user data pages for any pupil by going to the correct url, but without allowing pupils access to each others' pages. However I'm using the same view function for both urls.
I have a working #user_is_staff decorator based on the existence of a user.staff object. Pupil users have a user.pupil object instead. These are discrete, naturally, as no user can have both a .staff and a .pupil entry.
urls.py
(r'^home/(?P<subject>[^/]+)/$', 'myproject.myapp.views.display_pupil')
(r'^admin/user/(?P<user>\d+)/(+P<subject>[^/]+)/$', 'myproject.myapp.views.display_pupil')
views.py
#login_required
def display_pupil(request, subject, pupil=None):
if pupil:
try:
thepupil = get_object_or_404(Pupil, id = pupil, cohort__school = request.user.staff.school)
except Staff.DoesNotExist:
return HttpResponseForbidden()
else:
thepupil = request.user.pupil
thesubject = get_object_or_404(Subject, shortname = subject)
# do lots more stuff here
return render_to_response('pupilpage.html', locals(), context_instance=RequestContext(request))
Doing it this way works, but feels very hacky, particularly as my '#user_is_staff' decorator has a more elegant redirect to a login page than the 403 error here.
What I don't know is how to apply the #user_is_staff decorator to the function only when it has been accessed with the pupil kwarg. There's a lot more code in the real view function, so I don't want to write a second one as that would be severely non-DRY.
Sounds like you want two separate views - one for a specific pupil and one for the current user - and a utility function containing the shared logic.
#login_required:
def display_current_pupil(request, subject):
thepupil = request.user.pupil
return display_pupil_info(request, subject, thepupil)
#user_is_staff
def display_pupil(request, subject, pupil):
thepupil = get_object_or_404(Pupil, id=pupil, cohort__school=request.user.staff.school)
return display_pupil_info(request, subject, thepupil)
def display_pupil_info(request, subject, thepupil):
thesubject = get_object_or_404(Subject, shortname=subject)
# do lots more stuff here
return render_to_response('pupilpage.html', locals(), context_instance=RequestContext(request))

Django - Check users messages every request

I want to check if a user has any new messages each time they load the page. Up until now, I have been doing this inside of my views but it's getting fairly hard to maintain since I have a fair number of views now.
I assume this is the kind of thing middleware is good for, a check that will happen every single page load. What I need it to do is so:
Check if the user is logged in
If they are, check if they have any messages
Store the result so I can reference the information in my templates
Has anyone ever had to write any middleware like this? I've never used middleware before so any help would be greatly appreciated.
You could use middleware for this purpose, but perhaps context processors are more inline for what you want to do.
With middleware, you are attaching data to the request object. You could query the database and find a way to jam the messages into the request. But context processors allow you to make available extra entries into the context dictionary for use in your templates.
I think of middleware as providing extra information to your views, while context processors provide extra information to your templates. This is in no way a rule, but in the beginning it can help to think this way (I believe).
def messages_processor(request):
return { 'new_messages': Message.objects.filter(unread=True, user=request.user) }
Include that processor in your settings.py under context processors. Then simply reference new_messages in your templates.
I have written this middleware on my site for rendering messages. It checks a cookie, if it is not present it appends the message to request and saves a cookie, maybe you can do something similar:
class MyMiddleware:
def __init__(self):
#print 'Initialized my Middleware'
pass
def process_request(self, request):
user_id = False
if request.user.is_active:
user_id = str(request.user.id)
self.process_update_messages(request, user_id)
def process_response(self, request, response):
self.process_update_messages_response(request, response)
return response
def process_update_messages(self, request, user_id=False):
update_messages = UpdateMessage.objects.exclude(expired=True)
render_message = False
request.session['update_messages'] = []
for message in update_messages:
if message.expire_time < datetime.datetime.now():
message.expired = True
message.save()
else:
if request.COOKIES.get(message.cookie(), True) == True:
render_message = True
if render_message:
request.session['update_messages'].append({'cookie': message.cookie(), 'cookie_max_age': message.cookie_max_age})
messages.add_message(request, message.level, message)
break
def process_update_messages_response(self, request, response):
try:
update_messages = request.session['update_messages']
except:
update_messages = False
if update_messages:
for message in update_messages:
response.set_cookie(message['cookie'], value=False, max_age=message['cookie_max_age'], expires=None, path='/', domain=None, secure=None)
return response