DRY in django queries, reverse queries, and predicates - django

I am frustrated that in Django I often end up having to write methods on a custom Manager:
class EntryManager(Manager):
def filter_beatle(self, beatle):
return self.filter(headline__contains=beatle)
... and repeat pretty much the same method in a different Manager for a reverse query:
class BlogManager(Manager):
def filter_beatle(self, beatle):
return self.filter(entry__headline__contains=beatle)
... and a predicate on Entry:
def headline_contains(self, beatle):
return self.headline.find(beatle) != -1
[Note that the predicate on Entry will work on Entry objects that haven't even been saved yet.]
This feels like a violation of DRY. Is there some way to express this once and use it in all three places?
What I would like to be able to do is write something like:
q = Q(headline__contains="Lennon")
lennon_entries = Entry.objects.filter(q)
lennon_blogs = Blog.objects.filter(q.reverse(Entry))
is_lennon = entry.would_filter(q)
... where 'headline__contains="Lennon"' expresses exactly once what it means to be 'an Entry about "Lennon"', and this can be used to construct reverse queries and a predicate.

The best place for this is a custom manager. According to django's guidelines a manager class is the best place for code that is affecting more than one object of a class.
class EntryManager(models.Manager):
def filter_lennons(self):
return self.get_query_set().filter(headline__contains='Lennon')
class Entry(models.Model):
headline = models.CharField(max_length=100)
objects = EntryManager()
lennons = Entry.objects.filter_lennons()

You should never rarely have to do the following:
if entry.headline.find('Lennon') >= 0:
because the filter should take care of restricting the result set to the instances you're interested in.
If you're going to be using the same filter multiple times, you can create a custom manager or a simple class method.
class Entry(models.Model):
...
# this really should be on a custom manager, but this was quicker to demonstrate
#classmethod
def find_headlines(cls, text):
return cls.objects.filter(headline__contains=text)
entries = Entry.find_headlines('Lennon')
But really, the DRYness has already been contained within the Queryset API. How often are you really going to be hard coding the string 'Lennon' into a query? Usually, the search parameter will be passed into a view from a GET or POST. Perfectly DRY.
So, what is the actual problem? Other than exploring the queryset API, have you ever had to hard code lookup values in multiple queries like your question?

For the "reverse filter" case you can use a subquery:
Blog.objects.filter(entries__in=Entry.objects.filter_beatle("Lennon"))
Reusing or generating predicates is not possible (in general) as there are predicates that cannot be expressed as queries and queries that cannot be expressed as predicates without db access.

My most common use for the predicate seems to be in asserts. Often something like:
class Thing(Model):
class QuerySet(query.QuerySet):
def need_to_be_whacked():
# ... code ...
def needs_to_be_whacked(self):
return Thing.objects.need_to_be_whacked().filter(id=self.id).exists()
def whack(self):
assert self.needs_to_be_whacked()
for thing in Thing.objects.need_to_be_whacked():
thing.whack()
I want to make sure that no other code is calling whack() in state where it doesn't need to be whacked. It costs a database hit, but it works.

Related

Django Rest Framework serializers.DictField constraint on possible keys

I have a REST API output that looks like this:
{
"something": "xyz"
"something_dict" {
"a": 10,
"b": 20,
"c": 30
}
}
I'm trying to create a serializer for that (mostly for automatic documentation generation purposes, but I believe that doesn't really matter), and for the something_dict field I thought it could be a good idea to use serializers.DictField.
The problem is, the keys that can show up in the dictionary are limited to a certain set (values of an enum type), but I can't seem to find a good way to define it.
Something like serializers.DictField(key=serializers.ChoiceField(choices=[enum.value for enum in Enum])) would be great but that's obviously not available.
The only reasonable way to specify this is to use a nested serializer:
class SomethingDict(serializers.Serializer):
a = serializers.IntField(required=False)
b = serializers.IntField(required=False)
c = serializers.IntField(required=False)
But that's slightly inconvenient as the set of available keys is potentially dynamic (values of an enum, as mentioned previously).
The question is then:
Is it possible to specify possible values for serializer.DictField keys or is there any way around it to make it dynamic? (I'd like to avoid defining the class in dynamic way, if possible)
Django Rest Framework provides a DictField serializer field that can be used to handle dictionaries in your API.
You can add a constraint on the possible keys of the dictionary by subclassing DictField and overriding the to_internal_value() method to perform the necessary validation.
Have a look at this example,
from rest_framework import serializers
class ConstrainedDictField(serializers.DictField):
def to_internal_value(self, data):
# Perform validation on the keys of the dictionary here
if set(data.keys()) != {"key1", "key2"}:
raise serializers.ValidationError("Invalid keys for dictionary.")
return super().to_internal_value(data)
You can then use the ConstrainedDictField in your serializer like this:
class MySerializer(serializers.Serializer):
my_dict = ConstrainedDictField()
It will only accept dictionary with key key1 and key2.

how to get child instance of a foreignkeyfield in a query

i have three classes. Place is foreignkey of Activity(where the activity takeplace), Park and Restarant are inherited from Place.
class Activity(Model):
place=ForeignKeyField('Place')
....
class Place(Model):
address=CharField(max_length=200)
....
class Park(Place):
ticket_price=Decimal()
....
class Restarant(Place):
manager_name=CharField(max_length=100)
how can i get the real type of 'Place' in a query on Activity,so i can get the additional attributes of child type.
activity_list= Activity.objects.all()
all the activity_list[n].place are 'Place' ,neither Park or Restarant, so i can't get 'ticket_price' or 'manager_name' directly.
i have checked the model-utils, and can't find a easy way to do this.
any help will be appreciated.
inspect.getmro(cls)
Return a tuple of class cls’s base classes, including cls, in method resolution order. No class appears more than once in this tuple. Note that the method resolution order depends on cls’s type. Unless a very peculiar user-defined metatype is in use, cls will be the first element of the tuple.
try:
import inspect
for activity in activity_list:
print inspect.getmro(activity.place)
or try :
for activity in activity_list:
print activity.place._type()
try:
activity_list= Activity.objects.all()
for act in activity_list:
for b in act.place.__class__.__bases__:
print b.__name__
I use Django Inheritance Managers for this, from django-model-utils. It allows you to do exactly what you want...you said you checked model-utils to do this, but it didn't work. It should work -- what did you try with django-model-utils? You need to add this line to the parent model of Place:
objects = InheritanceManager()
I haven't tried this with your exact model design, but have you tried something like:
places = Activity.objects.all().values_list('place_id', flat=True)
places_of_right_type = Place.objects.filter(pk__in=places).select_subclasses()
?
------ UPDATE --------
Not sure this will work, but throwing it out there as an idea...would only work for a get(), not all():
places = Activity.objects.get(pk=#).place.objects.select_subclasses()
The docs say that you can call select_subclasses() on the InheritanceManager itself, which is stored in .objects.

Django: How to use django.forms.ModelChoiceField with a Raw SQL query?

I'm trying to render a form with a combo that shows related entities. Therefore I'm using a ModelChoiceField.
This approach works well, until I needed to limit which entities to show. If I use a simple query expression it also works well, but things break if I use a raw SQL query.
So my code that works, sets the queryset to a filter expression.
class ReservationForm(forms.Form):
location_time_slot = ModelChoiceField(queryset=LocationTimeSlot.objects.all(), empty_label="Select your prefered time")
def __init__(self,*args,**kwargs):
city_id = kwargs.pop("city_id") # client is the parameter passed from views.py
super(ReservationForm, self).__init__(*args,**kwargs)
# TODO: move this to a manager
self.fields['location_time_slot'].queryset = LocationTimeSlot.objects.filter(city__id = city_id )
BUT, if I change that to a raw query I start having problems. Code that does not work:
class ReservationForm(forms.Form):
location_time_slot = ModelChoiceField(queryset=LocationTimeSlot.objects.all(), empty_label="Select your prefered time")
def __init__(self,*args,**kwargs):
city_id = kwargs.pop("city_id") # client is the parameter passed from views.py
super(ReservationForm, self).__init__(*args,**kwargs)
# TODO: move this to a manager
query = """SELECT ts.id, ts.datetime_to, ts.datetime_from, ts.available_reserves, l.name, l.'order'
FROM reservations_locationtimeslot AS ts
INNER JOIN reservations_location AS l ON l.id = ts.location_id
WHERE l.city_id = %s
AND ts.available_reserves > 0
AND ts.datetime_from > datetime() """
time_slots = LocationTimeSlot.objects.raw(query, [city_id])
self.fields['location_time_slot'].queryset = time_slots
The first error I get when trying to render the widget is: 'RawQuerySet' object has no attribute 'all'
I could solve that one thanks to one of the commets in enter link description here, by doing:
time_slots.all = time_slots.__iter__ # Dummy fix to allow default form rendering with raw SQL
But now I'm getting something similar when posting the form:
'RawQuerySet' object has no attribute 'get'
Is there a proper way to prepare a RawQuerySet to be used by ModelChoiceField?
Thanks!
Are you sure you actually need a raw query there? Just looking at that query, I can't see any reason you can't just do it with filter(location__city=city_id, available_reserves__gte=0, datetime_from__gt=datetime.datetime.now()).
Raw query sets are missing a number of methods that are defined on conventional query sets, so just dropping them in place isn't likely to work without writing your own definitions for all those methods.
I temporarily fixed the problem adding the missing methods.
The way I'm currently using the ModelChoiceField I only needed to add the all() and get() methods, but in different scenarios you might need to add some other methods as well. Also this is not a perfect solution because:
1) Defining the get method this way migth produce incorrect results. I think the get() method is used to validate that the selected option is within the options returned by all(). The way I temporarily implemented it only validates that the id exists in the table.
2) I guess the get method is less performant specified this way.
If anyone can think of a better solution, please let me know.
So my temporary solution:
class LocationTimeSlotManager(models.Manager):
def availableSlots(self, city_id):
query = """SELECT ts.id, ts.datetime_to, ts.datetime_from, ts.available_reserves, l.name, l.'order'
FROM reservations_locationtimeslot AS ts
.....
.....
MORE SQL """
time_slots = LocationTimeSlot.objects.raw(query, [city_id])
# Dummy fix to allow default form rendering with raw SQL
time_slots.all = time_slots.__iter__
time_slots.get = LocationTimeSlot.objects.get
return time_slots

How to test Models in Django with Foreign Keys

I want to make sure I am testing Models/Objects in isolation and not as one huge system.
If I have an Order object and it has Foreign Keys to Customers, Payments, OrderItems, etc. and I want to test Order functionality, I need to create fixtures for all of that related data, or create it in code. I think what I really need to be doing is mocking out the other items, but I don't see an easy (or possible) solution for that if I am doing queries on these Foreign Keys.
The common solutions (fixtures) don't really let me test one Object at a time. I am sure this is partly caused by my app being way over coupled.
I am trying my darndest to adopt TDD as my main method of working, but the way things work with Django, it seems you can either run very trivial unit tests, or these massive integration tests.
[Edit] Better explicit example and a some more humility
What I mean is that I seem to be able to only run trivial unit tests. I have seen people with very well tested and granular modules. I am certain some of this can be followed back to poor design.
Example:
I have a model call Upsell which is linked to a Product model. Then I have a Choices model which are children of Upsell (do you want what's behind door #1, #2, #3).
The Upsell model has several methods on it that derive items necessary to render the template from their choices. The most important one is that it creates a URL for each choice. It does this through some string mangling etc. If I wanted to test the Upsell.get_urls() method, I want to have it not depend on the values of choices in the fixtures, and I want to not have it depend on the value of Product in the fixtures.
Right now I populate the db in the setUp method for the tests, and that works well with the way Django backs out the transaction every time, but only outside of setUp and tearDown. This works fairly well except some of the Models are fairly complex to set up, while I actually only need to get one attribute for it.
I can't give you an example of that, since I can't accomplish it, but here is the type of thing I am doing now. Basically I input an entire order, create the A/B experiment it was attached to, etc. And that's not counting Product, Categories, etc. all set up by fixtures. It's not the extra work I am concerned as I can't even test one database-based object at a time. The Tests below are important, but they are integration tests. I would like to be building up to something like this by testing each item separately. As you pointed out, maybe I shouldn't have chosen a framework so closely tied to the db. Does any sort of dependency injection exist with something like this? (beyond my testing, but the code itself as well)
class TestMultiSinglePaySwap(TestCase):
fixtures = ['/srv/asm/fixtures/alchemysites.json','/srv/asm/fixtures/catalog.json','/srv/asm/fixtures/checkout_smallset.json','/srv/asm/fixtures/order-test-fixture.json','/srv/asm/fixtures/offers.json']
def setUp(self):
self.o = Order()
self.sp = SiteProfile.objects.get(pk=1)
self.c = Customer.objects.get(pk=1)
signals.post_save.disconnect(order_email_first, sender=Order)
self.o.customer = self.c
p = Payment()
p.cc_number = '4444000011110000'
p.cc_exp_month = '12'
p.cc_type = 'V'
p.cc_exp_year = '2020'
p.cvv2 = '123'
p.save()
self.o.payment = p
self.o.site_profile = self.sp
self.o.save()
self.initial_items = []
self.main_kit = Product.objects.get(pk='MOA1000D6')
self.initial_items.append(self.main_kit)
self.o.add_item('MOA1000D6', 1, False)
self.item1 = Product.objects.get(pk='MOA1041A-6')
self.initial_items.append(self.item1)
self.o.add_item('MOA1041A-6', 1, False)
self.item2 = Product.objects.get(pk='MOA1015-6B')
self.initial_items.append(self.item2)
self.o.add_item('MOA1015-6B', 1, False)
self.item3 = Product.objects.get(pk='STP1001-6E')
self.initial_items.append(self.item3)
self.o.add_item('STP1001-6E', 1, False)
self.swap_item1 = Product.objects.get(pk='MOA1041A-1')
def test_single_pay_swap_wholeorder(self):
o = self.o
swap_all_skus(o)
post_swap_order = Order.objects.get(pk = o.id)
swapped_skus = ['MOA1000D','MOA1041A-1','MOA1015-1B','STP1001-1E']
order_items = post_swap_order.get_all_line_items()
self.assertEqual(order_items.count(), 4)
pr1 = Product()
pr1.sku = 'MOA1000D'
item = OrderItem.objects.get(order = o, sku = 'MOA1000D')
self.assertTrue(item.sku.sku == 'MOA1000D')
pr2 = Product()
pr2.sku = 'MOA1015-1B'
item = OrderItem.objects.get(order = o, sku = 'MOA1015-1B')
self.assertTrue(item.sku.sku == 'MOA1015-1B')
pr1 = Product()
pr1.sku = 'MOA1041A-1'
item = OrderItem.objects.get(order = o, sku = 'MOA1041A-1')
self.assertTrue(item.sku.sku == 'MOA1041A-1')
pr1 = Product()
pr1.sku = 'STP1001-1E'
item = OrderItem.objects.get(order = o, sku = 'STP1001-1E')
self.assertTrue(item.sku.sku == 'STP1001-1E')
Note that I have never actually used a Mock framework though I have tried. So I may also just be fundamentally missing something here.
Look into model mommy. It can automagically create objects with Foreign Keys.
This will probably not answer your question but it may give you food for thought.
In my opinion when you are testing a database backed project or application there is a limit to what you can mock. This is especially so when you are using a framework and an ORM such as the one Django offers. In Django there is no distinction between the business model class and the persistence model class. If you want such a distinction then you'll have to add it yourself.
Unless you are willing to add that additional layer of complexity yourself it becomes tricky to test the business objects alone without having to add fixtures etc. If you must do so you will have to tackle some of the auto magic vodoo done by Django.
If you do choose to grit your teeth and dig in then Michael Foord's Python Mock library will come in quite handy.
I am trying my darndest to adopt TDD as my main method of working, but the way things work with Django, it seems you can either run very trivial unit tests, or these massive integration tests.
I have used Django unit testing mechanism to write non-trivial unit tests. My requirements were doubtless very different from yours. If you can provide more specific details about what you are trying to accomplish then users here would be able to suggest other alternatives.

Django's list_details views saving queryset to memory (not updating)?

I have a custom model manager that looks like this:
class MyManager(models.Manager)
def get_query_set(self):
'''Only get items that are 'approved' and have a `pub_date` that is in
the past. Ignore the rest.'''
queryset = super(MyManager, self).get_query_set()
queryset = queryset.filter(status__in=('a',))
return queryset.filter(pub_date__lte=datetime.utcnow())
And this works well enough; however, I have a problem using Django's generic.list_detail views object_detail and object_list: the queryset seems to be only loading once and, because of this, it isn't fetching the items it should be because, I assume, the utcnow() time has been called only once (when it first loaded).
I assume this is intentional and meant as a performance boost - however, it means that video's show up elsewhere on the site (in places I am not in a object_detail view) before they are available in an object_detail view (see urls.py below). This is leading to 404s ...
Any ideas ? Or do I have to write my own custom views to avoid this ?
Thanks!
urls.py
url(r'^video/(?P<object_id>\d+)$',
list_detail.object_detail,
{ 'queryset': Video.objects.all(), },
name='video_detail',
),
It is not a problem of cache: as you do it now, the queryset definition is evaluated once, while parsing urls, and then, it is never evaluated again.
Solution is actually pretty simple and described in the online documentation: Complex filtering with wrapper functions: just create a small custom view, that will simply call the generic view.
I am actually using a similar solution quite a lot and I feel it quite comfortable.
By the way, a small side note, for this case I would suggest not using a custom manager, and go back instead on a normal filtering.
Try correcting urls.py to:
url(r'^video/(?P<object_id>\d+)$',
list_detail.object_detail,
{ 'queryset': Video.objects.all, }, # here's the difference
name='video_detail',
)
Edit:
If this fail, try apply similar technique(passing callable instead of calling it) to filter():
return queryset.filter(pub_date__lte=datetime.utcnow)
I have an almost identical model Manager to thornomad, and the same problem with generic views.
I have to point out that neither of the suggestions above work:
doing Video.objects.all without parentheses gives an error
doing queryset.filter(pub_date__lte=datetime.utcnow), again without the parentheses, does not give an error but does not fix the problem
I have also tried another way, which is to use a lambda to return the queryset, eg:
qs = lambda *x: Video.objects.all()
url(r'^video/(?P<object_id>\d+)$',
list_detail.object_detail,
{ 'queryset': qs(), },
name='video_detail',
),
...it didn't work either and I can see now I must have been desperate to think it would :)
lazy_qs = lambda *x: lazy(Post.live_objects.all, QuerySet)
blog_posts = {
'queryset': lazy_qs(),
...doesn't work either (gives an error) as utils.functional.lazy doesn't know how to convert the result to a QuerySet properly, as best I can tell.
I think Roberto's answer of wrapping the generic view is the only one that will help.
The django docs should be amended to point out the limitations of the queryset used by generic views (currently the docs have a special note to tell you everything will be okay!)