I have a model:
class MyModel(models.Model):
fieldA = models.IntegerField()
fieldB = models.IntegerField()
fieldC = models.IntegerField()
Now, let's get a QuerySet, e.g.
qs = MyModel.objects.all()
I'd like to be able to change fieldB and fieldC of all instances in qs with the same value and save them as NEW records in my database. I need something similar to qs.update(fieldB=2, fieldC=3) but I don't want to override the original records in qs. Is there a Django-way to do so (i.e., something not involving a manually coded for loop)?
I'm not aware of a Django call that will do what you want in a single call, like update does.
This is the closest I've been able to come to. Assuming that the objects you wanted to operate on are in qs then:
MyModel.objects.bulk_create(
MyModel(**{pair[0]: pair[1] for pair in x.iteritems()
if pair[0] != MyModel._meta.pk.name})
for x in qs.values())
Notes:
qs.values() returns one dictionary per object in qs.
The {pair[0]...} expression is a dictionary comprehension that creates a new dictionary minus primary field defined on MyModel.
MyModel(**...) creates a new MyModel object with its primary key set to None. It effectively creates a copy.
bulk_create creates all the objects in one query. It has caveats that one should be aware of.
If it needs be mentioned, what you have here are dict and list comprehensions, not for loops.
Related
I have a model that has arbitrary key/value pairs (attributes) associated with it. I'd like to have the option of sorting by those dynamic attributes. Here's what I came up with:
class Item(models.Model):
pass
class Attribute(models.Model):
item = models.ForeignKey(Item, related_name='attributes')
key = models.CharField()
value = models.CharField()
def get_sorted_items():
return Item.objects.all().annotate(
first=models.select_attribute('first'),
second=models.select_attribute('second'),
).order_by('first', 'second')
def select_attribute(attribute):
return expressions.RawSQL("""
select app_attribute.value from app_attribute
where app_attribute.item_id = app_item.id
and app_attribute.key = %s""", (attribute,))
This works, but it has a bit of raw SQL in it, so it makes my co-workers wary. Is it possible to do this without raw SQL? Can I make use of Django's ORM to simplify this?
I would expect something like this to work, but it doesn't:
def get_sorted_items():
return Item.objects.all().annotate(
first=Attribute.objects.filter(key='first').values('value'),
second=Attribute.objects.filter(key='second').values('value'),
).order_by('first', 'second')
Approach 1
Using Djagno 1.8+ Conditional Expressions
(see also Query Expressions)
items = Item.objects.all().annotate(
first=models.Case(models.When(attribute__key='first', then=models.F('attribute__value')), default=models.Value('')),
second=models.Case(models.When(attribute__key='second', then=models.F('attribute__value')), default=models.Value(''))
).distinct()
for item in items:
print item.first, item.second
Approach 2
Using prefetch_related with custom models.Prefetch object
keys = ['first', 'second']
items = Item.objects.all().prefetch_related(
models.Prefetch('attributes',
queryset=Attribute.objects.filter(key__in=keys),
to_attr='prefetched_attrs'),
)
This way every item from the queryset will contain a list under the .prefetched_attrs attribute.
This list will contains all filtered-item-related attributes.
Now, because you want to get the attribute.value, you can implement something like this:
class Item(models.Model):
#...
def get_attribute(self, key, default=None):
try:
return next((attr.value for attr in self.prefetched_attrs if attr.key == key), default)
except AttributeError:
raise AttributeError('You didnt prefetch any attributes')
#and the usage will be:
for item in items:
print item.get_attribute('first'), item.get_attribute('second')
Some notes about the differences in using both approaches.
you have a one idea better control over the filtering process using the approach with the custom Prefetch object. The conditional-expressions approach is one idea harder to be optimized IMHO.
with prefetch_related you get the whole attribute object, not just the value you are interested in.
Django executes prefetch_related after the queryset is being evaluated, which means a second query is being executed for each clause in the prefetch_related call. On one way this can be good, because it this keeps the main queryset untouched from the filters and thus not additional clauses like .distinct() are needed.
prefetch_related always put the returned objects into a list, its not very convenient to use when you have prefetchs returning 1 element per object. So additional model methods are required in order to use with pleasure.
I have a following model:
class UserProfile(models.Model):
user = models.OneToOneField(User)
location = models.PointField(blank=True, null=True, srid=CONSTANTS.SRID)
objects = models.GeoManager()
class Item(models.Model):
owner = models.ForeignKey(UserProfile)
objects = models.GeoManager()
Now I need to sort the Items by distance to some point:
p = Point(12.5807203, 50.1250706)
Item.objects.all().distance(p, field='owner__location')
But that throws me an error:
TypeError: ST_Distance output only available on GeometryFields.
From GeoDjango GeoQuerySet.distance() results in 'ST_Distance output only available on GeometryFields' when specifying a reverse relationship in field_name I can see there is already ticket for this.
Now I don't like the solution proposed in that question since that way I would not get the distance and I would lose the distances.
So I was thinking that I could achieve this by making a custom sql query. I know that this:
UserProfile.objects.distance(p)
will produce something like this:
SELECT (ST_distance_sphere("core_userprofile"."location",ST_GeomFromEWKB('\x0101000020e6100000223fd12b5429294076583c5002104940'::bytea))) AS "distance", "core_userprofile"."id", "core_userprofile"."user_id", "core_userprofile"."verified", "core_userprofile"."avatar_custom", "core_userprofile"."city", "core_userprofile"."location", "core_userprofile"."bio" FROM "core_userprofile"
So my question is: is there some easy way how to manually construct such query that would sort items by distance?
Since the geometry you're measuring distance to is on UserProfile, it makes sense to query for UserProfile objects and then handle each Item object they own. (The distance is the same for all items owned by a profile.)
For example:
all_profiles = UserProfile.objects.all()
for profile in all_profiles.distance(p).order_by('distance'):
for item in profile.item_set.all():
process(item, profile.distance)
You may be able to make this more efficient with prefetch_related:
all_profiles = UserProfile.objects.all()
all_profiles = all_profiles.prefetch_related('item_set') # we'll need these
for profile in all_profiles.distance(p).order_by('distance'):
for item in profile.item_set.all(): # items already prefetched
process(item, profile.distance)
If it's important for some reason to query directly for Item objects, try using extra:
items = Item.objects.all()
items = items.select_related('owner')
distance_select = "st_distance_sphere(core_userprofile.location, ST_GeomFromEWKT('%s'))" % p.wkt
items = items.extra({'distance': distance_select})
items = items.order_by('distance')
Raw queries are another option, which let you get model objects from a raw SQL query:
items = Item.objects.raw("SELECT core_item.* FROM core_item JOIN core_userprofile ...")
Using Django's ORM, I am trying to find instances of myModel based on two of its datetime variables; specifically, where the months of these two datetimes are not equal. I understand to filter by the value of a modelfield, you can use Django's F( ) expressions, so I thought I'd try something like this:
myModel.objects.filter(fixed_date__month=F('closed_date__month'))
I know this wouldn't find instances where they aren't equal, but I thought it'd be a good first step since I've never used the F expressions before. However, it doesn't work as I thought it should. I expected it to give me a queryset of objects where the value of the fixed_date month was equal to the value of the closed_date month, but instead I get an error:
FieldError: Join on field 'closed_date' not permitted. Did you misspell 'month' for the lookup type?
I'm not sure if what I'm trying to do isn't possible or straightforward with the ORM, or if I'm just making a simple mistake.
It doesn't look like django F objects currently support extracting the month inside a DateTimeField, the error message seems to be stating that the F object is trying to convert the '__' inside the string 'closed_date__month' as a Foreignkey between different objects, which are usually stored as joins inside an sql database.
You could carry out the same query by iterating across the objects:
result = []
for obj in myModel.objects.all():
if obj.fixed_date.month != obj.closed_date.month:
result.append(obj)
or as a list comprehension:
result = [obj for obj in myModel.objects.all() if obj.fixed_date.month != obj.closed_date.month]
Alternatively, if this is not efficient enough, the months for the two dates could be cached as IntegerFields within the model, something like:
class myModel(models.Model):
....other fields....
fixed_date = models.DateTimeField()
closed_date = models.DateTimeField()
fixed_month = models.IntegerField()
closed_month = models.IntegerField()
store the two integers when the relevant dates are updated:
myModel.fixed_month = myModel.fixed_date.month
myModel.save()
Then use an F object to compare the two integer fields:
myModel.objects.filter(fixed_month__ne=F('closed_month'))
The ne modifier will do the not equal test.
Edit - using raw sql
If you are using an sql based database, then most efficient method is to use the .raw() method to manually specify the sql:
myModel.objects.raw('SELECT * FROM stuff_mymodel WHERE MONTH(fixed_date) != MONTH(close_date)')
Where 'stuff_mymodel' is the correct name of the table in the database. This uses the SQL MONTH() function to extract the values from the month fields, and compare their values. It will return a collection of objects.
There is some nay-saying about the django query system, for example: http://charlesleifer.com/blog/shortcomings-in-the-django-orm-and-a-look-at-peewee-a-lightweight-alternative/. This example could be taken as demonstrating another inconsistency in it's query api.
My thinking is this:
class myModel(models.Model):
fixed_date = models.DateTimeField()
closed_date = models.DateTimeField()
def has_diff_months(self):
if self.fixed_date.month != self.closed_date.month:
return True
return False
Then:
[x for x in myModel.objects.all() if x.has_diff_months()]
However, for a truly efficient solution you'd have to use another column. It makes sense to me that it'd be a computed boolean field that is created when you save, like so:
class myModel(models.Model):
fixed_date = models.DateTimeField()
closed_date = models.DateTimeField()
diff_months = models.BooleanField()
#overriding save method
def save(self, *args, **kwargs):
#calculating the value for diff_months
self.diff_months = (self.fixed_date.month != self.closed_date.month)
#aaand... saving:
super(Blog, self).save(*args, **kwargs)
Then filtering would simply be:
myModel.objects.filter(diff_months=True)
I want the user to be able to order a list of objects in a table using javascript. Then, in a django function I would like to sort those object based on the same ordering, not on an attribute.
Is it possible? I was thinking about passing a list of pk from the template to the view and then ordering the objects according to this list, but I have not found a way to do it yet.
I don't think this is possible with queryset. Try following:
pk_list = [2, 1, 3, 4]
pk2obj = {obj.pk: obj for obj in Model.objects.all()}
objects_ordered = [pk2obj[pk] for pk in pk_list]
pkg2obj is mapping between pk and model instance object. To make a dictionary I used dictionary comprehension.
If you want to omit deleted objects:
objects_ordered = [pk2obj[pk] for pk in pk_list if pk in pk2obj]
Else if you want to replace deleted objects with default value (None in following code):
objects_ordered = [pk2obj.get(pk, None) for pk in pk_list]
I've had to solve this exact problem before.
If you want the user to be able to reorder them into a user-defined order, you can easily define a field to store this order.
As you say, initially, you could serve them in order according to id or an upload_date DateTimeField. But you could also have an PositiveIntegerField in the model, named position or order, to represent the user-defined order.
class MediaItem(models.Model):
user = models.ForeignKey(User)
upload_date = models.DateTimeField(auto_now_add = True)
position = models.PositiveIntegerField()
Whenever a user changes the order on the frontend, the JS can send the new order as an array of objects (ie. new_order = [{"pk":3, "position":1}, {"pk":1, "position":2}, {"pk":2, "position":3}]). The view can look up each instance by pk, and change the position:
for obj in new_order:
media_item = MediaItem.objects.get(pk=obj['pk'])
media_item.position = obj['position']
media_item.save()
Then always query using
objects_ordered.objects.order_by('position')
That's how we managed to do it. If you have more specific questions regarding this approach, feel free to ask in the comments.
Edit:
If the same object can be a member of many different groups or lists, and you want to store the position of the membership within that list, you can achieve this using a through model. A through model is useful when you need to store data that relates to the relationship between two objects that are related. In addition to the MediaItem class shown above, this is what your other models would look like:
class Album(models.Model):
media_items = models.ManyToManyField(MediaItem,
related_name = 'album_media_items',
through = 'Membership')
class Membership(models.Model):
album = models.ForeignKey(Album,
related_name = 'album')
media_item = models.ForeignKey(MediaItem,
related_name = 'media_item')
date = models.DateTimeField(auto_now_add = True)
position = models.PositiveIntegerField()
Then, you could query the Membership instances, instead of the MediaItem instances.
# get id of list, or album...
alb = Album.objects.get(pk=id_of_album)
media_items = Membership.objects.filter(album=alb).order_by('position')
for item in media_items:
# return the items, or do whatever...
# keep in mind they are now in the user-defined order
You can do this:
pk_list = [1,5,3,9]
foo = Foo.objects.filter(id__in=pk_list)
#Order your QuerySet in base of your pk_list using Lambda
order_foo = sorted(foo, key = lambda:x , pk_list.index(x.pk))
I'm optimising some queries to return only the information we need using values_list rather than loading entire objects from the database.
For each object, I need to access the set of its children for some information.
My code is structured similar to this:
class MyObj(models.Model):
field_a = models.Integer()
class Child(models.Model):
parent = models.ForeignKey(MyObj, related_name='children')
field_b = models.Integer()
For each MyObj, I'd like to access the values of field_b for all its children as efficiently as possible.
I expected that a query such as models.MyObj.values_list('children__field_b') would return a list of the values I wanted, but it only returns a single value.
Is it possible to get a list of values for a field like I've described using values_list?
If not, what would be the most performant way of iterating over each MyObj and its children?
foo = MyObject.objects.get(pk=1)
foo.child_set.values_list('field_b')