How to make "manual" select_related imitation to avoid undesirable DB hits?
we have:
class Country:
name = CharField()
class City:
country = models.ForeignKey(Country)
name = models.CharField()
cities = City.objects.raw("select * from city inner join country on city.country_id = country.id where name = 'london'")
#this will hill hit DB
print cities[0].country.name
How to tell django that related models are already fetched.
A solution with prefetch_related (this means that two queries will be made, 1 for the cities and 1 for the countries) taken from django-users which is not part of the public API but is working on Django 1.7
from django.db.models.query import prefetch_related_objects
#raw querysets do not have len()
#thats why we need to evaluate them to list
cities = list(City.objects.raw("select * from city inner join country on city.country_id = country.id where name = 'london'"))
prefetch_related_objects(cities, ['country'])
UPDATE
Now in Django 1.10 prefetch_related_objects is part of the public API.
Not sure if you still need this, but I solved it starting with Alasdair's answer. You want to use the info from the query to build the model or it'll still fire additional queries when you try to access the foreign key field. So in your case, you'd want:
cities = list(City.objects.raw("""
SELECT
city.*, country.name as countryName
FROM
cities INNER JOIN country ON city.country_id = country.id
WHERE
city.name = 'LONDON"""))
for city in cities:
city.country = Country(name=city.countryName)
The line that assigns the country doesn't hit the database, it's just creating a model. Then after that, when you access city.country it won't fire another database query.
I'm not sure if you can do this. As an alternative, you can select individual fields from the country table and access them on each instance.
cities = City.objects.raw("select city.*, name as country_name from city inner join country on city.country_id = country.id where name = 'london'")
city = cities[0]
# this will not hit the database again
city.country_name
Related
I'm trying to get all values in current table, and also get some fields in related tables.
class school(models.Model):
school_name = models.CharField(max_length=256)
school_type = models.CharField(max_length=128)
school_address = models.CharField(max_length=256)
class hometown(models.Model):
hometown_name = models.CharField(max_length=32)
class person(models.Model):
person_name = models.CharField(max_length=128)
person_id = models.CharField(max_length=128)
person_school = models.ForeignKey(school, on_delete=models.CASCADE)
person_ht = models.ForeignKey(hometown, on_delete=models.CASCADE)
how to quick select all info i needed into a dict for rendering.
there will be many records in person, i got school_id input, and want to get all person in this school, and also want these person's hometown_name shown.
i tried like this, can get the info i wanted. And any other quick way to do it?
m=person.objects.filter(person_school_id=1)
.values('id', 'person_name', 'person_id',
school_name=F('person_school__school_name'),
school_address=F('person_school__school_address'),
hometown_name=F('person_ht__hometown_name'))
person_name, person_id, school_name, school_address, hometown_name
if the person have many fields, it will be a hard work for list all values.
what i mean, is there any queryset can join related tables' fields together, which no need to list fields in values.
Maybe like this:
m=person.objects.filter(person_school_id=1).XXXX.values()
it can show all values in school, and all values in hometown together with person's values in m, and i can
for x in m:
print(x.school_name, x.hometown_name, x.person_name)
You add a prefetch_related query on top of your queryset.
prefetch_data = Prefetch('person_set, hometown_set, school_set', queryset=m)
Where prefetch_data will prepare your DB to fetch related tables and m is your original filtered query (so add this below your Person.objects.filter(... )
Then you do the actual query to the DB:
query = query.prefetch_related(prefetch_data)
Where query will be the actual resulting query with a list of Person objects (so add that line below the prefetch_data one).
Example:
m=person.objects.filter(person_school_id=1)
.values('id', 'person_name', 'person_id',
school_name=F('person_school__school_name'),
school_address=F('person_school__school_address'),
hometown_name=F('person_ht__hometown_name'))
prefetch_data = Prefetch('person_set, hometown_set, school_set', queryset=m)
query = query.prefetch_related(prefetch_data)
In that example I've broken down the queries into more manageable pieces, but you can do the whole thing in one big line too (less manageable to read though):
m=person.objects.filter(person_school_id=1)
.values('id', 'person_name', 'person_id',
school_name=F('person_school__school_name'),
school_address=F('person_school__school_address'),
hometown_name=F('person_ht__hometown_name')).prefetch_related('person, hometown, school')
I have a database model as shown below. Consider the data as 2 different books each having 3 ratings.
class Book(models.Model):
name = models.CharField(max_length=50)
class Review(models.Model):
book = models.ForeignKey(Book)
review = models.CharField(max_length=1000)
rating = models.IntegerField()
Question : Is it possible to group all the ratings in a list, for each book with a single query. I'm looking to do this at database level, without iterating over the Queryset in my code. Output should look something like :
{
'book__name':'book1',
'rating' : [3, 4, 4],
'average' : 3.66,
'book__name':'book2',
'rating : [2, 1, 1] ,
'average' : 1.33
}
I've tried this query, but neither are the ratings grouped by book name, nor is the average correct :
Review.objects.annotate(average=Avg('rating')).values('book__name','rating','average')
Edit : Added clarification that I'm looking for a method to group the elements at database level.
You can do this. Hope this helps.
Review.objects.values('book__name').annonate(average=Avg('rating'))
UPDATE:
If you want all the ratings of a particular book in a list, then you can do this.
from collections import defaultdict
ratings = defaultdict(list)
for result in Review.objects.values('book__name', 'rating').order_by('book__name', 'rating'):
ratings[result['book__name']].append(result['rating'])
You will get a structure like this :
[{ book__name: [rating1, rating2, ] }, ]
UPDATE:
q = Review.objects.values('book__name').annonate(average=Avg('rating')).filter().prefetech_related('rating')
q[0].ratings.all() # gives all the ratings of a particular book name
q[0].average # gives average of all the ratings of a particular book name
Hope this works (I'm not sure, sorry), but you need to add related_ name attribute
class Review(models.Model):
book = models.ForeignKey(Book, related_name='rating')
UPDATE:
Sorry to say, but you need something called as GROUP_CONCAT in SQL , but it is not supported in Django ORM currently.
You can use Raw SQL or itertools
from django.db import connection
sql = """
SELECT name, avg(rating) AS average, GROUP_CONCAT(rating) AS rating
FROM book JOIN review on book.id = review.book_id
GROUP BY name
"""
cursor = connection.cursor()
cursor.execute(sql)
data = cursor.fetchall()
DEMO
Here i want to do is that ,i want to list all the person who didn't blocked me.Here in the table Blocked there is two columns name
who and whose . In whose column i store the id of the person whom i blocked and in the who column i store my id. Now i want to do that, when the blocked person click on
view-person button in my web page he cannot see profile of the person one who blocked him.
when i did this query blocked_list = Blocked.objects.filter(whose = user_id). Now i got the list of the persons who blocked me. Now i want to exclude all this person from this query total_profiles = persons.objects.all().exclude(blocked_list). How can i do this.
models.py
class persons(models.Model):
full_name = models.CharField(max_length=200)
class blocked(models.Model):
who = models.ForeignKey(persons)
whose = models.IntegerField(null=True)
views.py
def blocked(request):
blocked_list = Blocked.objects.filter(whose = user_id)
total_profiles = persons.objects.all().exclude(blocked_list)
return render_to_response('profiles/view_all.html', {'total_profiles':total_profiles,'}, context_instance=RequestContext(request),)
please correct the question if it is not correct.
You can try this:
total_profiles = persons.objects.all().exclude(id__in = blocked_list.values_list('id', flat=True))
It's untested, but adapted from this answer.
Some notes:
if persons has the default manager, you can omit all().
whose does not have an index, so it will become slow when your dataset gets big. You can use a ForeignKey field instead of an IntegerField
the common convention is to capitalize class names and to write model names in singular i.e. Person instead of persons
How can I make an select with two (or more) joined tables using Django Models?
For example:
class Album(models.Model):
artist = models.ForeignKey(Musician)
name = models.CharField(max_length=100)
release_date = models.DateField()
num_stars = models.IntegerField()
class Song(models.Model):
album = models.ForeignKey(Album)
name = models.CharField(max_length=100)
num_stars = models.IntegerField()
SELECT * from album, song where (album.id = song.album_id) and (album.artist_id = X)
Django querysets want to return lists of model instances, which doesn't quite map to the the SQL notion of returning data from multiple tables joined into a single result table.
To achieve the effect of the SQL query you provide as an example, you could do one of the following:
1) If you want to iterate over your data by Songs, you can limit the songs you query by the artist id, like so:
songs = Song.objects.filter(album__artist__id=123)
for song in songs:
print song.name
print song.album.name
... do what ever you need to do with song here ...
If you are concerned about performance, you can add select_related() to your queryset.
# the depth argument tells django to only follow foreign key relationships
# from Song one level deep
songs = Song.objects.select_related(depth=1).filter(album__artist__id=123)
for song in songs:
# django has now already fetched the album data, so accessing it
# from the song instance does not cause an additional database hit
print song.album.name
2) If you want to iterate over your data by Album, you can do things like so:
albums = Album.objects.filter(artist__id = 123)
for album in albums:
for song in album.song_set.all():
print song.name
print album.name
... etc ...
You can perform raw sql query, but that's probably not the thing you want to do. Explain how your Models look like and what you want to achieve and probably django ORM will be enough to perform the query.
Is there an easy way to fetch the ManyToMany objects from a query that returns more than one object? The way I am doing it now doesn't feel as sexy as I would like it to. Here is how I am doing it now in my view:
contacts = Contact.objects.all()
# Use Custom Manager Method to Fetch Each Contacts Phone Numbers
contacts = PhoneNumber.objects.inject(contacts)
My Models:
class PhoneNumber(models.Model):
number = models.CharField()
type = models.CharField()
# My Custom Manager
objects = PhoneNumberManager()
class Contact(models.Model):
name = models.CharField()
numbers = models.ManyToManyField(PhoneNumber, through='ContactPhoneNumbers')
class ContactPhoneNumbers(models.Model):
number = models.ForeignKey(PhoneNumber)
contact = models.ForeignKey(Contact)
ext = models.CharField()
My Custom Manager:
class PhoneNumberManager(models.Manager):
def inject(self, contacts):
contact_ids = ','.join([str(item.id) for item in contacts])
cursor = connection.cursor()
cursor.execute("""
SELECT l.contact_id, l.ext, p.number, p.type
FROM svcontact_contactphonenumbers l, svcontact_phonenumber p
WHERE p.id = l.number_id AND l.contact_id IN(%s)
""" % contact_ids)
result = {}
for row in cursor.fetchall():
id = str(row[0])
if not id in result:
result[id] = []
result[id].append({
'ext': row[1],
'number': row[2],
'type': row[3]
})
for contact in contacts:
id = str(contact.id)
if id in result:
contact.phonenumbers = result[id]
return contacts
There are a couple things you can do to find sexiness here :-)
Django does not have any OOTB way to inject the properties of the through table into your Contact instance. A M2M table with extra data is a SQL concept, so Django wouldn't try to fight the relations, nor guess what should happen in the event of namespace collision, etc... . In fact, I'd go so far as to say that you probably do not want to inject arbitrary model properties onto your Contact object... if you find yourself needing to do that, then it's probably a sign you should revise your model definition.
Instead, Django provides convenient ways to access the relation seamlessly, both in queries and for data retrieval, all the while preserving the integrity of the entities. In this case, you'll find that your Contact object offers a contactphonenumbers_set property that you can use to access the through data:
>>> c = Contact.objects.get(id=1)
>>> c.contactphonenumbers_set.all()
# Would produce a list of ContactPhoneNumbers objects for that contact
This means, in your case, to iterate of all contact phone numbers (for example) you would:
for contact in Contact.objects.all():
for phone in contact.contactphonenumbers_set.all():
print phone.number.number, phone.number.type, phone.ext
If you really, really, really want to do the injection for some reason, you'll see you can do that using the 3-line code sample immediately above: just change the print statements into assignment statements.
On a separate note, just for future reference, you could have written your inject function without SQL statements. In Django, the through table is itself a model, so you can query it directly:
def inject(self, contacts):
contact_phone_numbers = ContactPhoneNumbers.objects.\
filter(contact__in=contacts)
# And then do the result construction...
# - use contact_phone_number.number.phone to get the phone and ext
# - use contact_phone_number.contact to get the contact instance