I have a model which is related with other model.
class Foo(...)
...
class Bar(...)
foo = models.ForeignKey(Foo, related_name('bars'))
I need to load all related Bars for many Foos so I use prefetch_related.
Foo.objects.filter(...).prefetch_related('bars')
In debug_toolbar I see additional query which takes Bars for all foos, but there are also queries which takes Bars for every single Foo.
Doesn't prefetch_related work in sqlite? Or am I doing something wrong?
I iterate through all Foos in template, but I think this does not matter.
Ok, the problem was in my code. I used latest() method in my manager which executed another query.
Related
Descirption
There is a Foo model that has a ForeignKey field bar:
class Foo(models.Model):
...
bar = models.ManyToManyField('bar.Bar', related_name='some_bar')
...
Also Foo has get_config() method which returns its fields including bar like:
def get_config(self):
return {
...
'bar': map(lambda x: x.get_config(), self.bar.all())
...
Now there are 10,000 rows of Foo in the database. There are some Bar rows as well.
Trying to retrieve the data about 10,000 Foo including the nested Bar data:
query = Foo.objects.all().prefetch_related('bar')
return [obj.get_config() for obj in query]
Problem
The query executes around 6 seconds. If there is no bar field - only 400 milliseconds.
The prefetch seems to not work completely bar.get_config() seem to hit the database for each iteration step. It is supposed to simply load all Bar objects once and get config from that bar-query to populate each foo config.
Thoughts
If use iterator() for the for loop, the call almost stalls and take dozens of seconds: [obj.get_config() for obj in query.iterator()]
The prefetch_related is working. Django prefetches Bar with a single SELECT containing WHERE clause with ids of all Foo. This SELECT with 10k ids is much faster than 10k SELECTS, but much slower than single SELECT without any filter.
I'm trying to find an optimal way to execute a query, but got myself confused with the prefetch_related and select_related use cases.
I have a 3 table foreign key relationship: A -> has 1-many B h-> as 1-many C.
class A(models.model):
...
class B(models.model):
a = models.ForeignKey(A)
class C(models.model):
b = models.ForeignKey(B)
data = models.TextField(max_length=50)
I'm trying to get a list of all C.data for all instances of A that match a criteria (an instance of A and all its children), so I have something like this:
qs1 = A.objects.all().filter(Q(id=12345)|Q(parent_id=12345))
qs2 = C.objects.select_related('B__A').filter(B__A__in=qs1)
But I'm wary of the (Prefetch docs stating that:
any subsequent chained methods which imply a different database query
will ignore previously cached results, and retrieve data using a fresh
database query
I don't know if that applies here (because I'm using select_related), but reading it makes it seem as if anything gained from doing select_related is lost as soon as I do the filter.
Is my two-part query as optimal as it can be? I don't think I need prefetch as far as I'm aware, although I noticed I can swap out select_related with prefetch_related and get the same result.
I think your question is driven by a misconception. select_related (and prefetch_related) are an optimisation, specifically for returning values in related models along with the original query. They are never required.
What's more, neither has any impact at all on filter. Django will automatically do the relevant joins and subqueries in order to make your query, whether or not you use select_related.
I have three models in play, and want to avoid N+1 query.
class Rule(models.Model):
pass
class RuleConstraint(models.Model):
rules = models.ManyToManyField(Rule)
class Foo(models.Model):
rule = models.ForeignKey(Rule, related_name='foos')
for a given foo, I can get related RuleConstraints like the following
RuleContraint.objects.filter(rules__foos=foo)
Question is, how do I avoid N+1 query symptom, when I have foos instead of a single foo.
ie, is there a better way of doing than
for foo in foos:
rule_constraints = RuleConstraint.objects.filter(rules__foos=foo)
You want prefetch_related
foos = Foo.objects.prefetch_related('rule__rule_constraint')
You can then iterate through the queryset with:
for foo in foos:
rule_constraints = foo.rule.ruleconstraint_set.all()
You can improve this further by using select_related to fetch the rule.
foos = Foo.objects.select_related('rule').prefetch_related('rule__rule_constraint')
For more information see the prefetch related docs - your models are very similar to those in the examples.
I'm working with a model:
class Foo(models.Model):
name = models.CharField(max_length=255)
bars = models.ManyToManyField('Bar')
In a view I have access to a list of Barobjects and need to get all Foo objects that have any of the Bar objects in their list of bars, so I do this:
foos = Foo.objects.filter(bars__in=list_of_bars)
The issue is that there are duplicates, if a Foo has 2 bars, and both of those bars are in my list_of_bars, which a simple distinct solves:
foos = Foo.objects.distinct().filter(bars__in=list_of_bars)
That's all good and well, except that adding DISTINCT to the query makes it very slow, due to there being 2 million Foo objects in the database.
With all that said, what way(s) can you think of that don't use DISTINCT, but achieve the same result set? If it involves altering models, that's OK.
You could always select uniques in python:
foos = set(Foo.objects.filter(bars__in=list_of_bars))
You need query set. So, maybe (just maybe) this quick nasty hack could help You. It is ugly, but maybe it is fast (not sure about that).
def get_query_set(self):
foos = set(Foo.objects.filter(bars__in=list_of_bars))
return Foo.objects.filter(id__in=[f.id for f in foos])
When I do this,
>>> b = Blog.objects.all()
>>> b
I get this:
>>>[<Blog: Blog Title>,<Blog: Blog Tile>]
When I query what type b is,
>>> type(b)
I get this:
>>> <class 'django.db.models.query.QuerySet'>
What does this mean? Is it a data type like dict, list, etc?
An example of how I can build data structure like a QuerySet will be appreciated.
I would want to know how Django builds that QuerySet (the gory details).
A django queryset is like its name says, basically a collection of (sql) queries, in your example above print(b.query) will show you the sql query generated from your django filter calls.
Since querysets are lazy, the database query isn't done immediately, but only when needed - when the queryset is evaluated. This happens for example if you call its __str__ method when you print it, if you would call list() on it, or, what happens mostly, you iterate over it (for post in b..). This lazyness should save you from doing unnecessary queries and also allows you to chain querysets and filters for example (you can filter a queryset as often as you want to).
Yes, it's just another type, built like every other type.
A QuerySet represents a collection of objects from your database. It can have zero, one or many filters. Filters narrow down the query results based on the given parameters. In SQL terms, a QuerySet equates to a SELECT statement, and a filter is a limiting clause such as WHERE or LIMIT.
https://docs.djangoproject.com/en/1.8/topics/db/queries/
A QuerySet is a list of objects of a given model, QuerySet allow you to read data from database