Iterating over results, sorted by a m2m - django

My data model is simple:
class Neighborhood(models.Model):
name = models.CharField(max_length = 50)
slug = models.SlugField()
class Location(models.Model):
company = models.ForeignKey(Company)
alt_name = models.CharField()
neighborhoods = models.ManyToManyField(Neighborhood)
I would like to supply a page on my site that lists all locations by their neighborhood (s). If it were singular, I think {% regroup %} with a {% ifchanged %} applied to the neighborhood name would be all that I need, but in my case, having it be a m2m, I'm not sure how do this. A location may have multiple neighborhoods, and so I would like them to be redundantly displayed under each matching neighborhood.
I'm also aware of FOO_set but that's per Object; I want to load the entire data set.
The final result (in the template) should be something like:
Alameda
Crazy Thai
Castro
Kellys Burgers
Pizza Orgasmica
Filmore
Kellys Burgers
Some Jazz Bar
Mission
Crazy Thai
Elixir
...
The template syntax would (ideally?) look something like:
{% for neighborhood in neighborhood_list %}
{% ifchanged %}{{ neighborhood.name }}{% endifchanged %}
{% for location in neighborhood.restaurants.all %}
{{ location.name }}
{% endfor %}
{% endfor %}

I'd just do it the expensive way and cache the result over scratching my head. Your template example would work fine until performance becomes an issue in generating that one page per X cache timeout.
You could do it in python as well if the result set is small enough:
# untested code - don't have your models
from collections import defaultdict
results = defaultdict(list)
for location_m2m in Location.neighborhoods.through.objects.all() \ # line wrap
.select_related('neighborhood', 'location', 'location__company__name'):
if location_m2m.location not in results[location_m2m.neighborhood]:
results[location_m2m.neighborhood].append(location_m2m.location)
# sort now unique locations per neighborhood
map(lambda x: x.sort(key=lambda x: x.location.company.name), results.values())
# sort neighborhoods by name
sorted_results = sorted(results.items(), key=lambda x:x[0].name)

create another model that would define your m2m relationship:
class ThroughModel(models.Model):
location = models.ForeignKey('Location')
neighborhood = models.ForeignKey('Neighborhood')
and use it as the through model for your m2m field:
class Location(models.Model):
...
neighborhoods = models.ManyToManyField(Neighborhood, through='ThroughModel')
Then you can get all your neighbourhoods sorted:
ThroughModel.objects.select_related().order_by('neighborhood__name')
This is untested.
Or, if you cannot change the database, just do a raw SQL join.

Related

Prefetching model with GenericForeignKey

I have a data structure in which a Document has many Blocks which have exactly one Paragraph or Header. A simplified implementation:
class Document(models.Model):
title = models.CharField()
class Block(models.Model):
document = models.ForeignKey(to=Document)
content_block_type = models.ForeignKey(to=ContentType)
content_block_id = models.CharField()
content_block = GenericForeignKey(
ct_field="content_block_type",
fk_field="content_block_id",
)
class Paragraph(models.Model):
text = models.TextField()
class Header(models.Model):
text = models.TextField()
level = models.SmallPositiveIntegerField()
(Note that there is an actual need for having Paragraph and Header in separate models unlike in the implementation above.)
I use jinja2 to template a Latex file for the document. Templating is slow though as jinja performs a new database query for every Block and Paragraph or Header.
template = get_template(template_name="latex_templates/document.tex", using="tex")
return template.render(context={'script': self.script})
\documentclass[a4paper,10pt]{report}
\begin{document}
{% for block in chapter.block_set.all() %}
{% if block.content_block_type.name == 'header' %}
\section{ {{- block.content_block.latex_text -}} }
{% elif block.content_block_type.name == 'paragraph' %}
{{ block.content_block.latex_text }}
{% endif %}
{% endfor %}
\end{document}
(content_block.latex_text() is a function that converts a HTML string to a Latex string)
Hence I would like to prefetch script.blocks and blocks.content_block. I understand that there are two methods for prefetching in Django:
select_related() performs a JOIN query but only works on ForeignKeys. It would work for script.blocks but not for blocks.content_block.
prefetch_related() works with GenericForeignKeys as well, but if I understand the docs correctly, it can only fetch one ContentType at a time while I have two.
Is there any way to perform the necessary prefetching here? Thank you for your help.
Not really an elegant solution but you can try using reverse generic relations:
from django.contrib.contenttypes.fields import GenericRelation
class Paragraph(models.Model):
text = models.TextField()
blocks = GenericRelation(Block, related_query_name='paragraph')
class Header(models.Model):
text = models.TextField()
level = models.SmallPositiveIntegerField()
blocks = GenericRelation(Block, related_query_name='header')
and prefetch on that:
Document.objects.prefetch_related('block_set__header', 'block_set__paragraph')
then change the template rendering to something like (not tested, will try to test later):
\documentclass[a4paper,10pt]{report}
\begin{document}
{% for block in chapter.block_set.all %}
{% if block.header %}
\section{ {{- block.header.0.latex_text -}} }
{% elif block.paragraph %}
{{ block.paragraph.0.latex_text }}
{% endif %}
{% endfor %}
\end{document}
My bad, I did not notice that document is an FK, and reverse FK can not be joined with select_related.
First of all, I would suggest to add related_name="blocks" anyway.
When you prefetch, you can pass the queryset. But you should not pass filters by doc_id, Django's ORM adds it automatically.
And if you pass the queryset, you can also add select/prefetch related call there.
blocks_qs = Block.objects.all().prefetch_related('content_block')
doc_prefetched = Document.objects.prefetch_related(
Prefetch('blocks', queryset=blocks_qs)
).get(uuid=doc_uuid)
But if you don't need extra filters or annotation, the simpler syntax would probably work for you
document = (
Document.objects
.prefecth_related('blocks', 'blocks__content_block')
.get(uuid=doc_uuid)
)

How to correctly display data from two related models in Django ListView

I have two models as below:
class Loo(models.Model):
loo_type = models.CharField(max_length=3, default="LOO"...)
loo_from = models.ForeignKey(Harr, on_delete=models.CASCADE, ...)
loo_fac = models.DecimalField(max_digits=7, decimal_places=.....)
class Woo(models.Model):
woo_item = models.AutoField(primary_key=True, ...)
woo_loo = models.ForeignKey(Loo, on_delete=models.CASCADE, ...)
woo_dt = models.DateField(null=True, ...)
woo_rate = models.DecimalField(max_digits=7, decimal_places=.....)
I am trying to display data from the models using the following listview:
class WhoLooView(ListView):
template_name = "who_loo_list.html"
context_object_name = 'wholoos'
model = Loo
def get_context_data(self, **kwargs):
context = super(WhoLooView, self).get_context_data(**kwargs)
context.update({
'woo_item_list': Woo.objects.order_by('-woo_dt'),
})
return context
def get_queryset(self):
return Loo.objects.order_by('loo_from')
Note that there can be more than one "woo_item" per instance of Loo (id), so in the listview there will be occasions when for the same Loo id it will have two instances of Woo/s, and thus need to be displayed separately (preferably in two distinct rows).
What I have tried so far creates extra (adjacent) columns for each Loo id and whichever Loo/s have a single instance of Woo, are shown in the normal fashion as expected.
How does one take care of such a situation. Can we have a nested row for cases where there are more than one instance of Woo?
Edit
What I have tried (based on your code sample):
{% for obj in wholoos %} <!-- wholoos : context object name -->
{{ obj.loo_type }}
{% for item in obj.woo_set.all %}
{{ item.woo_dt }}
{% endfor %}
{% endfor %}
But now I am not getting anything from the second model Woo.
Edit 2
I am getting the same result as earlier with my original code. Check the image below:
If you notice (in the image above), objects # 31 and 34 have one each of child objects (sub-objects). # 32 and 33 have two each. I want them to be in separate rows and not columns. The reason is, in case there are a number of items for each parent item (which my DB design makes it imperative to be), I would end up with an enormous number of extra columns for the sub-objects (and that too without any column header).
you can loop the instances of Loo as shown, in your templates, don't have to override the get_context_data.
{% for obj in object_list %}
{{ obj.loo_type }}
{% for item in obj.woo_set.all %}
{{ item.woo_dt }}
{% endfor %}{% endfor %}

Django do something with each different value in a model field

I know I've seen this before, but I can't find now that I am ready to implement it.
I'm trying to list items on a page by category and subcategory, but I only want 1 category and then the subcategories in the category. Hopefully my code will make sense.
class Objects(models.Model):
# Main Checkbox.
category = models.CharField(
max_length=100,
blank=True,
)
# Checkboxes under Main Checkbox.
subcategory = models.CharField(
max_length=100,
blank=True,
)
So my objects are stored as:
category1, subcategory1
category1, subcategory2
category2, subcategory1
category2, subcategory2
And when displayed should give me:
category1
subcategory1
subcategory2
category2
subcategory1
subcategory2
How do I set my query so that my results show me each "different" category?
First off you should look to normalise your objects, meaning you could keep your subcategories as a separate object and use models.foreignKey to link them together. See Willem Van Onsem's answer for this.
However, to solve the current problem you should be able to utilize the regroup template tag in order to aggregate the categories.
In your case it would look something like:
{% regroup objects by category as categories %}
{% for category in categories %}
{{ category.grouper }}
{% for subcategory in category.list %}
{{ subcategory }}
{% endfor %}
{% endfor %}
The data duplication anti-pattern
Many computer scientists see this as bad design, since it introduces data duplication. Imagine that you later want to change the name of a category, then that means you need to find all occurrences of that category, and rename them. If you only use this for the Objects model, then that is perhaps doable, but if all sorts of elements belong to Categorys, then this easily gets out of hand.
Furthermore it also restricts categories: two different categories can never have the same name (which might here be reasonable), nor can we attach much properties to the Category: imagine that we want to add a description for the category, then that description needs to be repeated over all rows, or if we decide to store it only in one row, then it will be hard to find that specific row. Furthermore if there are two rows with different descriptions, then what description to pick?
The database will also be very huge: each row should repeat the same category. If a category on average takes 15 characters, that means that we will - depending on the encoding - easily waste 8 bytes per row (well the row contains 16 bytes for a string given it is UTF-8 encoding, and only ASCII characters, but a ForeignKey will frequently use 8 bytes). If we would add a description that has on average 63 characters, then we would again waste another 64 bytes per row. For a small amount of rows, that is not a problem, but the problem easily scales problematic. The above are of course only estimates on what problems might arise, do not see this as the "real numbers": the size a database takes is determined by a lot of parameters that are here either ignored, or estimated.
Yes all these problems probably can be solved, but instead of solving the problems ad-hoc, it is better to normalize the database.
Normalizing the models
Normalization typically means that we introduce extra tables that store for example one record per Category, and use ForeignKeys to refer to that record. For you example a normalized variant would be:
class Category(models.Model):
name = models.CharField(max_length=100)
class SubCategory(models.Model):
name = models.CharField(max_length=100)
category = models.ForeignKey(Category, on_delete=models.CASCADE)
class Object(models.Model):
subcategory = models.ForeignKey(
SubCategory,
null=True,
on_delete=models.SET_NULL
)
So we store Categorys and SubCategorys in dedicated tables, and link the models together with ForeignKeys.
Rendering lists of (Sub)Categorys
Now that we normalized the models, we can effectively render the Categorys with:
# app/views.py
def some_view(request):
categories = Category.objects.prefetch_related('subcategory_set')
return render(request, 'app/some_template.html', {'categories': categories})
and in the app/templates/some_template.html we then can render it as:
<ul>
{% for cat in categories %}
<li>{{ cat.name }}</li>
<ul>
{% for subcat in cat.subcategory_set %}
<li>{{ subcat.name }}</li>
{% endfor %}
</ul>
{% endfor %}
</ul>
We thus iterate over all categories, and for every cat, we iterate over the subcategory_set.

django count specific rows in queryset

class Order(models.Model):
name = models.CharField(max_length=100)
# other fields..
user = models.ForeginKey(User)
old = models.BooleanField(default=False)
I want to display all the orders of a specific user, but I want to split them those which are "old" and the ones who are not.
So, currently I do in views.py:
orders = Order.objects.filter(user=user)
In template:
First table:
<table>
{% for order in orders %}
{% if not order.old %}
<tr>
<td>... </td>
</tr>
{% endif %}
{% endfor %}
</table>
And another table:
{% for order in orders %}
{% if order.old %}
<tr>
<td>...</td>
<tr>
{% endif %}
{% endfor %}
This way have some drawbacks, first, now I want to count how many of the orders are "old", to display this number in the template. I can't, unless I do another query.
Is it possible to annotate(number_of_old=Count('old'))? Or I have to do another query?
So what would be the best?
1. Do two queries, one with old=False, another with old=True, and pass two querysets to the template. And use |len filter on the querysets
2. Do one query like this and split them somehow in python? That will be less convenient as I have a similar structures which I want to split like that.
And should I call the DB .count() at all?
EDIT:
If I would write my model like this:
class Order(models.Model):
name = models.CharField(max_length=100)
# other fields..
user = models.ForeginKey(User)
old = models.BooleanField(default=False)
objects = CustomManager() # Custom manager
class CustomQueryset(models.QuerySet):
def no_old(self):
return self.filter(old=False)
class CustomManager(models.Manager):
def get_queryset(self):
return CustomQuerySet(model=self.model, using=self._db)
Is this template code produce one or two queries ?
{% if orders.no_old %}
{% for order orders.no_old %}
...
{% endfor %}
{% endif %}
You can't do any annotations, and there is no need to make .count() since you already have all the data in memory. So its really just between:
orders = Order.objects.filter(user=user)
old_orders = [o for o in orders if o.old]
new_orders = [o for o in orders if not o.old]
#or
old_orders = Order.objects.filter(user=user, old=True)
new_orders = Order.objects.filter(user=user, old=False)
In this specific scenario, I don't think there will be any performance difference. Personally I will choose the 2nd approach with the two queries.
A good read with tips about the problem: Django Database access optimization
Update
About the custom Manager which you introduce. I don't think you are doing it correctly I think what you want is this:
class CustomQueryset(models.QuerySet):
def no_old(self):
return self.filter(old=False)
class Order(models.Model):
name = models.CharField(max_length=100)
# other fields..
user = models.ForeginKey(User)
old = models.BooleanField(default=False)
#if you already have a manager
#objects = CustomManager.from_queryset(CustomQueryset)()
#if you dont:
objects = CustomQueryset.as_manager()
So having:
orders = Order.objects.filter(user=user)
If you do {% if orders.no_old %} will do another query, because this is new QuerySet instance which has no cache..
About the {% regroup %} tag
As you mention, in order to use it, you need to .order_by('old'), and if you have another order, you can still use it, just apply your order after the old, e.g. .order_by('old', 'another_field'). This way you will use only one Query and this will save you one iteration over the list (because Django will split the list iterating it only once), but you will get less readability in the template.

Django: dynamically access queryset data from different models

I'm interested in learning how to display all attributes of a list of querysets that come from different models.
Here's an example:
models.py
class MyModelA(models.Model):
attr1 = something
attr2 = something
class MyModelB(models.Model):
attr3 = something
attr4 = something
class MyModelC(models.Model):
attr5 = something
attr6 = something
views.py
Let's say we have three model instances that are stored in a list:
all_selected_queries = [mymodela, mymodelb, mymodelc]
For each queryset in the list, I want to display all model field titles and data in a template.
My approach:
# Loop through the list and get the verbose name title of each field ("titel")
for z in all_selected_queries:
queryset_fields = z._meta.get_fields()
for f in queryset_fields:
titel = f.verbose_name.title()
return titel
What challenges me is how to get the fields' values without having to include the actual attribute name (because they are different for each queryset).
So instead of explictly calling
f.attr1, f.attr2, f.attr3, f.attr4, f.attr5
for each field, I'd like to encounter a solution that works across model boundaries.
Thank you very much for your help!
You can try like this:
v_list = list()
for z in all_selected_queries:
queryset_fields = z._meta.get_fields()
values = dict()
for f in queryset_fields:
values[f.verbose_name.title()] = getattr(z, f.attname)
v_list.append(values)
return render(request,'some_template.html',{'values':v_list})
And show them in template:
{% for value in values %}
{% for key, val in value.items %}
<b>{{ key }}: </b>{{ val }}
{% endfor %}
{% endfor %}