Django three way join using Foreign Keys - django

I recently started evaluating Django for migrating our archaic web application written 10 years ago. I have been reading up Django documentation for the last few days, but haven't been able to figure out the best way to achieve a multi table database join in my case:
Model:
class Product(models.Model):
productid = models.IntegerField(primary_key=True, db_column='ProductId')
productname = models.CharField(max_length=120, db_column='ProductName')
class Testcases(models.Model):
testcaseid = models.IntegerField(primary_key=True, db_column='TestCaseId')
testcasename = models.CharField(max_length=240, db_column='TestCaseName')
class Testmatrix(models.Model):
testmatrixid = models.IntegerField(primary_key=True, db_column='TestMatrixId')
productid = models.ForeignKey(Product, db_column='ProductId')
testcaseid = models.ForeignKey(Testcases, db_column='TestCaseId')
class Status(models.Model):
testmatrixid = models.ForeignKey(Testmatrix, db_column='TestMatrixId')
title = models.CharField(max_length=240, db_column='Title', blank=True)
(Note that model was generated by inspectdb and I'd prefer not to modify it at this point in time)
View:
from django.shortcuts import render_to_response
from mysite.testmatrix.models import Product, Testcases, Testmatrix, Status
def get_products(request):
tm = list(Testmatrix.objects.filter(productid='abc'))
return render_to_response('products.html', {'tm': tm})
template is designed to be minimal at this point to help focus on the real issue in (views/model).
Template: (products.html)
{% extends "main.html" %}
{% block body %}
<table>
{% for tm in tm %}
<tr>
<td>{{ tm.testmatrixid }}</td>
<td>{{ tm.testcaseid.testcasename }}</td>
</tr>
{% endfor %}
</table>
{% endblock %}
Problem:
Although I'm able to join Testmatrix and Testcase models, I am unable to generate an equivalent queryset by joining all of TestMatrix, TestCase, Status records on say productid='abc'
I tried the following:
1) Use select_related between Testmatrix and Testcases and Product tables and was able to access attributes across all three models (testmatrixid, productid, productname, testcaseid, testcasename). However I'm not sure how to extend this auto foreign key referencing to Status model. This would have been easier if all Foreign Keys were defined within Testmatrix itself. But Status has a Foreign Key to TestMatrix.
2) I tried using something like: entries = Status.objects.filter(testmatrixid__productid=pid). This again gave me a queryset as a result of joining Testmatrix and Status, but not Testcases.
Pardon any blaring mistakes or bloopers. This is my very first post!

So you need to access a related_object. It is very simple.
First, add related_name here:
class Status(models.Model):
testmatrixid = models.ForeignKey(Testmatrix, db_column='TestMatrixId', related_name='statuses')
Now you can get all the statuses for desired Testmatrix like
test_matrix.statuses.all()
If you don't want to hit DB, when you access statuses, don't forget to use select_related.

Without any specific error messages, it is hard to diagnose the cause of said errors. However, in your example, in views.get_products: tm = list(Testmatrix.objects.filter(productid='abc')) will not work, because 'abc' is a string and your productid is actually a Product object (not just an integer, even though the field is an integer foreign key to your reference table's pk); you can do tm = list(Testmatrix.objects.filter(productid=Product.objects.get(product_name='abc')), assuming that 'abc' is the product name of the product record. When you set a field to models.ForeignKey(...), you address that reference record as an object, not an id.
Other than that, nothing blaring, your template looks solid and your models look fine to me. I would suggest creating some test cases to see where the errors lie: Django Testing; Also, this is also a great tutorial to understand TDD and unit testing with Django. Using unittests, you can verify every step of your application and make future updates with assurance.

Related

Django multiple choice and multi-select field in model and form

I am learning Django and have hit an inssue that I don't know what to Google for. I have the standard User model and a Profile model that points into the User model via a OneToOne field. That part plus the profile's Form and the CBVs all work.
A profile holder can have one or more professional affiliations. This is not for a union but my Dad, for example, as a welder, was affiliated with: "The Iron Workers", "Steam Fitters", "Boiler Makers", "Pipe Fitters" and a couple other specific unions that used welders. So if I want the profile and form to allow multiple choices and multiple selected results for affiliations then I see that you might use
class Profile(models.Model):
GROUPS = (
('IW', 'Iron Workers'),
('PF', 'Pipe Fitters'),
...
)
affiliation = models.CharField(
max_length=3,
choices=RANK,
default='U',
blank=True,
null=True,
)
...
In the associated form it looks like you use a
GROUPS = (
('IW', 'Iron Workers'),
('PF', 'Pipe Fitters'),
...
)
affiliation = forms.MultipleChoiceField(
choices=GROUPS
)
What if I want a db table of these choice items instead of tuples? If I have to add to a list of choices then editing the code might not be the most convenient response. Maybe the list of groups should be in the Django admin area.
How do I tell the model and form to look in a separate db table for the possible values and then how do I modify the model and form to allow for zero or more group affiliations?
You need to create a many-to-many relationship to a new model, let's call it Skill, that represents the available skills that can be dynamically created.
class Profile(models.Model):
# ...
skills = models.ManyToManyField(
to='myapp.Skill', # use a string in the format `app_name.model_name` to reference models to avoid issues using the model before it was defined
related_name='user_profiles', # the name for that relation from the point of view of a skill
)
class Skill(models.Model):
name = models.CharField(max_length=255)
abbreviation = models.CharField(max_length=3)
Then in your form you can just add that many-to-many field skills. It will by default be a multiple choice field in the django admin.
The relation can have zero to n entries for each pair of profile and skill. In other words, every user can have zero to n skills and vice versa.
To allow for a default value you will have to write a custom migration that creates that default entity of a Skill to make sure it is in the database at runtime.
In my case, I solved this way with a select2 list option.
It returns in the view a list of option selected.
FORMS
class PrecoattivoForm(forms.Form):
entrate = AccertamentiEnte.objects.all().values("entrata_provvedimento").distinct()
lista = []
for i in entrate:
lista.append([i.get('entrata_provvedimento'),i.get('entrata_provvedimento')])
entrata_accertamento= forms.MultipleChoiceField(choices=lista, required=False)
TEMPLATE
<div class="col-xl-10">
<selectclass="js-example-placeholder-multiple col-sm-12"
multiple="multiple"
name="{{ form.entrata_accertamento.name }}"
id="{{ form.entrata_accertamento.id_for_label }}">
{% for entrata in form.entrata_accertamento %}
<option {% if form.name.value != None %}value="
{{form.entrata_accertamento.value|upper }}"{% endif %}>{{ entrata }}</option>
{% endfor %}
</select>
</div>

Django query caching through a ForeignKey on a GenericRelation

Using a GenericRelation to map Records to Persons, I have the following code, which works correctly, but there is a performance problem I am trying to address:
models.py
class RecordX(Record): # Child class
....
class Record(models.Model): # Parent class
people = GenericRelation(PersonRecordMapping)
def people_all(self):
# Use select_related here to minimize the # of queries later
return self.people.select_related('person').all()
class Meta:
abstract = True
class PersonRecordMapping(models.Model):
person = models.ForeignKey(Person)
content_type = models.ForeignKey(ContentType)
object_id = models.PositiveIntegerField()
content_object = GenericForeignKey('content_type', 'object_id')
class Person(models.Model):
...
In summary I have:
RecordX ===GenericRelation===> PersonRecordMapping ===ForeignKey===> Person
The display logic in my application follows these relationships to display all the Persons mapped to each RecordX:
views.py
#rendered_with('template.html')
def show_all_recordXs(request):
recordXs = RecordX.objects.all()
return { 'recordXs': recordXs }
The problem comes in the template:
template.html
{% for recordX in recordXs %}
{# Display RecordX's other fields here #}
<ul>
{% for map in record.people_all %}{# <=== This generates a query every time! #}
<li>{{ map.person.name }}</li>
{% endfor %}
</ul>
{% endfor %}
As indicated, every time I request the Persons related to the RecordX, it generates a new query. I cannot seem to figure out how prefetch these initially to avoid the redundant queries.
If I try selected_related, I get an error that no selected_related fields are possible here (error message: Invalid field name(s) given in select_related: 'xxx'. Choices are: (none)). Not surprising, I now see -- there aren't any FKs on this model.
If I try prefetch_related('people'), no error is thrown, but I still get the same repeated queries on every loop in the template as before. Likewise for prefetch_related('people__person'). If I try prefetch_related('personrecordmapping'), I get an error.
Along the lines of this answer, I considered doing trying to simulate a select_related via something like:
PersonRecordMapping.objects.filter(object_id__in=RecordX.objects.values_list('id', flat=True))
but I don't understand the _content_object_cache well enough to adapt this answer to my situation (if even the right approach).
So -- how can I pre-fetch all the Persons to the RecordXs to avoid n queries when the page will display n RecordX objects?
Thanks for your help!
Ack, apparently I needed to call both -- prefetch_related('people', 'people__person'). SIGH.

Django: how to do this query?

I have 3 models:
a model Product that is linked with a ManyToMany Tag model
a model Tag that is linked with a ManyToMany Product model
a model TagContent that is linked with a OneToMany Tag model (= one content is linked to only one Tag and one Tag may have one or more TagContent)
The model TagContent is for multilanguage: one Product may have many Tag, but those Tag show up in the Web page through TagContent:
class Tag(BaseModel):
name = models.CharField(max_length=60, unique=True, default='')
class TagContent(BaseTranslatableModel):
tag = models.ForeignKey(Tag, null=True, default=None)
slug = models.SlugField(max_length=60, unique=True)
locale = models.CharField(max_length=2) # (e.g. "fr")
class Produit(BaseModel):
name = models.CharField(max_length=60, unique=True)
tags = models.ManyToManyField(Tag, related_name='produits')
Here's what I want to do: in my main page, the customer choose the language. Thus I will display all the products based on the language the customer has chosen.
In my main view, I want to display all the products, and all the tags so the user can click on a tag, and i'll filter the product.
The problem is that the tag should be translated in the current language. This means I have to filter the TagContent with the current locale, then get all Tag that are linked to those TagContent then get all the Product that are linked to those tags.
This should give something like (but it doesn't work because the foreign key is in the TagContent, and this is the main problem that blocks me):
Product.objects.filter(tags__in=
Tag.objects.filter(contents__in=
TagContent.objects.filter(langue__locale__exact=locale)
)
)
And in the templating model I need something like:
{% for p in products %}
{% for tag. in p.tags.all %}
{{ tag.name }}
{% endfor %}
{% endfor %}
In other words, I'd like to do this SQL query:
SELECT tc.slug, tc.name
FROM produit p
JOIN produit_tag pt
ON pt.produit_id = p.id
JOIN tag t
ON pt.tag_id = t.id
JOIN tag_content tc
ON tc.tag_id = t.id
JOIN langue l
ON tc.langue_id=l.id
WHERE l.locale='fr'
-> 2 minutes to write this SQL, 3 hours that I'm looking for the solution.
You can use this orm query to get the products:
products = Product.objects.prefetch_related("Tag", "TagContent").
filter(tags__tagcontent__locale="fr")
Django will produce a SQL just like your hand written one. However, multiple JOINs within one SQL is perhaps not a good idea, especially if the joined tables are quite large. Consider breaking the query into 2 might result in better performance (depends on the DB you are using):
fr_tags = Tag.objects.filter(tagcontent__locale="fr")
products = Product.objects.filter(tags__in=fr_tags)
Read about Field lookup that spans relationships:
https://docs.djangoproject.com/en/1.8/topics/db/queries/#lookups-that-span-relationships
Since you already have the SQL query, why don't you just send a raw query instead. And you can just pass the data into your template. It would be something similar to this:
from django.db import connections
cursor = connection.cursor()
query = (""" SELECT tc.slug, tc.name
FROM produit p
JOIN produit_tag pt
ON pt.produit_id = p.id
JOIN tag t
ON pt.tag_id = t.id
JOIN tag_content tc
ON tc.tag_id = t.id
JOIN langue l
ON tc.langue_id=l.id
WHERE l.locale='fr' """)
cursor.execute(query)
data = []
for row in cursor.fetchall():
slug = row[0]
name = row[1]
data_point = {'name': name, 'slug': slug}
data.append(data_point)
If you're not using PostgreSQL this is useless to you, but my former lead developer now maintains the django-pgviews project. It's useful when you have SQL queries containing complex joins that Django might not be able to do efficiently (or at all), which you can represent in a custom SQL View. You sacrifice Django's management, but you can keep using the ORM to query data for reading. Note also the django.contrib.postgres project that is in development.

Django left join m2m field

Here's my Model:
class User(models.Model):
pass
class Item(models.Model):
pass
class ItemVote(models.Model):
user = models.ForeignKey(User)
item = models.ForeignKey(Item)
vote = models.BooleanField()
I want to retrieve a list of Items, and I want to know if the current user has voted for each Item. How do I alter my query object so that it will generate sql similar to:
SELECT ...
FROM items
LEFT OUTER JOIN item_votes ON (item_votes.user_id = ? AND
item_votes.item_id = items.id)
You cannot in plain django queries. This is a many to many relation ship, you can even specify it more clearly by doing something like this:
class Item(models.Model):
votes = models.ManyToManyField(User, through='ItemVote', related='votedItems')
In a Many To Many relationship, we can speak of related sets (because there are multiple objects). While django can filter on related sets, something like:
Item.objects.filter(votes=request.user)
this will return you all Items that a user has voted for. However when you will list the .votes attribute in those objects, you will get all users that have voted for that item. the filtering is for the original object, never for the related set.
In a purely django way, you can expand my previous Item class to:
class Item(models.Model):
votes = models.ManyToManyField(User, through='ItemVote', related='votedItems')
def markVoted(self, user):
self.voted = user in self.votes
And then call this method for every object. This will however create an additional query to the votes set for every object (and no, select_related does not work for many to many relationships).
The only way to solve this is to add some SQL to your queryset using the extra method, like this:
Item.objects.extra('voted' : 'SELECT COUNT(*) FROM app_itemvote WHERE app_itemvote.item_id = app_item.id AND app_itemvote.user_id=%d'%request.user.pk)
Return ItemVote items:
items = ItemVote.objects.filter(user=user)
use in Django template:
{% for i in items %}
item: {{ i.item }}
voted: {{ i.voted }}
{% endfor %}

Django annotate query set with a count on subquery

This doesn't seem to work in django 1.1 (I believe this will require a subquery, therefore comes the title)
qs.annotate(interest_level= \
Count(Q(tags__favoritedtag_set__user=request.user))
)
There are items in my query set which are tagged and tags can be favorited by users, I would like to calculate how many times a user had favorited each item in the set via tags.
is there a way to construct a query like this without using extra()?
Thanks.
Looking at the add_aggregate function within django/db/models/sql/query.py, query objects will not be accepted as input values.
Unfortunately, there is currently no direct way within Django to aggregate/annotate on what amounts to a queryset, especially not one that is additionally filtered somehow.
Assuming the following models:
class Item(models.Model):
name = models.CharField(max_length=32)
class Tag(models.Model):
itemfk = models.ForeignKey(Item, related_name='tags')
name = models.CharField(max_length=32)
class FavoritedTag(models.Model):
user = models.ForeignKey(User)
tag = models.ForeignKey(Tag)
Also, you cannot annotate a queryset on fields defined via .extra().
One could drop into SQL in views.py like so:
from testing.models import Item, Tag, FavoritedTag
from django.shortcuts import render_to_response
from django.contrib.auth.decorators import login_required
from django.utils.datastructures import SortedDict
#login_required
def interest_level(request):
ruid = request.user.id
qs = Item.objects.extra(
select = SortedDict([
('interest_level', 'SELECT COUNT(*) FROM testing_favoritedtag, testing_tag \
WHERE testing_favoritedtag.user_id = %s \
AND testing_favoritedtag.tag_id = testing_tag.id \
AND testing_tag.itemfk_id = testing_item.id'),
]),
select_params = (str(ruid),)
)
return render_to_response('testing/interest_level.html', {'qs': qs})
Template:
{% for item in qs %}
name: {{ item.name }}, level: {{ item.interest_level }}<br>
{% endfor %}
I tested this using MySQL5. Since I'm no SQL expert though, I'd be curious as to how to optimize here, or if there is another way to "lessen" the amount of SQL. Maybe there is some interesting way to utilize the related_name feature here directly within SQL?
If you want to avoid dropping to raw SQL, another way to skin this cat would be to use a model method, which will then give you a new attribute on the model to use in your templates. Untested, but something like this on your Tags model should work:
class Tag(models.Model):
itemfk = models.ForeignKey(Item, related_name='tags')
name = models.CharField(max_length=32)
def get_favetag_count(self):
"""
Calculate the number of times the current user has favorited a particular tag
"""
favetag_count = FavoritedTag.objects.filter(tag=self,user=request.user).count()
return favetag_count
Then in your template you can use something like :
{{tag}} ({{tag.get_favetag_count}})
The downside of this approach is that it could hit the database more if you're in a big loop or something. But in general it works well and gets around the inability of annotate to do queries on related models. And avoids having to use raw SQL.