i'm facing an annoing bug in Django and i don't know how i can solve it.
I'm trying to show up names from one other table using INNER JOIN but isn't working.
Just to clarify, i have two database canaisAssoc and formatosAssoc, i'm trying to show up from canaisAssoc the name of the canais and from formatosAssoc the name of the formato, if you take a look at my views.py, i'm using ajax, because i need to choose one value from one select in template to fill up others, the issue is, when i run the canaisAssoc.objects.select_related('idCanal').filter(idSite_id = site, ativo = True).all()
if i run this
print(channels.query)
The return of the query is:
SELECT campanhas_canaisassoc.id, campanhas_canaisassoc.idSite_id, campanhas_canaisassoc.idCanal_id, campanhas_canaisassoc.ativo, campanhas_canais_dados.id, campanhas_canais_dados.nomeCanais, campanhas_canais_dados.ativo FROM campanhas_canaisassoc INNER JOIN campanhas_canais_dados ON (campanhas_canaisassoc.idCanal_id = campanhas_canais_dados.id) WHERE (campanhas_canaisassoc.ativo = True AND campanhas_canaisassoc.idSite_id = 3)
If i run this in phpMyAdmin, they are working as expected, showing up all values i'm need it
id|idSite_id|idCanal_id|ativo|id|nomeCanais|ativo
However, if i inspect the call of JSON, they are returning this:
{"channel": [{"model": "campanhas.canaisassoc", "pk": 14, "fields": {"idSite": 3, "idCanal": 13, "ativo": true}}, {"model": "campanhas.canaisassoc", "pk": 15, "fields": {"idSite": 3, "idCanal": 1, "ativo": true}}, {"model": "campanhas.canaisassoc", "pk": 16, "fields": {"idSite": 3, "idCanal": 4, "ativo": true}}, {"model": "campanhas.canaisassoc", "pk": 17, "fields": {"idSite": 3, "idCanal": 10, "ativo": true}}, {"model": "campanhas.canaisassoc", "pk": 63, "fields": {"idSite": 3, "idCanal": 30, "ativo": true}}]
Note in the fields there are only 3 values, idSite,idCanal and ativo, nothing i can do in the query returns me the nomeCanais, even using canaisAssoc.objects.raw with que query above doesn't works, any ideas on how i can solve this? below are my codes!
Thanks!
Models.py
class Canais_dados(models.Model):
id = models.AutoField(primary_key=True)
nomeCanais = models.TextField()
ativo = models.BooleanField(default=True)
def __str__(self):
return self.nomeCanais
class Formatos_dados(models.Model):
id = models.AutoField(primary_key=True)
nomeFormato = models.TextField()
ativo = models.BooleanField(default=True)
def __str__(self):
return self.nomeFormato `Preformatted text`
class canaisAssoc(models.Model):
id = models.AutoField(primary_key=True)
idSite = models.ForeignKey(Sites_dados, on_delete=models.CASCADE, related_name='idSitesCanaisAssoc',null=False)
idCanal = models.ForeignKey(Canais_dados, on_delete=models.CASCADE, related_name='idCanaisCanaisAssoc',null=False)
ativo = models.BooleanField(default=True,null=True)
class formatosAssoc(models.Model):
id = models.AutoField(primary_key=True)
idSite = models.ForeignKey(Sites_dados, on_delete=models.CASCADE, related_name='idSiteFormatoAssoc',null=False)
idFormato = models.ForeignKey(Formatos_dados, on_delete=models.CASCADE, related_name='idFormatoFormatoAssoc',null=False)
ativo = models.BooleanField(default=True,null=True)
views.py
# AJAX
def load_channel(request,site):
channels = canaisAssoc.objects.select_related('idCanal').filter(idSite_id = site, ativo = True).all()
formats = formatosAssoc.objects.select_related('idFormato').filter(idSite_id = site,ativo = True).all()
channelName = serialize("json",channels)
serializedChannel = json.loads(channelName)
formatName = serialize("json",formats)
serializedFormat = json.loads(formatName)
return JsonResponse({'formats':serializedFormat,'channel': serializedChannel})
I try objects.raw and even then is not working as expected.
I'm expecting something like that in JSON return:
{"channel": [{"model": "campanhas.canaisassoc", "pk": 14, "fields": {"idSite": 3, "idCanal": 13, "ativo": true,"nomeCanais": "Canal Y"}}, {"model": "campanhas.canaisassoc", "pk": 15, "fields": {"idSite": 3, "idCanal": 1, "ativo": true,"nomeCanais": "CanalXX"}}, {"model": "campanhas.canaisassoc", "pk": 16, "fields": {"idSite": 3, "idCanal": 4, "ativo": true,"nomeCanais": "CanalXXY"}}, {"model": "campanhas.canaisassoc", "pk": 17, "fields": {"idSite": 3, "idCanal": 10, "ativo": true,"nomeCanais": "CanalXXYZ"}}, {"model": "campanhas.canaisassoc", "pk": 63, "fields": {"idSite": 3, "idCanal": 30, "ativo": true,"nomeCanais": "CanalXXYZ"}}]
You misunderstand how select_related works. It's purpose is to reduce amount of DB requests to get data for related models, not to change their fields.
Consider example:
class ModelA(models.Model):
a_data = ... # example field
... # other-fields
class ModelB(models.Model):
a_object = models.ForeignKey(ModelA, ...)
b_data = ... # example field
... # other-fields
Without select_related:
b_objects = ModelB.objects.all() # first request
for b in b_objects:
a = b.a_object # +1 request for each a_object.
a_data = a.a_data # get a_data.
b_data = b.b_data # get b_data.
... # do something.
Here first request gets data from ModelB's table. Then for each a_object an extra request is made to ModelA's table.
With select_related:
b_objects = ModelB.objects.select_related('a_object') # first request.
for b in b_objects:
a = b.a_object # no extra requests.
a_data = a.a_data # get a_data.
b_data = b.b_data # get b_data.
... # do something.
Here first request gets ModelB's and related ModelA's data all at once. No extra requests are made. But the way you access that data from the code does not change.
See docs on select_related for more details.
Similar method for reverse or m-2-m relations: prefetch_related.
What you need is values + field lookups:
b_objects = ModelB.objects.values(
'pk', # b's own fields
'b_data', # b's own fields
'a_object__pk', # related a_object's fields
'a_object__a_data', # related a_object's fields
... # other fields
)
This will return a queryset of dictionaries like this:
<QuerySet [
{'pk': 1, 'b_data': 'b_1', 'a_object__pk': 1, 'a_object__a_data': 'a_1'},
{'pk': 2, 'b_data': 'b_2', 'a_object__pk': 2, 'a_object__a_data': 'a_2'},
...]>
Then you may serialize it or convert to other data-types before serializing.
Similar method values_list returns lists of values without keys:
<QuerySet [
[1, 'b_1', 1, 'a_1'],
[2, 'b_2', 2, 'a_2'],
...]>
Other way is annotate + F-objects with field lookups:
b_objects = ModelB.objects.annotate(
a_pk = F('a_object__pk'), # add a_object's pk as 'a_pk'
a_data = F('a_object__data'), # add a_object's a_data as 'a_data'
)
It adds extra properties to b objects with corresponding values from related a objects.
Combine that with values to give keys shorter names:
b_objects = ModelB.objects.annotate(
a_pk = F('a_object__pk'), # add a_object's pk as 'a_pk'
a_data = F('a_object__data'), # add a_object's a_data as 'a_data'
).values(
'pk', # b's own fields
'b_data', # b's own fields
'a_pk', # annotated fields
'a_data', # annotated fields
...
)
Raw is not recommended for security reasons.
Also consider using DRF for APIs larger than 1-2 endpoints. It's made just for that.
Related
t = PurchaseHeader.objects.first()
t.__dict__
{
'_state': <django.db.models.base.ModelState object at 0x7f4b34aa7fa0>,
'id': 3,
'ref': 'jhkh',
'goods': Decimal('-100.00'),
'discount': Decimal('0.00'),
'vat': Decimal('-20.00'),
'total': Decimal('-120.00'),
'paid': Decimal('-120.00'),
'due': Decimal('0.00'),
'date': datetime.date(2020, 11, 7),
'due_date': datetime.date(2020, 11, 14),
'period': '202007',
'status': 'c',
'created': datetime.datetime(2020, 11, 7, 15, 46, 48, 191772, tzinfo=<UTC>),
'cash_book_id': None,
'supplier_id': 1128,
'type': 'pc'
}
When I joined the supplier table I was disappointed to find that the columns are not included in the dict. Below, t.__dict__ is the same as above. I noticed that the Supplier model instance is cached inside of t._state so I guess I could create my own method which all models inherit from which does what i want - all the columns from all tables inside a dict. But I wondered if anybody knew a way of doing this sort of thing out of the box?
t = PurchaseHeader.objects.select_related("supplier").first()
t.__dict__
select_related's goal is actually to prefetch data so that it doesn't need to be fetched in a second query when accessing "supplier". Instead it already fetched this data using a join in the original query.
If you want to obtain a dict based of your model that also contains the data of a relation in it, your best bet is using ModelSerializer with a nested serializer. Assuming that your supplier model is called Supplier it would look something like this:
class SupplierSerializer(serializers.ModelSerializer):
class Meta:
model = Supplier
fields = ['name', 'other_field'] # Add more Supplier fields
class PurchaseHeaderSerializer(serializers.ModelSerializer):
supplier = SupplierSerializer(read_only=True)
class Meta:
model = PurchaseHeader
fields = ['supplied', 'vat', 'total'] # Add more PurchaseHeader fields
You can then use the PurchaseHeaderSerializer like this:
purchase_header = PurchaseHeader.objects.select_related("supplier").first()
the_dict_you_want = PurchaseHeaderSerializer(instance=purchase_header).data
I have a data model where I store certain Events that happen. Each Event is linked to an EventType. The data model roughly looks like this:
class EventType(models.Model):
name = ...
class Event(models.Model):
date = ...
event_type = models.ForeignKey(EventType)
What I would like to know is how often each event time appeared. I tried it like this:
Event.objects.values('event_type', count=Count('event_type'))
But the result looks like this:
<QuerySet [{'count': 1, 'event_type': 71}, {'count': 1, 'event_type': 2}, {'count': 1, 'event_type': 71}, {'count': 1, 'event_type': 71}, ...
So the entries did not get grouped. How can I make it such that they are grouped?
You can use following query
Event.objects.all().values('event_type').annotate(count=Count('event_type'))
I'm trying to optimize the queries for my moderation system, build with Django and DRF.
I'm currently stuck with the duplicates retrieval: currently, I have something like
class AdminSerializer(ModelSerializer):
duplicates = SerializerMethodField()
def get_duplicates(self, item):
if item.allowed:
qs = []
else:
qs = Item.objects.filter(
allowed=True,
related_stuff__language=item.related_stuff.language
).annotate(
similarity=TrigramSimilarity('name', item.name)
).filter(similarity__gt=0.2).order_by('-similarity')[:10]
return AdminMinimalSerializer(qs, many=True).data
which works fine, but does at least one additional query for each item to display. In addition, if there are duplicates, I'll do additional queries to fill the AdminMinimalSerializer, which contains fields and related objects of the duplicated item. I can probably reduce the overhead by using a prefetch_related inside the serializer, but that doesn't prevent me from making several queries per item (assuming I have only one related item to prefetch in AdminMinimalSerializer, I'd still have ~2N + 1 queries: 1 for the items, N for the duplicates, N for the related items of the duplicates).
I've already looked at Subquery, but I can't retrieve an object, only an id, and this is not enough in my case. I tried to use it in both a Prefetch object and a .annotate.
I also tried something like Item.filter(allowed=False).prefetch(Prefetch("related_stuff__language__related_stuff_set__items", queryset=Items.filter..., to_attr="duplicates")), but the duplicates property is added to "related_stuff__language__related_stuff_set", so I can't really use it...
I'll welcome any idea ;)
Edit: the real code lives here. Toy example below:
# models.py
from django.db.models import Model, CharField, ForeignKey, CASCADE, BooleanField
class Book(Model):
title = CharField(max_length=250)
serie = ForeignKey(Serie, on_delete=CASCADE, related_name="books")
allowed = BooleanField(default=False)
class Serie(Model):
title = CharField(max_length=250)
language = ForeignKey(Language, on_delete=CASCADE, related_name="series")
class Language(Model):
name = CharField(max_length=100)
# serializers.py
from django.contrib.postgres.search import TrigramSimilarity
from rest_framework.serializers import ModelSerializer, SerializerMethodField
from .models import Book, Language, Serie
class BookAdminSerializer(ModelSerializer):
class Meta:
model = Book
fields = ("id", "title", "serie", "duplicates", )
serie = SerieAdminAuxSerializer()
duplicates = SerializerMethodField()
def get_duplicates(self, book):
"""Retrieve duplicates for book"""
if book.allowed:
qs = []
else:
qs = (
Book.objects.filter(
allowed=True, serie__language=book.serie.language)
.annotate(similarity=TrigramSimilarity("title", book.title))
.filter(similarity__gt=0.2)
.order_by("-similarity")[:10]
)
return BookAdminMinimalSerializer(qs, many=True).data
class BookAdminMinimalSerializer(ModelSerializer):
class Meta:
model = Book
fields = ("id", "title", "serie")
serie = SerieAdminAuxSerializer()
class SerieAdminAuxSerializer(ModelSerializer):
class Meta:
model = Serie
fields = ("id", "language", "title")
language = LanguageSerializer()
class LanguageSerializer(ModelSerializer):
class Meta:
model = Language
fields = ('id', 'name')
I'm trying to find a way to prefetch related objects and duplicates so that I can get rid of the get_duplicates method in BookSerializer, with the N+1 queries it causes, and have only a duplicates field in my BookSerializer.
Regarding data, here would be an expected output:
[
{
"id": 2,
"title": "test2",
"serie": {
"id": 2,
"language": {
"id": 1,
"name": "English"
},
"title": "series title"
},
"duplicates": [
{
"id": 1,
"title": "test",
"serie": {
"id": 1,
"language": {
"id": 1,
"name": "English"
},
"title": "first series title"
}
}
]
},
{
"id": 3,
"title": "random",
"serie": {
"id": 3,
"language": {
"id": 1,
"name": "English"
},
"title": "random series title"
},
"duplicates": []
}
]
I'm using Django 2.0 and Django REST Framework
I have a model like below.
class Contact(models.Model):
first_name = models.CharField(max_length=100)
class AmountGiven(models.Model):
contact = models.ForeignKey(Contact, on_delete=models.PROTECT)
amount = models.FloatField(help_text='Amount given to the contact')
#property
def total_payable(self):
return self.amount
#property
def amount_due(self):
returned_amount = 0
for returned in self.amountreturned_set.all():
returned_amount += returned.amount
return self.total_payable - returned_amount
class AmountReturned(models.Model):
amount_given = models.ForeignKey(AmountGiven, on_delete=models.CASCADE)
amount = models.FloadField()
I have to get the top 10 contacts of the amount given and due respectively.
In my view, I'm filtering data like
#api_view(http_method_names=['GET'])
def top_ten(request):
filter_type = request.query_params.get('type', None)
if filter_type == 'due':
# query for due type
elif filter_type == 'given':
qs = Contact.objects.filter(
user=request.user
).values('id').annotate(
amount_given=Sum('amountgiven__amount')
).order_by(
'-amount_given'
)[:10]
graph_data = []
for q in qs:
d = {}
contact = Contact.objects.get(pk=q['id'])
d['contact'] = contact.full_name if contact else 'Unknown'
d['value'] = q['amount_given']
graph_data.append(d)
return Response(graph_data)
else:
raise NotFound('No data found for given filter type')
the type query can be due or given.
The code for given type is working fine as all fields are in the database. But how can I filter based on the virtual field for due type?
What I have to do is to annotate Sum of amount_due property group by contact.
You cannot filter based on #property.
As far as I understand your problem correctly you can aggregate sum of related AmountGiven and sum of AmountReturned, then calculate due field which keep result of subtracting letter and former.
The query:
from django.db.models import Sum, Value
from django.db.models.functions import Coalesce
Contact.objects.filter(
amountgiven__amount__gt=0
).annotate(
due=Sum('amountgiven__amount') - Coalesce(Sum('amountgiven__amountreturned__amount'), Value(0))
).order_by('-due').values_list('due', 'id')
will return:
<QuerySet [{'id': 3, 'due': 2500.0}, {'id': 1, 'due': 2450.0}, {'id': 2, 'due': 1500.0}]>
However with this solution you cannot distinct between many AmountGiven across one Contact. You get big picture like results.
If you want split due value per AmountGiven instance the just annotate like so:
AmountGiven.objects.annotate(
due=Sum('amount') - Coalesce(Sum('amountreturned__amount'), Value(0))
).order_by('-due').values_list('due', 'contact__id', 'id')
which returns
<QuerySet [
{'contact__id': 3, 'id': 3, 'due': 2500.0},
{'contact__id': 1, 'id': 1, 'due': 1750.0},
{'contact__id': 2, 'id': 2, 'due': 1500.0},
{'contact__id': 1, 'id': 4, 'due': 350.0},
{'contact__id': 1, 'id': 5, 'due': 350.0}
]>
References
Coalesce
I have an app with lots of investors that invest in the same rounds, which belong to companies, as seen below. However when a user(investor) is logged in, i only want him to be able to see HIS investments.
{
"id": 1,
"name": "Technology Company",
"rounds": [
{
"id": 1,
"kind": "priced round",
"company": 1,
"investments": [
{
"id": 1,
"investor": 1,
"round": 1,
"size": 118000,
},
{
"id": 2,
"investor": 2,
"round": 1,
"size": 183000,
},
]
}
]
},
Currently, my viewsets extend get_queryset as so:
class CompanyViewSet(viewsets.ModelViewSet):
def get_queryset(self):
user = self.request.user
investor = Investor.objects.get(user=user)
companies = Company.objects.filter(rounds__investments__investor=investor)
return companies
It retrieves the investments that belong to the investor, but when it takes those investments again to retrieve the rounds, it grabs the round with ALL investors.
How can i write this such that it only displays the investments that below to the Investor?
Here are my models:
class Company(models.Model):
name = models.CharField(max_length=100)
class Round(PolymorphicModel):
company = models.ForeignKey(Company, related_name='rounds', blank=True, null=True)
class Investment(PolymorphicModel):
investor = models.ForeignKey(Investor, related_name='investor')
size = models.BigIntegerField(default=0)
Your description of what happens is pretty unclear. What does "when it takes those investments again" mean? Anyway, I'm guessing what you need to do is to use .prefetch_related and a Prefetch object.
from django.db.models import Prefetch
class CompanyViewSet(viewsets.ModelViewSet):
def get_queryset(self):
user = self.request.user
investor = Investor.objects.get(user=user)
companies = Company.objects.filter(
rounds__investments__investor_id=investor.id
).prefetch_related(Prefetch(
'rounds__investments',
queryset=Investment.objects.filter(
investor_id=investor.pk,
),
))
return companies
I haven't tested this snippet but it should give you a pointer in the right direction. I also optimized the investor lookup to check the id only, this will save you an unnecessary indirection.