Annotate QuerySet using raw SQL - django

In case I am asking the wrong question, let me first state the end goal: I need to allow users to filter a ListView by a field that's not in the primary model (Salesleadquote), but instead the field comes from a model (Salesleadbusinessgroup) with a FK to a related model (Saleslead).
The way I am trying to approach this is by annotating a field on Salesleadquote.
The models:
class Salesleadquote(models.Model):
salesleadquoteid = models.AutoField(db_column='SalesLeadQuoteId', primary_key=True)
salesleadid = models.ForeignKey(Saleslead, models.DO_NOTHING, db_column='SalesLeadId')
...
class Saleslead(models.Model):
salesleadid = models.AutoField(db_column='SalesLeadId', primary_key=True)
...
class Salesleadbusinessgroup(models.Model):
salesleadbusinessgroupid = models.AutoField(db_column='SalesLeadBusinessGroupId', primary_key=True)
salesleadid = models.ForeignKey(Saleslead, models.DO_NOTHING, db_column='SalesLeadId')
businessgroupid = models.ForeignKey(Businessgroup, models.DO_NOTHING, db_column='BusinessGroupId')
The desired result (queryset), in SQL:
SELECT slq.*, slbg.BusinessGroupId FROM crm.SalesLeadQuote slq
LEFT JOIN
(SELECT SalesLeadId, BusinessGroupId
FROM crm.SalesLeadBusinessGroup ) slbg
ON slbg.SalesLeadId = slq.SalesLeadId
WHERE slbg.BusinessGroupId IN (5,21)
I know I can get a RawQuerySet by doing something like
Salesleadquote.objects.raw("SELECT salesleadquote.*, \
salesleadbusinessgroup.businessgroupid \
FROM salesleadquote \
LEFT JOIN salesleadbusinessgroup \
ON salesleadquote.salesleadid = salesleadbusinessgroup.salesleadid \
WHERE salesleadbusinessgroup.businessgroupid IN (5,21)")
But I need the functionality of a QuerySet, so my idea was to annotate the desired field (businessgroupid) in Salesleadquote, but I've been struggling with how to accomplish this.

I implemented a work-around that doesn't address my original question but works for my use case. I created a view at the database level (called SalesLeadQuoteBG) using the SQL I had posted and then tied that to a model to use with Django's ORM.
class Salesleadquotebg(models.Model):
"""
This model represents a database view that extends the Salesleadquote table with a business group id column.
There can be multiple business groups per quote, resulting in duplicate quotes, but this is handled at the view and template layer
via filtering (users are required to select a business group).
"""
salesleadquoteid = models.IntegerField(db_column='SalesLeadQuoteId', primary_key=True) # Field name made lowercase.
salesleadid = models.ForeignKey(Saleslead, models.DO_NOTHING, db_column='SalesLeadId') # Field name made lowercase.
...
businessgroupid = models.ForeignKey(Businessgroup, models.DO_NOTHING, db_column='BusinessGroupId')
I am using django-filters for the filtering.
filters.py:
BG_CHOICES = (
(5, 'Machine Vision'),
(21, 'Process Systems'),
)
class BGFilter(django_filters.FilterSet):
businessgroupid = django_filters.ChoiceFilter(choices=BG_CHOICES)
class Meta:
model = Salesleadquotebg
fields = ['businessgroupid', ]

This can be done with pure Django ORM. Relationships can be followed backwards (filtering on salesleadbusinessgroup) and the double underscore syntax can be used to query attributes of the related model and also follow more relationships
Salesleadquote.objects.filter(
salesleadbusinessgroup__businessgroupid__in=(5,21)
)

Related

How to apply an arbitrary filter on a specific chained prefetch_related() within Django?

I'm trying to optimize the fired queries of an API. I have four models namely User, Content, Rating, and UserRating with some relations to each other. I want the respective API returns all of the existing contents alongside their rating count as well as the score given by a specific user to that.
I used to do something like this: Content.objects.all() as a queryset, but I realized that in the case of having a huge amount of data tons of queries will be fired. So I've done some efforts to optimize the fired queries using select_related() and prefetch_related(). However, I'm dealing with an extra python searching, that I hope to remove that, using a controlled prefetch_related() — applying a filter just for a specific prefetch in a nested prefetch and select.
Here are my models:
from django.db import models
from django.conf import settings
class Content(models.Model):
title = models.CharField(max_length=50)
class Rating(models.Model):
count = models.PositiveBigIntegerField(default=0)
content = models.OneToOneField(Content, on_delete=models.CASCADE)
class UserRating(models.Model):
user = models.ForeignKey(
settings.AUTH_USER_MODEL, blank=True, null=True, on_delete=models.CASCADE
)
score = models.PositiveSmallIntegerField()
rating = models.ForeignKey(
Rating, related_name="user_ratings", on_delete=models.CASCADE
)
class Meta:
unique_together = ["user", "rating"]
Here's what I've done so far:
contents = (
Content.objects.select_related("rating")
.prefetch_related("rating__user_ratings")
.prefetch_related("rating__user_ratings__user")
)
for c in contents: # serializer like
user_rating = c.rating.user_ratings.all()
for u in user_rating: # how to remove this dummy search?
if u.user_id == 1:
print(u.score)
Queries:
(1) SELECT "bitpin_content"."id", "bitpin_content"."title", "bitpin_rating"."id", "bitpin_rating"."count", "bitpin_rating"."content_id" FROM "bitpin_content" LEFT OUTER JOIN "bitpin_rating" ON ("bitpin_content"."id" = "bitpin_rating"."content_id"); args=(); alias=default
(2) SELECT "bitpin_userrating"."id", "bitpin_userrating"."user_id", "bitpin_userrating"."score", "bitpin_userrating"."rating_id" FROM "bitpin_userrating" WHERE "bitpin_userrating"."rating_id" IN (1, 2); args=(1, 2); alias=default
(3) SELECT "users_user"."id", "users_user"."password", "users_user"."last_login", "users_user"."is_superuser", "users_user"."first_name", "users_user"."last_name", "users_user"."email", "users_user"."is_staff", "users_user"."is_active", "users_user"."date_joined", "users_user"."user_name" FROM "users_user" WHERE "users_user"."id" IN (1, 4); args=(1, 4); alias=default
As you can see on the above fired queries I've only three queries rather than too many queries which were happening in the past. However, I guess I can remove the python searching (the second for loop) using a filter on my latest query — users_user"."id" IN (1,) instead. According to this post and my efforts, I couldn't apply a .filter(rating__user_ratings__user_id=1) on the third query. Actually, I couldn't match my problem using Prefetch(..., queryset=...) instance given in this answer.
I think you are looking for Prefetch object:
https://docs.djangoproject.com/en/4.0/ref/models/querysets/#prefetch-objects
Try this:
from django.db.models import Prefetch
contents = Content.objects.select_related("rating").prefetch_related(
Prefetch(
"rating__user_ratings",
queryset=UserRating.objects.filter(user__id=1),
to_attr="user_rating_number_1",
)
)
for c in contents: # serializer like
print(c.rating.user_rating_number_1[0].score)

Django query joining two tables

I am new in django. I want to create a query in django I tried with select_related but I don't know how to insert the second part of the condition: AND model1.qty >= model2.items
I've tried:
Model1.objects.select_related('model2).filter(model1.qty__gte=?)
But it's not working properly.
Below is the SQL query which I want to implement with django queryset:
SELECT model1.name,model2.name WHERE model1.id=model2.model1.id AND model1.qty >= model2.items
My models:
class Article(models.Model):
date_crea = models.DateTimeField('Créer le', auto_now_add=True)
designation = models.TextField('designation', max_length=500)
seuil = models.IntegerField('Seuil d\'alerte')
class Stock(models.Model):
date_crea = models.DateTimeField('Créer le', auto_now_add=True)
article = models.ForeignKey(Article, on_delete=models.CASCADE)
qte_reel = models.IntegerField('stock reel',default=0)
You use an F expression to refer to the value of a field in the database. There have to be relations to follow from the current object to the field on the object that you want to compare against.
I'm not clear how the question relates to the posted models, but an F expression can follow a foreign key so
Stock.objects.filter( qte_reel__gte = F( 'article__seuil' ))
would work, if those fields are the ones you want to compare.

Query intermediate through fields in django

I have a simple Relation model, where a user can follow a tag just like stackoverflow.
class Relation(models.Model):
user = AutoOneToOneField(User)
follows_tag = models.ManyToManyField(Tag, blank=True, null=True, through='TagRelation')
class TagRelation(models.Model):
user = models.ForeignKey(Relation, on_delete=models.CASCADE)
following_tag = models.ForeignKey(Tag, on_delete=models.CASCADE)
pub_date = models.DateTimeField(default=timezone.now)
class Meta:
unique_together = ['user', 'following_tag']
Now, to get the results of all the tags a user is following:
kakar = CustomUser.objects.get(email="kakar#gmail.com")
tags_following = kakar.relation.follows_tag.all()
This is fine.
But, to access intermediate fields I have to go through a big list of other queries. Suppose I want to display when the user started following a tag, I will have to do something like this:
kakar = CustomUser.objects.get(email="kakar#gmail.com")
kakar_relation = Relation.objects.get(user=kakar)
t1 = kakar.relation.follows_tag.all()[0]
kakar_t1_relation = TagRelation.objects.get(user=kakar_relation, following_tag=t1)
kakar_t1_relation.pub_date
As you can see, just to get the date I have to go through so much query. Is this the only way to get intermediate values, or this can be optimized? Also, I am not sure if this model design is the way to go, so if you have any recomendation or advice I would be very grateful. Thank you.
You need to use Double underscore i.e. ( __ ) for ForeignKey lookup,
Like this :
user_tags = TagRelation.objects.filter(user__user__email="kakar#gmail.com").values("following_tag__name", "pub_date")
If you need the name of the tag, you can use following_tag__name in the query and if you need id you can use following_tag__id.
And for that you need to iterate through the result of above query set, like this:
for items in user_tags:
print items['following_tag__name']
print items['pub_date']
One more thing,The key word values will return a list of dictionaries and you can iterate it through above method and if you are using values_list in the place of values, it will return a list of tuples. Read further from here .

Django queryset - Adding HAVING constraint

I have been using Django for a couple of years now but I am struggling today with adding a HAVING constraint to a GROUP BY.
My queryset is the following:
crm_models.Contact.objects\
.filter(dealercontact__dealer__pk__in=(265,),
dealercontact__activity='gardening',
date_data_collected__gte=datetime.date(2012,10,1),
date_data_collected__lt=datetime.date(2013,10,1))\
.annotate(nb_rels=Count('dealercontact'))
which gives me the following MySQL query:
SELECT *
FROM `contact`
LEFT OUTER JOIN `dealer_contact` ON (`contact`.`id_contact` = `dealer_contact`.`id_contact`)
WHERE (`dealer_contact`.`active` = True
AND `dealer_contact`.`activity` = 'gardening'
AND `contact`.`date_data_collected` >= '2012-10-01'
AND `contact`.`date_data_collected` < '2013-10-01'
AND `dealer_contact`.`id_dealer` IN (265))
GROUP BY `contact`.`id_contact`
ORDER BY NULL;
I would get exactly what I need with this HAVING constraint:
HAVING SUM(IF(`dealer_contact`.`type`='customer', 1, 0)) = 0
How can I get this fixed with a Django Queryset? I need a queryset in this instance.
Here I am using annotate only in order to get the GROUP BY on contact.id_contact.
Edit: My goal is to get the Contacts who have no "customer" relation in dealercontact but have "ref" relation(s) (according to the WHERE clause of course).
Models
class Contact(models.Model):
id_contact = models.AutoField(primary_key=True)
title = models.CharField(max_length=255L, blank=True, choices=choices_custom_sort(TITLE_CHOICES))
last_name = models.CharField(max_length=255L, blank=True)
first_name = models.CharField(max_length=255L, blank=True)
[...]
date_data_collected = models.DateField(null=True, db_index=True)
class Dealer(models.Model):
id_dealer = models.AutoField(primary_key=True)
address1 = models.CharField(max_length=45L, blank=True)
[...]
class DealerContact(Auditable):
id_dealer_contact = models.AutoField(primary_key=True)
contact = models.ForeignKey(Contact, db_column='id_contact')
dealer = models.ForeignKey(Dealer, db_column='id_dealer')
activity = models.CharField(max_length=32, choices=choices_custom_sort(ACTIVITIES), db_index=True)
type = models.CharField(max_length=32, choices=choices_custom_sort(DEALER_CONTACT_TYPE), db_index=True)
I figured this out by adding two binary fields in DealerContact: is_ref and is_customer.
If type='ref' then is_ref=1 and is_customer=0.
Else if type='customer' then is_ref=0 and is_customer=1.
Thus, I am now able to use annotate(nb_customers=Sum('is_customer')) and then use filter(nb_customers=0).
The final queryset consists in:
Contact.objects.filter(dealercontact__dealer__pk__in=(265,),
dealercontact__activity='gardening',
date_data_collected__gte=datetime.date(2012,10,1),
date_data_collected__lt=datetime.date(2013,10,1))\
.annotate(nb_customers=Sum('dealercontact__is_customer'))\
.filter(nb_customers=0)
Actually there is a way you can add your own custom HAVING and GROUP BY clauses if you need.
Just use my example with caution - if Django ORM code/paths will change in future Django versions, you will have to update your code too.
Image you have Book and Edition models, where for each book there can be multiple editions and you want to select first US edition date within Book queryset.
Adding custom HAVING and GROUP BY clauses in Django 1.5+:
from django.db.models import Min
from django.db.models.sql.where import ExtraWhere, AND
qs = Book.objects.all()
# Standard annotate
qs = qs.annotate(first_edition_date=Min("edition__date"))
# Custom HAVING clause, to limit annotation by US country only
qs.query.having.add(ExtraWhere(['"app_edition"."country"=%s'], ["US"]), AND)
# Custom GROUP BY clause will be needed too
qs.query.group_by.append(("app_edition", "country"))
ExtraWhere can contain not just fields, but any raw sql conditions and functions too.
Are you not using raw query just because you want orm object? Using Contact.objects.raw() generate instances similar filter. Refer to https://docs.djangoproject.com/en/dev/topics/db/sql/ for more help.
My goal is to get the Contacts who have no "customer" relation in
dealercontact but have "ref" relation(s) (according to the WHERE
clause of course).
This simple query fulfills this requirement:
Contact.objects.filter(dealercontact__type="ref").exclude(dealercontact__type="customer")
Is this enough, or do you need it to do something more?
UPDATE: if your requirement is
Contacts that have a "ref" relations, but do not have "customer"
relations with the same dealer
you can do this:
from django.db.models import Q
Contact.objects.filter(Q(dealercontact__type="ref") & ~Q(dealercontact__type="customer"))

Django ManyToManyField not present in Sqlite3

I'm new to Django and I have some issues with a ManyToMany relationship.
I work on a blastn automatisation and here are my classes:
class Annotation(models.Model):
sequence = models.IntegerField()
annotation = models.TextField()
start = models.IntegerField()
end = models.IntegerField()
class Blast(models.Model):
sequence = models.ManyToManyField(Annotation, through="AnnotBlast")
expectValue = models.IntegerField()
class AnnotBlast(models.Model):
id_blast = models.ForeignKey(Blast, to_field="id")
id_annot = models.ForeignKey(Annotation, to_field="id")
class Hit(models.Model):
id_hit = models.ForeignKey(Blast, to_field="id")
length = models.IntegerField()
evalue = models.IntegerField()
start_seq = models.IntegerField()
end_seq = models.IntegerField()
In a view, I want to access to Annotation's data from the rest of the model via this many to many field and then apply filters based on a form. But when I do a syncdb , the "sequence" field of the Blast class disappear :
In Sqlite3 :
.schema myApp_blast
CREATE TABLE "myApp_blast" (
"id" integer not null primary key,
"expectValue" integer not null
);
So I can't load data in this table as I want. I don't understand why this field disappear during the syncdb. How can I do to link the first class to the others (and then be able to merge data in a template) ?
A ManyToManyField isn't itself a column in the database. It's represented only by an element in the joining table, which you have here defined explicitly as AnnotBlast (note that since you're not defining any extra fields on the relationship, you didn't actually need to define a through table - Django would have done it automatically if you hadn't).
So to add data to your models, you add data to the AnnotBlast table pointing at the relevant Blast and Annotation rows.
For a many-to-many relationship an intermediate join table is created, see documentation here: https://docs.djangoproject.com/en/1.2/ref/models/fields/#id1