I have three tables with relations between;
Customer --> Order --> OrderLine
I want to know which customers placed orders with most order lines. Using pure SQL, I would go on using following query;
SELECT a.*, COUNT( c.id ) AS total FROM customer AS a
LEFT JOIN order AS b ON a.id = b.customer_id
LEFT JOIN order_line AS c ON b.id = c.order_id
GROUP BY a.id ORDER BY total DESC;
I saw another question here which I suspect could be the solution to my problem, but I can't really make sense of the answer to work over multiple relations.
This will get you the max order lines per customer:
from django.db.models import Max
custs = Customer.objects.annotate(max_order_lines=Max(order__orderline)).values('id', 'max_order_lines').order_by('-max_order_lines')
max_cust = custs[0]
Related
I have a table of Orders (Order_ID; Product_ID) which in most of case shows more than one Product_ID in one Order_ID.
With keeping the same format, i want a measure to return the count of Product_ID for each Order_ID.
Note, i don't want to group by Order_ID, so i want to keep the duplication.
You can use the following:
CountDups= COUNTROWS(FILTER(yourtable; yourtable[Order_ID] = EARLIER(yourtable[Order_ID]) && yourtable[Product_ID] = EARLIER((yourtable[Product_ID ]))))
It matches all Product_ID and Order_ID with other rows and counts them.
I ran into a surprising conundrum with Window Functions on Filtered QuerySets.
Consider two models: mymodel and relatedmodel where there is a one to many relationship (i.e. relatedmodel has a ForeignKey into mymodel).
I am using something like this:
window_lag = Window(expression=Lag("pk"), order_by=order_by)
window_lead = Window(expression=Lead("pk"), order_by=order_by)
window_rownnum = Window(expression=RowNumber(), order_by=order_by)
qs1 = mymodel.objects.filter(relatedmodel__field=XXX)
qs2 = qs1.annotate(row=window_rownnum, prior=window_lag, next=window_lead)
qs3 = qs2.filter(pk=myid)
which returns a lovely result for the object pk=myid. I now know its position in the filtered list and its prior and next and I use this to great effect in browsing filtered lists.
Significantly len(qs1) = len(qs2) is the size of the list and len(qs3)=1
Alas I just discovered this breaks badly when the filter is less specific:
window_lag = Window(expression=Lag("pk"), order_by=order_by)
window_lead = Window(expression=Lead("pk"), order_by=order_by)
window_rownnum = Window(expression=RowNumber(), order_by=order_by)
qs1 = mymodel.objects.filter(relatedmodel__field__contains=X)
qs2 = qs1.annotate(row=window_rownnum, prior=window_lag, next=window_lead)
qs3 = qs2.filter(pk=myid)
In this instance, qs2 suddenly has more rows in it that qs1! And len(qs2)>len(qs1).
Which totally breaks the browser in a sense (as prior and next are not reliable any more). The extra rows are duplicate mymodel objects, wherever more than one relatedmodel object matches the criterion.
I've traced this to the generated SQL.
This is the form of qs1's SQL:
SELECT DISTINCT
"mymodel"."id", "mymodel"."order_by" ....
FROM "mymodel"
INNER JOIN "relatedmodel" ON ("mymodel"."id" = "relatedmodel"."mymodel_id")
WHERE ("related_model"."field"::text LIKE '%X%')
ORDER BY "mymodel"."order_by" ASC
And this runs fine in my database engine as a SQL query, and returns the same number of rows that Django sees. All good.
Then the SQL that qs2 produces resembles:
SELECT DISTINCT
ROW_NUMBER() OVER (ORDER BY "mymodel"."order_by" ASC) AS "row",
LAG("mymodel"."id", 1) OVER (ORDER BY "mymodel"."order_by" ASC) AS "prior,
LEAD("mymodel"."id", 1) OVER (ORDER BY "mymodel"."order_by" ASC) AS "next",
"mymodel"."id", "mymodel"."order_by" ....
FROM "mymodel"
INNER JOIN "relatedmodel" ON ("mymodel"."id" = "relatedmodel"."mymodel_id")
WHERE ("related_model"."field"::text LIKE '%X%')
ORDER BY "mymodel"."order_by" ASC
And again this produces the same number of rows as I see in Django but that's more than qs1 when the relatedmodel matches more than once.
I can doctor the SQL and get what I want, namely by windowing after filtering:
SELECT
ROW_NUMBER() OVER (ORDER BY "mymodel"."order_by" ASC) AS "row"
LAG("mymodel"."id", 1) OVER (ORDER BY "mymodel"."order_by" ASC) AS "prior,
LEAD("mymodel"."id", 1) OVER (ORDER BY "mymodel"."order_by" ASC) AS "next",
"id", "order_by" ....
FROM (
SELECT DISTINCT
"mymodel"."id", "mymodel"."order_by" ....
FROM "mymodel"
INNER JOIN "relatedmodel" ON ("mymodel"."id" = "relatedmodel"."mymodel_id")
WHERE ("related_model"."field"::text LIKE '%X%')
ORDER BY "mymodel"."order_by" ASC
) AS QS
Which works beautifully and returns the same number of rows as qs1 again.
Adding just one window function inside the SELECT causes the DISTINCT to fail for some reason. DISTINCT works fine without a window function (returning only unique mymodel rows) but adding a window function breaks this.
Using the filter as a subquery on the window function works.
And Django supports subqueries, but I can't find a way to apply them here.
So I wonder if there is a way to do this. Namely, to apply the annotation as a wrapper around the QuerySet rather than as additional columns in the queryset.
I have an issue with DQL in Doctrine 2.
Subqueries seem to be unavailable in DQL, so I don't know how to transform :
SELECT DISTINCT a.ID_DOMAINE, L_DOMAINE, b.ID_SS_DOMAINE, L_SS_DOMAINE, c.ID_COMPETENCE, L_COMPETENCE
FROM ((qfq_prod.REF_DOMAINE a inner join qfq_prod.REF_SS_DOMAINE b on a.id_domaine = b.id_domaine)
inner join qfq_prod.REF_COMPETENCE c on b.id_ss_domaine = c.id_ss_domaine)
inner join qfq_prod.REF_PERS_COMP d on c.id_competence = d.id_competence
into a DQL expression.
I tried it and got
"Error: Class '(' is not defined."
I saw that we can use Query Builder to do this as well.
Being new with Doctrine 2, can someone explain to me how I can do this please ?
My DQL is currently :
$query = $this->getEntityManager()->createQuery ( "SELECT DISTINCT a.ID_DOMAINE, L_DOMAINE, b.ID_SS_DOMAINE, L_SS_DOMAINE, c.ID_COMPETENCE, L_COMPETENCE
FROM ((BdDoctrine\Entity\Domaine a inner join BdDoctrine\Entity\SsDomaine b on a.id_domaine = b.id_domaine)
inner join BdDoctrine\Entity\Competence c on b.id_ss_domaine = c.id_ss_domaine)
inner join BdDoctrine\Entity\LienPersComp d on c.id_competence = d.id_competence" );
$res = $query->getResult ();
Subqueries seem to be unavailable in DQL, so I don't know how to transform :
Actually, they are. Your code (no offence) is hardly readable so I will give you an example:
//controller
$repo = $this->getDoctrine()->getRepository("Your:Bundle:Category") ;
$results = $repo->findAllForSomePage() ;
// CategoryRepository.php
public function findAllForSomePage()
{
return $this->createQueryBuilder("o")
->innerJoin("o.products", "p", "WITH", "p.price>:price")->addSelect("p")
->setParameter("price", 50)
->where("o.id IN (SELECT s1.id FROM Your:Bundle:Something s1 WHERE s1.col1=5)")
->getQuery()->getResult() ;
}
Here is presumed you have Category hasMany Products relation and that you defined CategoryRepository file. You should never create queries in controller.
This example will fetch Categories only if they have Products with price bigger than 50, AND the ID of categories are those fetched by fictional subquery. This 100% works.
You should apply the same logic on your requirement.
Also, you should not use ON statement when using joins, that is handled by doctrine.
If you have the relationships properly defined in your entities, then you can make your joins on those relationships. And as Zeljko mentioned, you don't need to specify the ON condition, as the entities should already know how they are related. You are joining entities not tables. (That's under the hood.)
I don't know what your entities look like, so I made a guess at the relationship names below, but it should give you the idea.
$dql =
<<<DQL
SELECT
DISTINCT a.ID_DOMAINE, b.L_DOMAINE, b.ID_SS_DOMAINE, b.L_SS_DOMAINE, c.ID_COMPETENCE, c.L_COMPETENCE
FROM
BdDoctrine\Entity\Domaine a
JOIN a.ss_domaine b
JOIN b.competence c
JOIN c.lien_pers_comp d
DQL;
$query = $this->getEntityManager()->createQuery($dql);
$res = $query->getResult();
I want to get a list of the latest purchase of each customer, sorted by the date.
The following query does what I want except for the date:
(Purchase.objects
.all()
.distinct('customer')
.order_by('customer', '-date'))
It produces a query like:
SELECT DISTINCT ON
"shop_purchase.customer_id"
"shop_purchase.id"
"shop_purchase.date"
FROM "shop_purchase"
ORDER BY "shop_purchase.customer_id" ASC,
"shop_purchase.date" DESC;
I am forced to use customer_id as the first ORDER BY expression because of DISTINCT ON.
I want to sort by the date, so what the query I really need should look like this:
SELECT * FROM (
SELECT DISTINCT ON
"shop_purchase.customer_id"
"shop_purchase.id"
"shop_purchase.date"
FROM "shop_purchase"
ORDER BY "shop_purchase.customer_id" ASC,
"shop_purchase.date" DESC;
)
AS result
ORDER BY date DESC;
I don't want to sort using python because I still got to page limit the query. There can be tens of thousands of rows in the database.
In fact it is currently sorted by in python now and is causing very long page load times, so that's why I'm trying to fix this.
Basically I want something like this https://stackoverflow.com/a/9796104/242969. Is it possible to express it with django querysets instead of writing raw SQL?
The actual models and methods are several pages long, but here is the set of models required for the queryset above.
class Customer(models.Model):
user = models.OneToOneField(User)
class Purchase(models.Model):
customer = models.ForeignKey(Customer)
date = models.DateField(auto_now_add=True)
item = models.CharField(max_length=255)
If I have data like:
Customer A -
Purchase(item=Chair, date=January),
Purchase(item=Table, date=February)
Customer B -
Purchase(item=Speakers, date=January),
Purchase(item=Monitor, date=May)
Customer C -
Purchase(item=Laptop, date=March),
Purchase(item=Printer, date=April)
I want to be able to extract the following:
Purchase(item=Monitor, date=May)
Purchase(item=Printer, date=April)
Purchase(item=Table, date=February)
There is at most one purchase in the list per customer. The purchase is each customer's latest. It is sorted by latest date.
This query will be able to extract that:
SELECT * FROM (
SELECT DISTINCT ON
"shop_purchase.customer_id"
"shop_purchase.id"
"shop_purchase.date"
FROM "shop_purchase"
ORDER BY "shop_purchase.customer_id" ASC,
"shop_purchase.date" DESC;
)
AS result
ORDER BY date DESC;
I'm trying to find a way not to have to use raw SQL to achieve this result.
This may not be exactly what you're looking for, but it might get you closer. Take a look at Django's annotate.
Here is an example of something that may help:
from django.db.models import Max
Customer.objects.all().annotate(most_recent_purchase=Max('purchase__date'))
This will give you a list of your customer models each one of which will have a new attribute called "most_recent_purchase" and will contain the date on which they made their last purchase. The sql produced looks like this:
SELECT "demo_customer"."id",
"demo_customer"."user_id",
MAX("demo_purchase"."date") AS "most_recent_purchase"
FROM "demo_customer"
LEFT OUTER JOIN "demo_purchase" ON ("demo_customer"."id" = "demo_purchase"."customer_id")
GROUP BY "demo_customer"."id",
"demo_customer"."user_id"
Another option, would be adding a property to your customer model that would look something like this:
#property
def latest_purchase(self):
return self.purchase_set.order_by('-date')[0]
You would obviously need to handle the case where there aren't any purchases in this property, and this would potentially not perform very well (since you would be running one query for each customer to get their latest purchase).
I've used both of these techniques in the past and they've both worked fine in different situations. I hope this helps. Best of luck!
Whenever there is a difficult query to write using Django ORM, I first try the query in psql(or whatever client you use). The SQL that you want is not this:
SELECT * FROM (
SELECT DISTINCT ON
"shop_purchase.customer_id" "shop_purchase.id" "shop_purchase.date"
FROM "shop_purchase"
ORDER BY "shop_purchase.customer_id" ASC, "shop_purchase.date" DESC;
) AS result
ORDER BY date DESC;
In the above SQL, the inner SQL is looking for distinct on a combination of (customer_id, id, and date) and since id will be unique for all, you will get all records from the table. I am assuming id is the primary key as per convention.
If you need to find the last purchase of every customer, you need to do something like:
SELECT "shop_purchase.customer_id", max("shop_purchase.date")
FROM shop_purchase
GROUP BY 1
But the problem with the above query is that it will give you only the customer name and date. Using that will not help you in finding the records when you use these results in a subquery.
To use IN you need a list of unique parameters to identify a record, e.g., id
If in your records id is a serial key, then you can leverage the fact that the latest date will be the maximum id as well. So your SQL becomes:
SELECT max("shop_purchase.id")
FROM shop_purchase
GROUP BY "shop_purchase.customer_id";
Note that I kept only one field (id) in the selected clause to use it in a subquery using IN.
The complete SQL will now be:
SELECT *
FROM shop_customer
WHERE "shop_customer.id" IN
(SELECT max("shop_purchase.id")
FROM shop_purchase
GROUP BY "shop_purchase.customer_id");
and using the Django ORM it looks like:
(Purchase.objects.filter(
id__in=Purchase.objects
.values('customer_id')
.annotate(latest=Max('id'))
.values_list('latest', flat=True)))
Hope it helps!
I have a similar situation and this is how I'm planning to go about it:
query = Purchase.objects.distinct('customer').order_by('customer').query
query = 'SELECT * FROM ({}) AS result ORDER BY sent DESC'.format(query)
return Purchase.objects.raw(query)
Upside it gives me the query I want. Downside is that it is raw query and I can't append any other queryset filters.
This is my approach if I need some subset of data (N items) along with the Django query. This is example using PostgreSQL and handy json_build_object() function (Postgres 9.4+), but same way you can use other aggregate function in other database system. For older PostgreSQL versions you can use combination of array_agg() and array_to_string() functions.
Imagine you have Article and Comment models and along with every article in the list you want to select 3 recent comments (change LIMIT 3 to adjust size of subset or ORDER BY c.id DESC to change sorting of subset).
qs = Article.objects.all()
qs = qs.extra(select = {
'recent_comments': """
SELECT
json_build_object('comments',
array_agg(
json_build_object('id', id, 'user_id', user_id, 'body', body)
)
)
FROM (
SELECT
c.id,
c.user_id,
c.body
FROM app_comment c
WHERE c.article_id = app_article.id
ORDER BY c.id DESC
LIMIT 3
) sub
"""
})
for article in qs:
print(article.recent_comments)
# Output:
# {u'comments': [{u'user_id': 1, u'id': 3, u'body': u'foo'}, {u'user_id': 1, u'id': 2, u'body': u'bar'}, {u'user_id': 1, u'id': 1, u'body': u'joe'}]}
# ....
I have a Doctrine2 project with three tables: Product, Category and Rating. Each Product has a one and only one Category. A Category may have a parent and many children.
Each Product has many Rating.
I run the following query :
SELECT DISTINCT p, AVG(r.value)*COUNT(r) as globalrating
FROM AcmeProductBundle:Product p
JOIN p.ratings r JOIN p.category cat JOIN cat.parent par
WHERE par = '.$categoryID.' OR cat = '.$categoryID.'
ORDER BY globalrating DESC, p.name ASC
where $categoryID is the id of the category where I'm searching for the products.
The point is that, notwithstanding there are no Products in the category, one result is reported. In particular, the object is of type Acme\ProductBundle\Product and has no values into the fields and the id is 0! Also the "globalrating" value is empty! I don't understand why?
Any idea?
Thanks in advance.
Could you please try this:
$query = $this->em->createQuery('SELECT DISTINCT p, AVG(r.value)*COUNT(r) as globalrating
FROM AcmeProductBundle:Product p
JOIN p.ratings r JOIN p.category cat JOIN cat.parent par
WHERE par = :category OR cat = :category
ORDER BY globalrating DESC, p.name ASC');
$query->setParameter('category', $categoryID);
$returned = $query->getResult();
I come back on the problem today and I found the problem, finally!
Reading the related Doctrine docs, I discovered that a GROUP BY clause where missing. So, the right DQL query is:
SELECT DISTINCT p, AVG(r.value)*COUNT(r) as globalrating
FROM AcmeProductBundle:Product p
JOIN p.ratings r JOIN p.category cat JOIN cat.parent par
WHERE par = '.$categoryID.' OR cat = '.$categoryID.'
GROUP BY p.id
ORDER BY globalrating DESC, p.name ASC