Django m2m produce Unnecessary inner join, that change SQL result - django

Django 2.1.4 (the like behavior exist on 2.0.4 too)
Models:
class Application(models.Model):
# many fileds
name = models.CharField(max_length=255)
seers = models.ManyToManyField('Agency', through='ApplicationAgencySeer')
parent = models.ForeignKey("self", null=True, blank=True, on_delete=SET_NULL)
class ApplicationAgencySeer(models.Model):
application = models.ForeignKey(Application, on_delete=models.CASCADE)
agency = models.ForeignKey('Agency', on_delete=models.CASCADE)
created = models.DateTimeField(auto_now_add=True)
now I wanna filter
# count 0
Application.objects.filter(seers__agency__id='c3e5ed58-a4d9-4ca6-a8f7-6793eb8e3e24').count() # 0
# but count 1
ApplicationAgencySeer.objects.filter(agency__id='c3e5ed58-a4d9-4ca6-a8f7-6793eb8e3e24').count() # 1
SELECT *
FROM "app_application"
INNER JOIN "myapp_applicationagencyseer"
ON ("app_application"."id" = "app_applicationagencyseer"."application_id")
INNER JOIN "myapp_agency"
ON ("app_applicationagencyseer"."agency_id" = "app_agency"."organization_ptr_id")
INNER JOIN "myapp_agency" T4 ON ("app_agency"."organization_ptr_id" = T4."parent_id")
WHERE T4."organization_ptr_id" = 'c3e5ed58-a4d9-4ca6-a8f7-6793eb8e3e24'
if remove INNER JOIN "myapp_agency" T4 ON ("app_agency"."organization_ptr_id" = T4."parent_id") all be right.
WHY parent_id why-why
I found the bug, maybe it some related <djangoproject> but 6 years ago. I think it already fixes before 2.1.4 release.
How to me compose right filter query, or avoid this situation. help me I am stuck.
SELECT *
FROM "myapp_application"
WHERE NOT ("myapp_application"."id" IN (SELECT U1."application_id"
FROM "myapp_applicationagencyseer" U1
INNER JOIN "myapp_agency" U2 ON (U1."agency_id" = U2."organization_ptr_id")
INNER JOIN "myapp_agency" U3 ON (U2."organization_ptr_id" = U3."parent_id")
WHERE U3."organization_ptr_id" = '9e71cff4-443d-4c60-ac2d-9dcca2a9c147'))
ORDER BY "myapp_application"."created_date" DESC;
result
application_id
7d83d056-5a7d-4095-9037-98bde29a3d78 otherfields..
7cb60afc-109d-4570-ad24-6cad6b7ddd9a otherfields.. <-- this row error
--return 0
(SELECT U1."application_id"
FROM "myapp_applicationagencyseer" U1
INNER JOIN "myapp_agency" U2 ON (U1."agency_id" = U2."organization_ptr_id")
INNER JOIN "myapp_agency" U3 ON (U2."organization_ptr_id" = U3."parent_id")
WHERE U3."organization_ptr_id" = '9e71cff4-443d-4c60-ac2d-9dcca2a9c147')
--althouth I have myapp_applicationagencyseer
id created agency_id application_id status
1 2018-12-10 17:41:14.272684 9e71cff4-443d-4c60-ac2d-9dcca2a9c147 7cb60afc-109d-4570-ad24-6cad6b7ddd9a 1
2 2018-12-11 19:25:58.818000 9e71cff4-443d-4c60-ac2d-9dcca2a9c147 7cb60afc-109d-4570-ad24-6cad6b7ddd9a 0
-- myapp_agency
organization_ptr_id accreditation parent
aff44d42-ce81-4c3e-b6e1-056ed9351adb Null Null
9e71cff4-443d-4c60-ac2d-9dcca2a9c147 10АА71 Null <-- It have Null parent

Related

Django filter ManyToMany with Q and Or

class State(models.Model):
name = models.CharField(max_length=55, unique=True)
class City(models.Model):
name = models.CharField(max_length=255)
states = models.ManyToManyField(State, related_name='cities')
parent = models.ForeignKey('self', on_delete=models.CASCADE, null=True, related_name='children', blank=True)
I have a state and I want to find cities that either have null parent or whose parent states doesn't contain a state.
I tried state.cities.filter(Q(parent=None) | ~Q(parent__states=state)) but it doesn't work and I query is strange:
`SELECT "city"."id", "city"."name", "city"."parent_id", FROM "city" INNER JOIN "city_states" ON ("city"."id" = "city_states"."city_id") WHERE ("city_states"."state_id" = 7 AND ("city"."parent_id" IS NULL OR NOT ("city"."parent_id" IN (SELECT U2."city_id" FROM "city_states" U2 WHERE (U2."state_id" = 7 AND U2."id" = "city_states"."id")) AND "city"."parent_id" IS NOT NULL))) ORDER BY "city"."name" ASC`
In particular what does the code AND U2."id" = "city_states"."id" perform?
You can simply work with the .exclude(…) [Django-doc] here:
state.cities.exclude(
parent__states=state
)
This will still work if parent is None/NULL, since then the parent__states=state does not hold.
This will construct a query that looks like:
SELECT app_name_city.*
FROM app_name_city
WHERE NOT (
app_name_city.parent_id IN (
SELECT U2.city_id
FROM `app_name_city_states U2
WHERE U2.state_id = id_of_state
)
AND app_name_city.parent_id IS NOT NULL
)
The subquery thus makes a list of cities for the given state, and we thus require the city to have a parent that is not in any of these states.

Django lt+gt condition Vs. not equal condition

I have the following model
class P(models.Model):
name = models.CharField(max_length=30, blank=False)
class pr(models.Model):
p = models.ForeignKey(P, on_delete=models.CASCADE, related_name='cs')
r = models.CharField(max_length=1)
c = models.ForeignKey(P, on_delete=models.CASCADE, related_name='ps')
rc = models.PositiveSmallIntegerField()
class Meta:
unique_together = (('p', 'c'),)
and the data
"id","name"
69,"Hunter"
104,"Savannah"
198,"Adrian"
205,"Andrew"
213,"Matthew"
214,"Aiden"
218,"Madison"
219,"Harper"
---
"id","r","rc","c_id","p_id"
7556,"F",1,219,213
7557,"M",1,219,218
7559,"H",3,218,213
7572,"F",1,214,213
7573,"M",1,214,218
7604,"F",1,198,213
7605,"M",1,198,218
7788,"H",3,104,205
7789,"F",1,104,213
7790,"M",1,104,218
7866,"M",1,69,104
7867,"F",1,69,205
the following two queries should produce similar results
A = P.objects.filter(Q(Q(ps__rc__lt = 3) | Q(ps__rc__gt = 3)), ps__p__cs__c = 198).exclude(pk = 198).annotate(bt=Count('ps__rc', filter=Q(ps__rc = 1, ps__p__cs__rc = 1)))
B = P.objects.filter(~Q(ps__rc = 3), ps__p__cs__c = 198).exclude(pk = 198).annotate(bt=Count('ps__rc', filter=Q(ps__rc = 1, ps__p__cs__rc = 1)))
strangely; query A produce the expected results but B is missing model instance 104!
After further troubleshooting I found that query B generates the following SQL:
SELECT "eftapp_p"."id", "eftapp_p"."name", COUNT("eftapp_pr"."rc") FILTER (WHERE (T4."rc" = 1 AND "eftapp_pr"."rc" = 1)) AS "bt" FROM "eftapp_p" LEFT OUTER JOIN "eftapp_pr" ON ("eftapp_p"."id" = "eftapp_pr"."c_id") LEFT OUTER JOIN "eftapp_p" T3 ON ("eftapp_pr"."p_id" = T3."id") LEFT OUTER JOIN "eftapp_pr" T4 ON (T3."id" = T4."p_id") WHERE (NOT ("eftapp_p"."id" IN (SELECT U1."c_id" FROM "eftapp_pr" U1 WHERE U1."rc" = 3)) AND T4."c_id" = 198 AND NOT ("eftapp_p"."id" = 198)) GROUP BY "eftapp_p"."id"
Is there a workaround to force the WHERE clause of generated SQL query to be:
WHERE NOT ("eftapp_pr"."rc" = 3) AND T4."c_id" = 198 AND NOT ("eftapp_p"."id" = 198)?
Django Version: 3.0.4 Python version: 3.7.3 Database:PostgreSQL 11.9 (Debian 11.9-0+deb10u1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 8.3.0-6) 8.3.0, 64-bit
I might be missing something here, but can't you just put ps__rc=3 in the .exclude filter?

Django ORM. Joining subquery on condition

I have a table TickerStatement, which contains financial statements about companies
class Statements(models.TextChoices):
"""
Supported statements
"""
capital_lease_obligations = 'capital_lease_obligations'
net_income = 'net_income'
price = 'price'
total_assets = 'total_assets'
short_term_debt = 'short_term_debt'
total_long_term_debt = 'total_long_term_debt'
total_revenue = 'total_revenue'
total_shareholder_equity = 'total_shareholder_equity'
class TickerStatement(TimeStampMixin):
"""
Model that represents ticker financial statements
"""
name = models.CharField(choices=Statements.choices, max_length=50)
fiscal_date_ending = models.DateField()
value = models.DecimalField(max_digits=MAX_DIGITS, decimal_places=DECIMAL_PLACES)
ticker = models.ForeignKey(Ticker, on_delete=models.CASCADE, null=False,
related_name='ticker_statements')
And now I'm trying to calculate a multiplier. The formula looks like:
(short_term_debt + total_long_term_debt) / total_shareholder_equity
I wrote a raw SQL query
SELECT "fin_tickerstatement"."fiscal_date_ending",
t2.equity AS "equity",
value AS "debt",
short_term_debt AS "short_term_debt",
(value + short_term_debt) / t2.equity AS "result"
FROM "fin_tickerstatement"
JOIN
(SELECT "fin_tickerstatement"."fiscal_date_ending",
fin_tickerstatement.value AS "equity"
FROM "fin_tickerstatement"
WHERE ("fin_tickerstatement"."ticker_id" = 12
AND "fin_tickerstatement"."fiscal_date_ending" >= date'2015-09-03'
AND "fin_tickerstatement"."name" = 'total_shareholder_equity')
GROUP BY "fin_tickerstatement"."fiscal_date_ending",
fin_tickerstatement.value
ORDER BY "fin_tickerstatement"."fiscal_date_ending" DESC) t2
ON fin_tickerstatement.fiscal_date_ending = t2.fiscal_date_ending
JOIN
(SELECT "fin_tickerstatement"."fiscal_date_ending",
fin_tickerstatement.value AS "short_term_debt"
FROM "fin_tickerstatement"
WHERE ("fin_tickerstatement"."ticker_id" = 12
AND "fin_tickerstatement"."fiscal_date_ending" >= date'2015-09-03'
AND "fin_tickerstatement"."name" = 'short_term_debt')
GROUP BY "fin_tickerstatement"."fiscal_date_ending",
fin_tickerstatement.value
ORDER BY "fin_tickerstatement"."fiscal_date_ending" DESC) t3
ON fin_tickerstatement.fiscal_date_ending = t3.fiscal_date_ending
WHERE ("fin_tickerstatement"."ticker_id" = 12
AND "fin_tickerstatement"."fiscal_date_ending" >= date'2015-09-03'
AND "fin_tickerstatement"."name" = 'total_long_term_debt')
GROUP BY "fin_tickerstatement"."fiscal_date_ending",
equity,
debt,
short_term_debt
ORDER BY "fin_tickerstatement"."fiscal_date_ending" DESC;
and have no idea how to translate it into Django ORM. Maybe you have some ideas or know some Django plugins that can help me.
The only way to solve this problem is to install django-query-builder.

Django Select across three models with filter criteria

I have three models:
class Vehicle(models.Model):
Vehicle_ID = models.AutoField('ID', primary_key= True)
Vehicle_VIN = models.CharField('FIN', max_length=30)
Vehicle_DELETED = models.BooleanField('Gelöscht',default=False)
class Recall(models.Model):
Recall_ID = models.AutoField('ID', primary_key= True)
Recall_CODE = models.CharField('Rückruf-Code',max_length=500, unique= True)
class Vehicle_Recall(models.Model):
Vehicle_Recall_ID = models.AutoField('ID', primary_key=True)
Vehicle = models.ForeignKey(Vehicle, on_delete=models.CASCADE)
Recall = models.ForeignKey(Recall, on_delete=models.CASCADE)
I want to make a Select Statement like this:
SELECT * FROM Vehicle INNER JOIN(Recall INNER JOIN Vehicle_Recall ON Recall.ID = Vehicle_Recall.Recall) ON Vehicle.ID = Vehicle_Recall.Vehicle WHERE Recall.ID=1 AND Vehicle.DELETED)=FALSE;
Is there a way to make such query in django?
You canuse the Django's ORM as follows (it is an example):
vehicles = Vehicle.objects.filter(Vehicle_DELETED=False).filter(
Q(vehicle_recall__Recall__Recall_ID=1)
)
which generates the following SQL query:
SQL SELECT "vehicle"."Vehicle_ID", "vehicle"."Vehicle_VIN", "vehicle"."Vehicle_DELETED" FROM "vehicle" INNER JOIN "vehicle_recall" ON ("vehicle"."Vehicle_ID" = "vehicle_recall"."Vehicle_id") WHERE ("vehicle"."Vehicle_DELETED" = False AND "vehicle_recall"."Recall_id" = 1)
You can use the raw query as follows:
vehicles = Vehicle.objects.raw(f"""
SELECT * FROM Vehicle
INNER JOIN(Recall INNER JOIN Vehicle_Recall ON Recall.ID = Vehicle_Recall.Recall)
ON Vehicle.ID = Vehicle_Recall.Vehicle
WHERE Recall.ID=1 AND Vehicle.DELETED=FALSE;
""")
and remember to adjust name of tables.
I propose the first solution, but the appropriate code depends on needs. I prepared only some example to present its simplicity.

What is the difference between these two Django Q-based queries:

Assuming a simple model like this:
class Item(models.Model):
name = models.CharField(max_length=10)
class Relation(models.Model):
item = models.ForeignKey(Item)
weight = models.IntegerField()
And a couple of Q objects like this:
some = Q(relation__x__gt=3)
others = Q(relation__x=7)
What is the semantic difference between:
first = Item.objects.filter(some, ~others)
and
second = Item.objects.filter(some).exclude(others)
NOTE: Querying through the relation appears to be different than querying a single simple object. The SQL generated for the two above queries is different.
Here's the generated SQL for first:
SELECT "item_item"."id", "item_item"."name"
FROM "item_item"
INNER JOIN "item_relation"
ON ("item_item"."id" = "item_relation"."item_id")
WHERE ("item_relation"."weight" > 3
AND NOT ("item_item"."id" IN
(SELECT U1."item_id"
FROM "item_relation" U1
WHERE (U1."weight" = 7 AND U1."item_id" IS NOT NULL))))
And the SQL for second:
SELECT "item_item"."id", "item_item"."name"
FROM "item_item"
INNER JOIN "item_relation"
ON ("item_item"."id" = "item_relation"."item_id")
INNER JOIN "item_relation" T3
ON ("item_item"."id" = T3."item_id")
WHERE ("item_relation"."weight" > 3 AND NOT (T3."weight" = 7 ))