I have a problem with Doctrine (Symfony2.1). I want to prevent lazy-loading by join fetching subentities (OneToMany relation) but I got only one result for those subentities.
For example:
public function getSingleProjectQuery($project){
$query = $this->createQueryBuilder('p')
->select(array("p", "fb"))
->where('p.id = :project_id')->setParameter('project_id', $project)
->leftJoin('p.feedbacks', 'fb')
->groupBy('p.id')
->getQuery();
return $query;
}
In this example Doctrine returns me the "Project"-object and one single "feedback" object (but there are more than one feedbacks...).
When I replace the select to this: ->select(array("p"))
I got all the "Feedback" objects but there are then lazy-loaded (many queries).
see http://docs.doctrine-project.org/en/latest/reference/dql-doctrine-query-language.html#joins
You should remove the groupBy clause.
Related
I'm trying to use select_related/prefetch_related to optimize some queries. However I have issues in "forcing" the queries to be evaluated all at once.
Say I'm doing the following:
fp_query = Fourprod.objects.filter(choisi=True).select_related("fk_fournis")
pf = Prefetch("fourprod", queryset=fp_query) #
products = Products.objects.filter(id__in=fp_query).prefetch_related(pf)
With models:
class Fourprod(models.Model):
fk_produit = models.ForeignKey(to=Produit, related_name="fourprod")
fk_fournis = models.ForeignKey(to=Fournis,related_name="fourprod")
choisi = models.BooleanField(...)
class Produit(models.Model):
... ordinary fields...
class Fournis(models.Model):
... ordinary fields...
So essentially, Fourprod has a fk to Fournis, Produit, and I want to prefetch those when I build the Produits queryset. I've checked in debug that the prefetch actually occurs and it does.
I have a bunch of fields from different models I need to use to compute results. I don't really control the table structure, so I have to work with this. I can't come up with a reasonable query to do it all with the queries (or using raw), so I want to compute stuff python-side. It's a few 1000 objects, so reasonable to do in-memory. So I cast to a list to force the query evaluation:
products = list(products)
At this point, I would think that the Products and the related objects that I have pre-fetched should have been fetched from the DB. In the logs, just after the list() call, I get this:
02/08/22 15:21:08 DEBUG DEFAULT: (0.019) SELECT "products_fourprod"."id", "products_fourprod"."fk_produit_id", "products_fourprod"."fk_fournis_id", "products_fourprod"."choisi", "products_fourprod"."code_four", "products_fourprod"."prix", "products_fourprod"."comment", "products_fournis"."id", "products_fournis"."fk_user_create_id", "products_fournis"."nom", "products_fournis"."adresse", "products_fournis"."ville", "products_fournis"."tel", "products_fournis"."fax", "products_fournis"."contact", "products_fournis"."note", "products_fournis"."pays", "products_fournis"."province", "products_fournis"."postal", "products_fournis"."monnaie", "products_fournis"."tel_long", "products_fournis"."inactif", "products_fournis"."inuse", "products_fournis"."par", "products_fournis"."fk_langue", "products_fournis"."NOTE2" FROM "products_fourprod" LEFT OUTER JOIN "products_fournis" ON ("products_fourprod"."fk_fournis_id" = "products_fournis"."id") WHERE ("products_fourprod"."choisi" AND "products_fourprod"."fk_produit_id" IN (... all Product.id meeting the conditions...)
But then, the list comprehension using the products takes forever to complete:
rows = [[p.id, p.fourprod.first().id, p.desuet, p.no_prod, ... ] for p in products]
With apparently each single call to p.fourprod resulting in a DB hit:
02/08/22 15:26:19 DEBUG DEFAULT: (0.000) SELECT "products_fourprod"."id", "products_fourprod"."fk_produit_id", "products_fourprod"."fk_fournis_id", "products_fourprod"."choisi", "products_fourprod"."code_four", "products_fourprod"."prix", "products_fourprod"."comment", "products_fournis"."id", "products_fournis"."fk_user_create_id", "products_fournis"."nom", "products_fournis"."adresse", "products_fournis"."ville", "products_fournis"."tel", "products_fournis"."fax", "products_fournis"."contact", "products_fournis"."note", "products_fournis"."pays", "products_fournis"."province", "products_fournis"."postal", "products_fournis"."monnaie", "products_fournis"."tel_long", "products_fournis"."inactif", "products_fournis"."inuse", "products_fournis"."par", "products_fournis"."fk_langue", "products_fournis"."NOTE2" FROM "products_fourprod" LEFT OUTER JOIN "products_fournis" ON ("products_fourprod"."fk_fournis_id" = "products_fournis"."id") WHERE ("products_fourprod"."choisi" AND "products_fourprod"."fk_produit_id" = 1185) ORDER BY "products_fourprod"."id" ASC LIMIT 1; args=(1185,)
02/08/22 15:26:19 DEBUG DEFAULT: (0.000) SELECT "products_fourprod"."id", (.... more similar db hits... )
If I remove all the uses of related objects, then the list() call has actually forced the db hit already and the query executes quickly.
So.... if simply calling products = list(products) does not force the db to be queried for the prefetched objects as well, is there any ways I can make django's orm do so?
From the docs:
Remember that, as always with QuerySets, any subsequent chained methods which imply a different database query will ignore previously cached results, and retrieve data using a fresh database query.
first() implies a database query, so that will cause your query to not use the prefetched values.
Try to use p.fourprod.all()[0] instead to access the first related fourprod instead.
To optimize a lot my database I would like to make as less as possible any query.
I'm trying to get an object, increment the field "count_limit" and make an If statement after on the Customer instance.
To achieve it I've made this query who worked well.
Customer.objects.filter(user=user).update(count_limit=F('count_limit') + 1)
So after this query, count_limit has been incremented by 1 as I wanted.
When I'm trying to get the Customer instance as a result of this query, it returns "1".
Is it possible to make both, update the instance and get it as a return object ?
Thanks a lot
The update() method will return the number of updated rows. If you are using Postgres, then you can use the returning clause with the raw query.
query = 'UPDATE customer SET count_limit=(customer.count_limit + 1) WHERE customer.user_id=%s returning *'
updated_obj = Customer.objects.raw(query, [user.id])
I don't know if this can be achieved by ORM, but suggestions will be appreciated.
Make sure that the table name in raw query is correct. If you haven't definer db_table in the meta class of your model, then by default it will be myapp_model.
And to prevent SQL injection, from the Docs:
Do not use string formatting on raw queries or quote placeholders in
your SQL strings!
Follow Docs on raw()
You are looking for F functions: https://docs.djangoproject.com/en/3.0/ref/models/expressions/#f-expressions
Example from their documentation how to increase a counter
from django.db.models import F
reporter = Reporters.objects.get(name='Tintin')
reporter.stories_filed = F('stories_filed') + 1
reporter.save()
Django 1.10.6
Asset.objects.annotate(
coupon_saved=Count(
Q(coupons__device_id='8ae83c6fa52765061360f5459025cb85e6dc8905')
)
).all().query
produces the following query:
SELECT
"assets_asset"."id",
"assets_asset"."title",
"assets_asset"."description",
"assets_asset"."created",
"assets_asset"."modified",
"assets_asset"."uid",
"assets_asset"."org_id",
"assets_asset"."subtitle",
"assets_asset"."is_active",
"assets_asset"."is_generic",
"assets_asset"."file_standalone",
"assets_asset"."file_ios",
"assets_asset"."file_android",
"assets_asset"."file_preview",
"assets_asset"."json_metadata",
"assets_asset"."file_icon",
"assets_asset"."file_image",
"assets_asset"."video_mobile",
"assets_asset"."video_standalone",
"assets_asset"."file_coupon",
"assets_asset"."where_to_buy",
COUNT("games_coupon"."device_id" = 8ae83c6fa52765061360f5459025cb85e6dc8905) AS "coupon_saved"
FROM
"assets_asset"
LEFT OUTER JOIN
"games_coupon"
ON ("assets_asset"."id" = "games_coupon"."asset_id")
GROUP BY
"assets_asset"."id"
I need to get that device_id=X into LEFT OUTER JOIN definition below.
How to achieve?
TL;DR:
The condition should be in filter.
qs = (
Asset.objects
.filter(coupons__device_id='8ae83c6fa52765061360f5459025cb85e6dc8905')
.annotate(coupon_saved=Count('coupons'))
)
If you want only count > 0 then it can be filtered.
qs = qs.filter(coupon_saved__gt=0)
Footnotes: A one to many query is compiled to LEFT OUTER JOIN in order to be possible to get also base objects (Asset) with zero children. JOINs in Django are based every times on a ForeignKey to the primary key or similarly on OneToOne or ManyToMany, other conditions are compiled to WHERE.
Conditions in annotation (that you used) are possible e.g. as part of Conditional Expressions but it is more complicated to be used correctly and useful e.g. if you want to get many aggregations with many conditions by one query without subqueries and if a full scan is acceptable. This is probably not a subject of a question.
I'm researching whether to try Doctrine2 or not. One thing that scares me is the over SELECTing of columns I don't need (ie. consider lots of varchars being selected unnecessarily).
You might ask: but don't you want your full entity object filled? Yes, unless I'm looking for an array hydration. However, many times I don't need the full aggregation filled. Take the association shown below. If I query the Users table with a JOIN on Address, will all the columns from the address table be SELECTed as well (and therefore populated into an address object inside of users object)? Now imagine we have more JOINs. This could get really bad. What if I only want the fields from User populated in just a users only object? I guess I'm a little confused at what Doctrine is doing behind the scenes with associations and query JOINs.
/** #Entity **/
class User
{
// ...
/**
* #ManyToOne(targetEntity="Address")
* #JoinColumn(name="address_id", referencedColumnName="id")
**/
private $address;
}
/** #Entity **/
class Address
{
// ...
}
So does Doctrine2 populate all the fields of all the objects within the aggregate after a query (unless I specifiy partial)?
It depends on your query, but generally it is not implicit.
Using the query builder, you can fetch the associated record like this:
<?php
$qb = $em->createQueryBuilder();
$query = $qb->select(array("u", "a"))
->from("User", "u")
->innerJoin("u.address", "a")
->getQuery();
In the select() statement you specify what to fetch, in this case you get both.
If you only fetch the User records, then when you get the associated record with $user->getAddress(), Doctrine will make the query on the fly and hydrate the Address record for you.
That said, performance wise it is better to select both entities so Doctrine will make only one query and not 1+N queries
I'm having an issue converting some legacy SQL for the doctrine query builder. I think the problem is in the inner join, but I can't quite work out the parameters the builder is expecting.
This is what I have so far:
$qb = $em->createQueryBuilder();
$qb->select('ob.size', 'ob.colour', 'ob.productId', 'p.title')
->from('m:Option', 'ob')
->innerJoin('m:Product', 'p', 'ON', 'ob.ProductId');
And this is the original query:
query="select size,colour,product_id,title from
products,options_new where
picture = '' and
products.id = options_new.product_id and
product_id like 'UTRW%'
group by product_id";
I normally write joins explicitly, so I'm not certain I'm understanding how the from clause is working here.
At the moment the new query is generating this error:
Expected Doctrine\ORM\Query\Lexer::T_WITH, got 'ON'
Cheers!
Try the query below (drop here your entities, would be more helpful) and for more details about fetching related entities, check the docs: http://doctrine-orm.readthedocs.org/en/latest/reference/dql-doctrine-query-language.html#joins
$qb = $em->createQueryBuilder('m');
$qb->select('ob.size', 'ob.colour', 'ob.productId', 'p.title')
->innerJoin('m.product', 'ob');