I have several database tables with 2 primary keys, id and date. I do not update the records but instead insert a new record with the updated information. This new record has the same id and the date field is NOW(). I will use a product table to explain my question.
I want to be able to request the product details at a specific date. I therefore use the following subquery in DQL, which works fine:
WHERE p.date = (
SELECT MAX(pp.date)
FROM Entity\Product pp
WHERE pp.id = p.id
AND pp.date < :date
)
This product table has some referenced tables, like category. This category table has the same id and date primary key combination. I want to be able to request the product details and the category details at a specific date. I therefore expanded the DQL as shown above to the following, which also works fine:
JOIN p.category c
WHERE p.date = (
SELECT MAX(pp.date)
FROM Entity\Product pp
WHERE pp.id = p.id
AND pp.date < :date
)
AND c.date = (
SELECT MAX(cc.date)
FROM Entity\ProductCategory cc
WHERE cc.id = c.id
AND cc.date < :date
)
However, as you can see, if I have multiple referenced tables I will have to copy the same piece of DQL. I want to somehow add these subqueries to the entities so that every time an entity is called it adds this subquery.
I have thought of adding this in a __construct($date) or some kind of setUp($date) method, but I'm kind of stuck here. Also, would it help to add #Id to Entity\Product::date?
I hope someone can help me. I do not expect a complete solution, one step in a good direction would be very much appreciated.
I think I've found my solution. The trick was (first, to update to Doctrine 2.2 and) using a filter:
namespace Filter;
use Doctrine\ORM\Mapping\ClassMetaData,
Doctrine\ORM\Query\Filter\SQLFilter;
class VersionFilter extends SQLFilter {
public function addFilterConstraint(ClassMetadata $targetEntity, $targetTableAlias) {
$return = $targetTableAlias . '.date = (
SELECT MAX(sub.date)
FROM ' . $targetEntity->table['name'] . ' sub
WHERE sub.id = ' . $targetTableAlias . '.id
AND sub.date < ' . $this->getParameter('date') . '
)';
return $return;
}
}
Add the filter to the configuration:
$configuration->addFilter("version", Filter\VersionFilter");
And enable it in my repository:
$this->_em->getFilters()->enable("version")->setParameter('date', $date);
Related
I have some One-To-Many, unidirectional with Join Table relationships in a Symfony App which I need to query and I can't figure out how to do that in DQL or Query Builder.
The Like entity doesn't have a comments property itself because it can be owned by a lot of different types of entities.
Basically I would need to translate something like this:
SELECT likes
FROM AppBundle:Standard\Like likes
INNER JOIN comment_like ON comment_like.like_id = likes.id
INNER JOIN comments ON comment_like.comment_id = comments.id
WHERE likes.created_by = :user_id
AND likes.active = 1
AND comments.id = :comment_id
I've already tried this but the join output is incorrect, it selects any active Like regardless of its association with the given comment
$this->createQueryBuilder('l')
->select('l')
->innerJoin('AppBundle:Standard\Comment', 'c')
->where('l.owner = :user')
->andWhere('c = :comment')
->andWhere('l.active = 1')
->setParameter('user', $user)
->setParameter('comment', $comment)
I see 2 options to resolve this:
Make relation bi-directional
Use SQL (native query) + ResultSetMapping.
For the last option, here is example of repository method (just checked that it works):
public function getLikes(Comment $comment, $user)
{
$sql = '
SELECT l.id, l.active, l.owner
FROM `like` l
INNER JOIN comment_like ON l.id = comment_like.like_id
WHERE comment_like.comment_id = :comment_id
AND l.active = 1
AND l.owner = :user_id
';
$rsm = new \Doctrine\ORM\Query\ResultSetMappingBuilder($this->_em);
$rsm->addRootEntityFromClassMetadata(Like::class, 'l');
return $this->_em->createNativeQuery($sql, $rsm)
->setParameter('comment_id', $comment->getId())
->setParameter('user_id', $user)
->getResult();
}
PS: In case of Mysql, 'like' is reserved word. So, if one wants to have table with name 'like' - just surround name with backticks on definition:
* #ORM\Table(name="`like`")
I find the Symfony documentation very poor about unidirectional queries.
Anyway I got it working by using DQL and sub-select on the owning entity, which is certainly not as fast. Any suggestion on how to improve that is more than welcomed!
$em = $this->getEntityManager();
$query = $em->createQuery('
SELECT l
FROM AppBundle:Standard\Like l
WHERE l.id IN (
SELECT l2.id
FROM AppBundle:Standard\Comment c
JOIN c.likes l2
WHERE c = :comment
AND l2.owner = :user
AND l2.active = 1
)'
)
->setParameter('user', $user)
->setParameter('comment', $comment)
;
Django is to making a query much more complicated than it needs to be.
A Sentiment may have a User and a Card, and I am getting the Cards which are not in the passed User's Sentiments
This is the query:
Card.objects.all().exclude(sentiments__in=user.sentiments.all())
this is what Django runs:
SELECT * FROM "cards_card"
WHERE NOT ("cards_card"."id" IN (
SELECT V1."card_id" AS "card_id"
FROM "sentiments_sentiment" V1
WHERE V1."id" IN (
SELECT U0."id"
FROM "sentiments_sentiment" U0
WHERE U0."user_id" = 1
)
)
)
This is a version I came up with which didn't do an N-Times full table scan:
Card.objects.raw('
SELECT DISTINCT "id"
FROM "cards_card"
WHERE NOT "id" IN (
SELECT "card_id"
FROM "sentiments_sentiment"
WHERE "user_id" = ' + user_id + '
)
)')
I don't know why Django has to do it with the N-Times scan. I've been scouring the web for answers, but nothing so far. Any suggestions on how to keep the performance but not have to fall back to raw SQL?
A better way of writing this query without the subqueries would be:
Card.objects.all().exclude(sentiments__user__id=user.id)
I have an issue with DQL in Doctrine 2.
Subqueries seem to be unavailable in DQL, so I don't know how to transform :
SELECT DISTINCT a.ID_DOMAINE, L_DOMAINE, b.ID_SS_DOMAINE, L_SS_DOMAINE, c.ID_COMPETENCE, L_COMPETENCE
FROM ((qfq_prod.REF_DOMAINE a inner join qfq_prod.REF_SS_DOMAINE b on a.id_domaine = b.id_domaine)
inner join qfq_prod.REF_COMPETENCE c on b.id_ss_domaine = c.id_ss_domaine)
inner join qfq_prod.REF_PERS_COMP d on c.id_competence = d.id_competence
into a DQL expression.
I tried it and got
"Error: Class '(' is not defined."
I saw that we can use Query Builder to do this as well.
Being new with Doctrine 2, can someone explain to me how I can do this please ?
My DQL is currently :
$query = $this->getEntityManager()->createQuery ( "SELECT DISTINCT a.ID_DOMAINE, L_DOMAINE, b.ID_SS_DOMAINE, L_SS_DOMAINE, c.ID_COMPETENCE, L_COMPETENCE
FROM ((BdDoctrine\Entity\Domaine a inner join BdDoctrine\Entity\SsDomaine b on a.id_domaine = b.id_domaine)
inner join BdDoctrine\Entity\Competence c on b.id_ss_domaine = c.id_ss_domaine)
inner join BdDoctrine\Entity\LienPersComp d on c.id_competence = d.id_competence" );
$res = $query->getResult ();
Subqueries seem to be unavailable in DQL, so I don't know how to transform :
Actually, they are. Your code (no offence) is hardly readable so I will give you an example:
//controller
$repo = $this->getDoctrine()->getRepository("Your:Bundle:Category") ;
$results = $repo->findAllForSomePage() ;
// CategoryRepository.php
public function findAllForSomePage()
{
return $this->createQueryBuilder("o")
->innerJoin("o.products", "p", "WITH", "p.price>:price")->addSelect("p")
->setParameter("price", 50)
->where("o.id IN (SELECT s1.id FROM Your:Bundle:Something s1 WHERE s1.col1=5)")
->getQuery()->getResult() ;
}
Here is presumed you have Category hasMany Products relation and that you defined CategoryRepository file. You should never create queries in controller.
This example will fetch Categories only if they have Products with price bigger than 50, AND the ID of categories are those fetched by fictional subquery. This 100% works.
You should apply the same logic on your requirement.
Also, you should not use ON statement when using joins, that is handled by doctrine.
If you have the relationships properly defined in your entities, then you can make your joins on those relationships. And as Zeljko mentioned, you don't need to specify the ON condition, as the entities should already know how they are related. You are joining entities not tables. (That's under the hood.)
I don't know what your entities look like, so I made a guess at the relationship names below, but it should give you the idea.
$dql =
<<<DQL
SELECT
DISTINCT a.ID_DOMAINE, b.L_DOMAINE, b.ID_SS_DOMAINE, b.L_SS_DOMAINE, c.ID_COMPETENCE, c.L_COMPETENCE
FROM
BdDoctrine\Entity\Domaine a
JOIN a.ss_domaine b
JOIN b.competence c
JOIN c.lien_pers_comp d
DQL;
$query = $this->getEntityManager()->createQuery($dql);
$res = $query->getResult();
I want to get a list of the latest purchase of each customer, sorted by the date.
The following query does what I want except for the date:
(Purchase.objects
.all()
.distinct('customer')
.order_by('customer', '-date'))
It produces a query like:
SELECT DISTINCT ON
"shop_purchase.customer_id"
"shop_purchase.id"
"shop_purchase.date"
FROM "shop_purchase"
ORDER BY "shop_purchase.customer_id" ASC,
"shop_purchase.date" DESC;
I am forced to use customer_id as the first ORDER BY expression because of DISTINCT ON.
I want to sort by the date, so what the query I really need should look like this:
SELECT * FROM (
SELECT DISTINCT ON
"shop_purchase.customer_id"
"shop_purchase.id"
"shop_purchase.date"
FROM "shop_purchase"
ORDER BY "shop_purchase.customer_id" ASC,
"shop_purchase.date" DESC;
)
AS result
ORDER BY date DESC;
I don't want to sort using python because I still got to page limit the query. There can be tens of thousands of rows in the database.
In fact it is currently sorted by in python now and is causing very long page load times, so that's why I'm trying to fix this.
Basically I want something like this https://stackoverflow.com/a/9796104/242969. Is it possible to express it with django querysets instead of writing raw SQL?
The actual models and methods are several pages long, but here is the set of models required for the queryset above.
class Customer(models.Model):
user = models.OneToOneField(User)
class Purchase(models.Model):
customer = models.ForeignKey(Customer)
date = models.DateField(auto_now_add=True)
item = models.CharField(max_length=255)
If I have data like:
Customer A -
Purchase(item=Chair, date=January),
Purchase(item=Table, date=February)
Customer B -
Purchase(item=Speakers, date=January),
Purchase(item=Monitor, date=May)
Customer C -
Purchase(item=Laptop, date=March),
Purchase(item=Printer, date=April)
I want to be able to extract the following:
Purchase(item=Monitor, date=May)
Purchase(item=Printer, date=April)
Purchase(item=Table, date=February)
There is at most one purchase in the list per customer. The purchase is each customer's latest. It is sorted by latest date.
This query will be able to extract that:
SELECT * FROM (
SELECT DISTINCT ON
"shop_purchase.customer_id"
"shop_purchase.id"
"shop_purchase.date"
FROM "shop_purchase"
ORDER BY "shop_purchase.customer_id" ASC,
"shop_purchase.date" DESC;
)
AS result
ORDER BY date DESC;
I'm trying to find a way not to have to use raw SQL to achieve this result.
This may not be exactly what you're looking for, but it might get you closer. Take a look at Django's annotate.
Here is an example of something that may help:
from django.db.models import Max
Customer.objects.all().annotate(most_recent_purchase=Max('purchase__date'))
This will give you a list of your customer models each one of which will have a new attribute called "most_recent_purchase" and will contain the date on which they made their last purchase. The sql produced looks like this:
SELECT "demo_customer"."id",
"demo_customer"."user_id",
MAX("demo_purchase"."date") AS "most_recent_purchase"
FROM "demo_customer"
LEFT OUTER JOIN "demo_purchase" ON ("demo_customer"."id" = "demo_purchase"."customer_id")
GROUP BY "demo_customer"."id",
"demo_customer"."user_id"
Another option, would be adding a property to your customer model that would look something like this:
#property
def latest_purchase(self):
return self.purchase_set.order_by('-date')[0]
You would obviously need to handle the case where there aren't any purchases in this property, and this would potentially not perform very well (since you would be running one query for each customer to get their latest purchase).
I've used both of these techniques in the past and they've both worked fine in different situations. I hope this helps. Best of luck!
Whenever there is a difficult query to write using Django ORM, I first try the query in psql(or whatever client you use). The SQL that you want is not this:
SELECT * FROM (
SELECT DISTINCT ON
"shop_purchase.customer_id" "shop_purchase.id" "shop_purchase.date"
FROM "shop_purchase"
ORDER BY "shop_purchase.customer_id" ASC, "shop_purchase.date" DESC;
) AS result
ORDER BY date DESC;
In the above SQL, the inner SQL is looking for distinct on a combination of (customer_id, id, and date) and since id will be unique for all, you will get all records from the table. I am assuming id is the primary key as per convention.
If you need to find the last purchase of every customer, you need to do something like:
SELECT "shop_purchase.customer_id", max("shop_purchase.date")
FROM shop_purchase
GROUP BY 1
But the problem with the above query is that it will give you only the customer name and date. Using that will not help you in finding the records when you use these results in a subquery.
To use IN you need a list of unique parameters to identify a record, e.g., id
If in your records id is a serial key, then you can leverage the fact that the latest date will be the maximum id as well. So your SQL becomes:
SELECT max("shop_purchase.id")
FROM shop_purchase
GROUP BY "shop_purchase.customer_id";
Note that I kept only one field (id) in the selected clause to use it in a subquery using IN.
The complete SQL will now be:
SELECT *
FROM shop_customer
WHERE "shop_customer.id" IN
(SELECT max("shop_purchase.id")
FROM shop_purchase
GROUP BY "shop_purchase.customer_id");
and using the Django ORM it looks like:
(Purchase.objects.filter(
id__in=Purchase.objects
.values('customer_id')
.annotate(latest=Max('id'))
.values_list('latest', flat=True)))
Hope it helps!
I have a similar situation and this is how I'm planning to go about it:
query = Purchase.objects.distinct('customer').order_by('customer').query
query = 'SELECT * FROM ({}) AS result ORDER BY sent DESC'.format(query)
return Purchase.objects.raw(query)
Upside it gives me the query I want. Downside is that it is raw query and I can't append any other queryset filters.
This is my approach if I need some subset of data (N items) along with the Django query. This is example using PostgreSQL and handy json_build_object() function (Postgres 9.4+), but same way you can use other aggregate function in other database system. For older PostgreSQL versions you can use combination of array_agg() and array_to_string() functions.
Imagine you have Article and Comment models and along with every article in the list you want to select 3 recent comments (change LIMIT 3 to adjust size of subset or ORDER BY c.id DESC to change sorting of subset).
qs = Article.objects.all()
qs = qs.extra(select = {
'recent_comments': """
SELECT
json_build_object('comments',
array_agg(
json_build_object('id', id, 'user_id', user_id, 'body', body)
)
)
FROM (
SELECT
c.id,
c.user_id,
c.body
FROM app_comment c
WHERE c.article_id = app_article.id
ORDER BY c.id DESC
LIMIT 3
) sub
"""
})
for article in qs:
print(article.recent_comments)
# Output:
# {u'comments': [{u'user_id': 1, u'id': 3, u'body': u'foo'}, {u'user_id': 1, u'id': 2, u'body': u'bar'}, {u'user_id': 1, u'id': 1, u'body': u'joe'}]}
# ....
I have a trouble with creating JPA 2 query (actually I'm using Spring Data JPA). I have a 2 tables:
- Users (id,company_id,name,surname,status)
- UsersGroup (id,user_id,group_id).
Users can(and often are) connected to more than one group.
What I'm trying to do is to get distinct list of users according to filtering options from search form (name,surname,status) and attributes taken from logged user (company_id, group_id), because I want to show this user only users within company and group he also belongs to.
I'm using Specification/Predicate/Criteria from JPA 2 and Spring Data to do this, and I wrote this code:
public static Specification<User> filteredList(final ListFilterBean searchFilter, final LoggedUserBean loggedUser) {
return new Specification<User>() {
#Override
public Predicate toPredicate(Root<User> root,
CriteriaQuery<?> query, CriteriaBuilder cb) {
List<Predicate> predicates = new ArrayList<Predicate>();
root.join("userGroups");
predicates.add(cb.equal(root.<Long>get(User_.companyId), loggedUser.getCompanyId()));
predicates.add(cb.like(cb.upper(root.<String>get(User_.name)), getLikePattern(searchFilter.getName())));
predicates.add(cb.like(cb.upper(root.<String>get(User_.surname)), getLikePattern(searchFilter.getSurname())));
predicates.add(cb.equal(cb.upper(root.<String>get(User_.status)), searchFilter.getStatusId().charAt(0)));
query.distinct(true);
return cb.and(predicates.toArray(new Predicate[0]));
}
};
}
Which produces SQL below:
select
*
from
( select
distinct user0_.id as id5_,
user0_.name as name5_,
user0_.surname as surname5_
from
users user0_
inner join
user_group usergroup1_
on user0_.id=usergroup1_.user_id
where
user0_.company_id=1
and (
user0_.id in (select distinct user_id from user_group where user_group.group_id in (1, 2, 3))
)
and (
upper(user0_.name) like '%'
)
and (
upper(user0_.surname) like '%'
)
and upper(user0_.status)='A' )
where
rownum <= 15
Bolded part of code is the part doesn't actually exist and this is my main question. How to write expression (Predicate) to incorporate this part of SQL into query?
Thanks in advance :)