In my model I have a "hybrid method" for getting derived attributes,
class PersonT(Base):
#...
#hybrid_method
def firstName_lastName(self):
return func.upper(func.concat( self.get_first_name(self), ' ', self.get_last_name(self)))
I need to concisely fetch a full PersonT row with this derived attribute. By default it doesn't come back if I do this:
p = db_session.query(PersonT).filter(PersonT.id == id).first()
The derived attribute would only come back if I were to spell out the specific columns I want,
p = (db_session.query(
PersonT.id,
#... list out all the columns ...
NedPersonT.firstName_lastName.label('firstName_lastName'),
I want to quickly select the full row using the concise (first) syntax, without listing out specific columns, because I have hundreds of them, and then add this derived attribute as well. Is there a way to append it to the concise query(PersonT)?
Related
I'm making some computations using values & annotate on queryset. Let's consider this model:
class Foo(models.Model):
fk_bar = models.ForeignKey(to=Bar, ....)
foo_val = models.IntegerField(...)
class Bar(models.Model):
attr = models.Charfield(...)
val = models.IntegerField(...)
So I can do:
Foo.object.all().values("fk_bar")
in order to group by the foreign relationship (some Foo might point to the same Bar).
Then I can do
Foo.object.all().values("fk_bar").annotate(qte=Sum("foo_val"))
To the the sum of foo_val for all the object with the same fk_bar, which yields something like:
{"fk_bar":<int>, "qte": <int>}
However I want the resulting dictionnary to calso contain Bar.attr, e.g. something like:
Foo.object.all().values("fk_bar").annotate(qte=Sum("foo_val")).annotate(bar_attr="fk_bar__attr")
To get something like:
{"fk_bar":<int>, "qte": <int>, "bar_attr":<str>}
However that fails (TypeError: Queryset.annotate() received a non-expression). Any ways to go around this?
One option is to specify the additional values to "keep" from before the aggregation in the values() clause, like so:
Foo.object.all().values("fk_bar", "fk_bar__attr").annotate(qte=Sum("foo_val"))
Which will yield roughly what's asked.
In general, if one puts more fields in the .values() clause, those fields will be available the in the output dictionary. Any "additional" fields (e.g. not from the raw queryset data) that needs to be computed beforehand (e.g. before aggregation) can be "created" with .annotation() (like qte=Sum(...) above), and then included in the subsequent .values("fk_bar", "qte", ...). A few examples:
Yields the sum for all the Foos with the same fk_bar:
Foo.object.all().values("fk_bar").annotate(qte=Sum("foo_val"))
This will group all the Foos by fk_bar, by foo_val. Then for each group, will sum up the values of the fk_bar__val field:
Foo.object.all().values("fk_bar", "foo_val").annotate(qte=Sum("fk_bar__val"))
The thing to note is that if many fields are specified in .values(), then the aggregation will be larger (e.g. 1 dictionnary for each unique combination of values).
I have a model that has (among other fields) one value for the current price of an item and one value for the usual price of an item. I'd like to include a field for the percentage saving. I've done this using the #property:
#property
def percSaving(self):
aval = self.stickerprice
bval = self.currentprice
if self.stickerprice > 0:
return "%0.2f" % ((aval-bval)/aval*100) + "%"
elif self.currentprice == 0:
return "Always Free"
else:
return "100% OFF"
This works, I can add this column to my django_table2 table with:
percSaving = tables.Column(verbose_name='% Saving')
Super easy and all good. However, I am unable to sort by this column. This is because it's not one of the columns of data from the query set. So I've been trying to annotate the query set to allow for this ordering I've based by annotation attempt on this queryset api reference page and have produced this:
annoed = products.objects.annotate(percSaving=((stickerprice)-(currentprice))/(stickerprice))
However this gives me an error of "name 'stickerprice' is not defined" which I thought might be because of not using inverted commas around the field names, but I tried that and got an error saying "unsupported operand type(s) for -: 'str' and 'str'" - basically using the inverted commas forces it to view the field names as strings.
What am I doing wrong? How can I annotate a query set to allow for ordering by a column I have defined as above!
Yes, you can. With providing some more code like Model etc. I can only make some general advise.
You should use F function to calculate in the query like:
annoed = products.objects.annotate(
percSaving=((F("stickerprice") - F("currentprice")) / F("stickerprice"))
)
Check also How to make sum query with type casting and calculation in django views? which provides also some type casting hints.
I am using "doctrine/doctrine-orm-module": "^2.1" (it is a module for zend framework 3). I want to create a query which will return rows with field names (trivial, right?). But instead of exact names of fields I am getting this query result:
SELECT
u0_.id AS id_0, u0_.username AS username_1, u0_.email AS email_2,
u0_.first_name AS first_name_3, u0_.last_name AS last_name_4,
u0_.password AS password_5, u0_.status AS status_6, u0_.created AS created_7,
u0_.modified AS modified_8
FROM
user_item u0_
ORDER BY
u0_.id DESC
This query is generated by this code:
$entityManager = $this->getEntityManager();
$queryBuilder = $entityManager->createQueryBuilder();
$queryBuilder->select('u')
->from(UserItem::class, 'u')
->orderBy('u.id', 'DESC')
;
$query = $queryBuilder->getQuery();
echo $query->getSql();
print_r($query->getParameters());
die('|||');
What is the "0_" appending to the table name? What is appendings "_x" to the fields name?
How can I get normal fields and tables names without appended "_x"?
Just names, I'm assuming both the first_name and last_name as shown in that generated SQL, right?
I changed the order below, makes it easier to read / understand.
What you want to do is (pseudo code): Select from UserItem all the first & last names
So, write the code that way :)
$queryBuilder
->from(UserItem::class, 'u')
->select(['u.first_name', 'u.last_name'])
->orderBy('u.id', 'DESC'); // Might want to sort by either u.first_name or u.last_name
What's in the QueryBuilder?
->from(UserItem::class, 'u') - First parameter is the FQCN (Fully Qualified Class Name) of the Entity you wish to use with the QueryBuilder. Not required is the second parameter, which is an alias to use for this instance of the QueryBuilder to recognize the FQCN defined class by. (Off of the top of my head it defaults to snake_case'd names of the class, in this case "user_item")
->select(['u.first_name', 'u.last_name']) - Function takes a "mixed" param. Click through to its definition and you'll see the following in the function:
$selects = is_array($select) ? $select : func_get_args();
Which indicates that it will always pass the "$selects" on the next bit as an array. (Another hint is that $selects is plural)
->orderBy('u.id', 'DESC') - Creates a rule to order results by. If you click through to this function, you'll see that this one ends like so:
return $this->add('orderBy', $orderBy);
Meaning: you can add more than 1 order by.
When it comes to the generated DQL:
u0_ is the table alias as defined in the DQL, from your question: FROM user_item u0_, this will later be transformed to MySQL (usually) which will be the same. It sets u0_ as an alias for user_item.
The _* appended to property names is just plain the order of the columns as they've been created in the database (have a look, they'll be in that order).
Lastly, the fact you were receiving entire entities and not just the names (first_name & last_name) is due to ->select('u'). Because no property (or properties as shown above) is defined, Doctrine assumes you wish to receive the whole enchalada. Doing ->select('u.first_name') would then get you just the first names, and using an array as above would get you more than 1 property.
Hope that helped you out :)
Usual sqlalchemy usage:
my_prop = Column("my_prop", Text)
I would like different semantics. Let's say an object has a set of fields (propA, propB, propC). I would like to maintain a database column which is derived from these fields (let's say, propA + propB + propC). I would like the column to be updated whenever any one of these set of fields is updated. Thank you.
Hybrid properties provide the functionality you are looking for. They allow you to write python properties that are usable in queries.
Here's how you might start if you wanted to have a name column and provide access to first and last name properties.
#hybrid_property
def first_name(self):
# get the first name from the name column
#first_name.setter
def first_name(self, value):
# update the name column with the first name replaced
#first_name.expression
def first_name(cls):
# return a sql expression that extracts the first name from the name column
# this is appropriate to be used in queries
Say I have a model:
class Foo(models.Model):
...
and another model that basically gives per-user information about Foo:
class UserFoo(models.Model):
user = models.ForeignKey(User)
foo = models.ForeignKey(Foo)
...
class Meta:
unique_together = ("user", "foo")
I'd like to generate a queryset of Foos but annotated with the (optional) related UserFoo based on user=request.user.
So it's effectively a LEFT OUTER JOIN on (foo.id = userfoo.foo_id AND userfoo.user_id = ...)
A solution with raw might look like
foos = Foo.objects.raw("SELECT foo.* FROM foo LEFT OUTER JOIN userfoo ON (foo.id = userfoo.foo_id AND foo.user_id = %s)", [request.user.id])
You'll need to modify the SELECT to include extra fields from userfoo which will be annotated to the resulting Foo instances in the queryset.
This answer might not be exactly what you are looking for but since its the first result in google when searching for "django annotate outer join" so I will post it here.
Note: tested on Djang 1.7
Suppose you have the following models
class User(models.Model):
name = models.CharField()
class EarnedPoints(models.Model):
points = models.PositiveIntegerField()
user = models.ForeignKey(User)
To get total user points you might do something like that
User.objects.annotate(points=Sum("earned_points__points"))
this will work but it will not return users who have no points, here we need outer join without any direct hacks or raw sql
You can achieve that by doing this
users_with_points = User.objects.annotate(points=Sum("earned_points__points"))
result = users_with_points | User.objects.exclude(pk__in=users_with_points)
This will be translated into OUTER LEFT JOIN and all users will be returned. users who has no points will have None value in their points attribute.
Hope that helps
Notice: This method does not work in Django 1.6+. As explained in tcarobruce's comment below, the promote argument was removed as part of ticket #19849: ORM Cleanup.
Django doesn't provide an entirely built-in way to do this, but it's not neccessary to construct an entirely raw query. (This method doesn't work for selecting * from UserFoo, so I'm using .comment as an example field to include from UserFoo.)
The QuerySet.extra() method allows us to add terms to the SELECT and WHERE clauses of our query. We use this to include the fields from UserFoo table in our results, and limit our UserFoo matches to the current user.
results = Foo.objects.extra(
select={"user_comment": "UserFoo.comment"},
where=["(UserFoo.user_id IS NULL OR UserFoo.user_id = %s)"],
params=[request.user.id]
)
This query still needs the UserFoo table. It would be possible to use .extras(tables=...) to get an implicit INNER JOIN, but for an OUTER JOIN we need to modify the internal query object ourself.
connection = (
UserFoo._meta.db_table, User._meta.db_table, # JOIN these tables
"user_id", "id", # on these fields
)
results.query.join( # modify the query
connection, # with this table connection
promote=True, # as LEFT OUTER JOIN
)
We can now evaluate the results. Each instance will have a .user_comment property containing the value from UserFoo, or None if it doesn't exist.
print results[0].user_comment
(Credit to this blog post by Colin Copeland for showing me how to do OUTER JOINs.)
I stumbled upon this problem I was unable to solve without resorting to raw SQL, but I did not want to rewrite the entire query.
Following is a description on how you can augment a queryset with an external raw sql, without having to care about the actual query that generates the queryset.
Here's a typical scenario: You have a reddit like site with a LinkPost model and a UserPostVote mode, like this:
class LinkPost(models.Model):
some fields....
class UserPostVote(models.Model):
user = models.ForeignKey(User,related_name="post_votes")
post = models.ForeignKey(LinkPost,related_name="user_votes")
value = models.IntegerField(null=False, default=0)
where the userpostvote table collect's the votes of users on posts.
Now you're trying to display the front page for a user with a pagination app, but you want the arrows to be red for posts the user has voted on.
First you get the posts for the page:
post_list = LinkPost.objects.all()
paginator = Paginator(post_list,25)
posts_page = paginator.page(request.GET.get('page'))
so now you have a QuerySet posts_page generated by the django paginator that selects the posts to display. How do we now add the annotation of the user's vote on each post before rendering it in a template?
Here's where it get's tricky and I was unable to find a clean ORM solution. select_related won't allow you to only get votes corresponding to the logged in user and looping over the posts would do bunch queries instead of one and doing it all raw mean's we can't use the queryset from the pagination app.
So here's how I do it:
q1 = posts_page.object_list.query # The query object of the queryset
q1_alias = q1.get_initial_alias() # This forces the query object to generate it's sql
(q1str, q1param) = q1.sql_with_params() #This gets the sql for the query along with
#parameters, which are none in this example
we now have the query for the queryset, and just wrap it, alias and left outer join to it:
q2_augment = "SELECT B.value as uservote, A.*
from ("+q1str+") A LEFT OUTER JOIN reddit_userpostvote B
ON A.id = B.post_id AND B.user_id = %s"
q2param = (request.user.id,)
posts_augmented = LinkPost.objects.raw(q2_augment,q1param+q2param)
voila! Now we can access post.uservote for a post in the augmented queryset.
And we just hit the database with a single query.
The two queries you suggest are as good as you're going to get (without using raw()), this type of query isn't representable in the ORM at present time.
You could do this using simonw's django-queryset-transform to avoid hard-coding a raw SQL query - the code would look something like this:
def userfoo_retriever(qs):
userfoos = dict((i.pk, i) for i in UserFoo.objects.filter(foo__in=qs))
for i in qs:
i.userfoo = userfoos.get(i.pk, None)
for foo in Foo.objects.filter(…).tranform(userfoo_retriever):
print foo.userfoo
This approach has been quite successful for this need and to efficiently retrieve M2M values; your query count won't be quite as low but on certain databases (cough MySQL cough) doing two simpler queries can often be faster than one with complex JOINs and many of the cases where I've most needed it had additional complexity which would have been even harder to hack into an ORM expression.
As for outerjoins:
Once you have a queryset qs from foo that includes a reference to columns from userfoo, you can promote the inner join to an outer join with
qs.query.promote_joins(["userfoo"])
You shouldn't have to resort to extra or raw for this.
The following should work.
Foo.objects.filter(
Q(userfoo_set__user=request.user) |
Q(userfoo_set=None) # This forces the use of LOUTER JOIN.
).annotate(
comment=F('userfoo_set__comment'),
# ... annotate all the fields you'd like to see added here.
)
The only way I see to do this without using raw etc. is something like this:
Foo.objects.filter(
Q(userfoo_set__isnull=True)|Q(userfoo_set__isnull=False)
).annotate(bar=Case(
When(userfoo_set__user_id=request.user, then='userfoo_set__bar')
))
The double Q trick ensures that you get your left outer join.
Unfortunately you can't set your request.user condition in the filter() since it may filter out successful joins on UserFoo instances with the wrong user, hence filtering out rows of Foo that you wanted to keep (which is why you ideally want the condition in the ON join clause instead of in the WHERE clause).
Because you can't filter out the rows that have an unwanted user value, you have to select rows from UserFoo with a CASE.
Note also that one Foo may join to many UserFoo records, so you may want to consider some way to retrieve distinct Foos from the output.
maparent's comment put me on the right way:
from django.db.models.sql.datastructures import Join
for alias in qs.query.alias_map.values():
if isinstance(alias, Join):
alias.nullable = True
qs.query.promote_joins(qs.query.tables)