I'm filtering a big number of users based on attributes. For example, they can be filtered by minimum gpa.
So far I've been doing the following to construct my queryset-
('gpa_gt' just means that the gpa property has to be greater than whatever is the query param)
if len(request.GET) != 0:
kwargs = {}
if request.GET['mingpa']:
kwargs['gpa__gt'] = request.GET['mingpa']
profiles = BlahUser.objects.filter(**kwargs)
Now, the users have a first school major attribute and a second school major attribute. If I filter by a major, I want the user to show up in the results if his first major OR his second major match any of my selections (I use a multiple select on the filtering). What I have now only works for one major-
if request.GET.has_key('major'):
kwargs['major__in'] = request.GET.getlist('major')
I know how to explicitly write out the query, but in my case I'm building it dynamically of sorts, and am stumped at how to do so. Any help appreciated, thanks!
You inclusively or in Django querysets with the Q object:
Poll.objects.get(
Q(pub_date=date(2005, 5, 2)) | Q(pub_date=date(2005, 5, 6)),
question__startswith='Who')
You can operate on them like variables so you would like something like this:
query = Q( gpa__gt=request.GET['mingpa'] )
if request.GET.has_key('major'):
query = query | Q( major__in=request.GET.getlist('major') )
profiles = BlahUser.objects.get( query )
Borgar is definitely on the right track, but I think this is a little closer to what you are
looking for:
# filter for minimum GPA
gpa_filter = Q( gpa__gt=request.GET['mingpa'] )
major_filter = Q()
if request.GET.has_key('major'):
majors = request.GET.getlist('major')
# filter for students with first OR second major in specified majors
major_filter = Q( first_major__in=majors| Q(second_major__in=majors)
# run filter (these are AND'ed together), so we get students that:
# 1. Have a first or second major in the specified list of majors AND
# 2. Have a GPA above the specified minimum
profiles = BlahUser.objects.get( gpa_filter, major_filter )
Note that you can continue to use profiles as a regular queryset (as in this contrived example):
seniors_matching_request_params = profiles.filter(status='senior')
So, just use Q objects where you need to, then continue to operate on the queryset as normal .
I'm thinking of this so far- (course being major)
if request.GET.has_key('course'):
extra = ['Q(first_course IN ' + request.GET.getlist('course') + ') | Q(second_course IN ' + request.GET.getlist('course') + ')']
profiles = sumaConnectUser.objects.extra(where=[extra])
'course' is what we call a major. I haven't checked whether the above works but it's a start..
Help? lol
Related
I have created two queries in django. One that returns number of handled calls per agent and a second one that returns the missed calls per agents.
I would like to know how to combine the two queries in order to get a percentage of handled calls par agent, which means take the total from the query3 and divide it on the total of query2 grouped by agent. Of course I will render the result to the web page,
Here are the two queries
Thanks
# A query that returns total calls that has been handled
queryset2 = CsqAgentReport.objects.values('agentname').filter(Q(originatordnhandeled__contains='1') | Q(missedcallshandeledy_n__contains='1')).exclude(agentname__contains='None').annotate(total=Count('nodeid_sessionid_sequenceno', distinct=True)).order_by('agentname')
# A query that returns total calls that has been missed
queryset3 = CsqAgentReport.objects.values('agentname').filter(originatordnnothandeled__contains='1').filter(ringtime__gt = '00:00:00').exclude(agentname__contains='None').annotate(total=Count('nodeid_sessionid_sequenceno', distinct=True)).order_by('agentname')
I already developed a script in Python that returns the expected result and I'm looking for something simple and similar in Django see code in Python here below:
print ("Nb of handled calls per agent" )
cursor.execute(""" SELECT "AgentName", count (*) FROM "CSQ Agent Report" WHERE "AgentName" != "None" AND "OriginatorDNHANDELED" = '1' or "OriginatorDNNOTHANDELED" = '1' Group by "AgentName" """)
liste8= cursor.fetchall()
for i in range (len(liste8)):
print (liste8[i][0],liste8[i][1])
print ('Nb of missed calls ')
cursor.execute(""" SELECT "AgentName", count (*) FROM "CSQ Agent Report" WHERE "AgentName" != "None" AND "TalkTime" > '00:00:00' AND "OriginatorDNHANDELED" = '1' OR "MISSEDCALLSHANDELEDY-N" = '1' Group by "AgentName" """)
liste7= cursor.fetchall()
for i in range (len(liste7)):
print (liste7[i][0],liste7[i][1])
print ('percantage of handled calls per agent ')
for i in range (len(liste8)):
x = liste8[i][1]
y = liste7[i][1]
d= y / x
print (liste8[i][0],'{:.0%}'.format(d))
You can work with:
from django.db.models import Count, Q
# since Django-2.0
CsqAgentReport.objects.exclude(
agentname__contains='None'
).values('agentname').filter(
Q(originatordnnothandeled__contains='1') | Q(missedcallshandeledy_n__contains='1')
).annotate(
total=Count(
'nodeid_sessionid_sequenceno',
distinct=True,
filter=Q(originatordnnothandeled__contains='1', ringtime__gt = '00:00:00')
) / Count('nodeid_sessionid_sequenceno', distinct=True)
)
But the modeling itself looks quite dangerous. You specify nearly everything as a TextField, even fields that contain quantitative data like a duration that is better stored as a DurationField [Django-doc], or booleans that should be stored with a BooleanField [Django-doc].
Filtering like agentname__contains='None' is not only inefficient, but also dangerous, since later an agent could have a name 'None'. It also results in queries with primitive obsession [refactoring.guru]. Normally one better uses a ForeignKey [Django-doc] to refer to an extra model. This means we can annotate the Agent model which is not only simpler, but also does not erode the model layer.
Given the following Django models:
class Room(models.Model):
name = models.CharField(max_length=20)
class Beacon(models.Model):
room = models.ForeignKey(Room)
uuid = models.UUIDField(default=uuid.uuid4)
major = models.PostiveIntegerField(max_value=65536)
minor = models.PositiveIntegerField(max_value=65536)
The Beacon model is a bluetooth beacon relationship to the room.
I want to select all Rooms that match a given uuid, major, minor combination.
The catch is, that I want to order the rooms by the beacon that is nearest to me. Because of this, I need to be able to assign a value to each beacon dynamically, and then sort by it.
Is this possible with the Django ORM? In Django 1.8?
NOTE - I will know the ordering of the beacons beforehand, I will be using the order they are passed in the query string. So the first beacon (uuid, major, minor) passed should match the first room that is returned by the Room QuerySet
I am envisioning something like this, though I know this won't work:
beacon_order = [
beacon1 = 1,
beacon0 = 2,
beacon3 = 3,
]
queryset = Room.objects.annotate(beacon_order=beacon_order).\
order_by('beacon_order')
If you already know the order of the beacons, there's no need to sort within the QuerySet itself. Take an ordered list called beacon_list, which contains the beacons' primary keys in order, e.g. the item at index 0 is the closest beacon's primary key, the item at index 1 is the second closest beacon's PK, etc. Then use a list comprehension:
ordered_rooms = [Room.objects.get(pk=x) for x in beacon_list]
You don't have to use the PK either, you can use anything which identifies the given object in the database, e.g. the name field.
Looks like this works:
from django.db.models import Case, Q, When
beacons = request.query_params.getlist('beacon[]')
query = Q()
order = []
for pos, beacon in enumerate(beacons):
uuid, major, minor = beacon.split(':')
query |= Q(
beacon__uuid=uuid,
beacon__major=major,
beacon__minor=minor,
)
order.append(When(
beacon__uuid=uuid,
beacon__major=major,
beacon__minor=minor,
then=pos,
))
rooms = Room.objects.filter(query).order_by(Case(*order))
What's an elegant way for fetching multiple objects in some custom order from a DB in django?
For example, suppose you have a few products, each with its name, and you want to fetch three of them to display in a row on your website page, in some fixed custom order. Suppose the names of the products which you want to display are, in order: ["Milk", "Chocolate", "Juice"]
One could do
unordered_products = Product.objects.filter(name__in=["Milk", "Chocolate", "Juice"])
products = [
unordered_products.filter(name="Milk")[0],
unordered_products.filter(name="Chocolate")[0],
unordered_products.filter(name="Juice")[0],
]
And the post-fetch ordering part could be improved to use a name-indexed dictionary instead:
ordered_product_names = ["Milk", "Chocolate", "Juice"]
products_by_name = dict((x.name, x) for x in unordered_products)
products = [products_by_name[name] for name in ordered_product_names]
But is there a more elegant way? e.g., convey the desired order to the DB layer somehow, or return the products grouped by their name (aggregation seems to be similar to what I want, but I want the actual objects, not statistics about them).
You can order your product by a custom order with only one query of your ORM (executing one SQL query only):
ordered_products = Product.objects.filter(
name__in=['Milk', 'Chocolate', 'Juice']
).annotate(
order=Case(
When(name='Milk', then=Value(0)),
When(name='Chocolate', then=Value(1)),
When(name='Juice', then=Value(2)),
output_field=IntegerField(),
)
).order_by('order')
Update
Note
Speaking about "elegant way" (and best practice) I think extra method (proposed by #Satendra) is absolutely to avoid.
Official Django documentation report this about extra :
Warning
You should be very careful whenever you use extra(). Every time you
use it, you should escape any parameters that the user can control by
using params in order to protect against SQL injection attacks .
Please read more about SQL injection protection.
Optimized version
If you want to handle more items whit only one query you can change my first query and use the Django ORM flexibility as suggested by #Shubhanshu in his answer:
products = ['Milk', 'Chocolate', 'Juice']
ordered_products = Product.objects.filter(
name__in=products
).order_by(Case(
*[When(name=n, then=i) for i, n in enumerate(products)],
output_field=IntegerField(),
))
The output of this command will be similar to this:
<QuerySet [<Product: Milk >, <Product: Chocolate>, <Product: Juice>]>
And the SQL generated by the ORM will be like this:
SELECT "id", "name"
FROM "products"
WHERE "name" IN ('Milk', 'Chocolate', 'Juice')
ORDER BY CASE
WHEN "name" = 'Milk' THEN 0
WHEN "name" = 'Chocolate' THEN 1
WHEN "name" = 'Juice' THEN 2
ELSE NULL
END ASC
When there is no relation between the objects that you are fetching and you still wish to fetch (or arrange) them in certain (custom) order, you may try doing this:
unordered_products = Product.objects.filter(name__in=["Milk", "Chocolate", "Juice"])
product_order = ["Milk", "Chocolate", "Juice"]
preserved = Case(*[When(name=name, then=pos) for pos, name in enumerate(product_order)])
ordered_products = unordered_products.order_by(preserved)
Hope it helps!
Try this into meta class from model:
class Meta:
ordering = ('name', 'related__name', )
this get your records ordered by your specified field's
then: chocolate, chocolate blue, chocolate white, juice green, juice XXX, milk, milky, milk YYYY should keep that order when you fetch
Creating a QuerySet from a list while preserving order
This means the order of output QuerySet will be same as the order of list used to filter it.
The solution is more or less same as #PaoloMelchiorre answer
But if there are more items lets say 1000 products in
product_names then you don't have to worry about adding more conditions in Case, you can use extra method of QuerySet
product_names = ["Milk", "Chocolate", "Juice", ...]
clauses = ' '.join(['WHEN name=%s THEN %s' % (name, i) for i, name in enumerate(product_names)])
ordering = 'CASE %s END' % clauses
queryset = Product.objects.filter(name__in=product_names).extra(
select={'ordering': ordering}, order_by=('ordering',))
# Output: <QuerySet [<Product: Milk >, <Product: Chocolate>, <Product: Juice>,...]>
I have this model:
class User_Data(AbstractUser):
date_of_birth = models.DateField(null=True,blank=True)
city = models.CharField(max_length=255,default='',null=True,blank=True)
address = models.TextField(default='',null=True,blank=True)
gender = models.TextField(default='',null=True,blank=True)
And I need to run a django query to get the count of each age. Something like this:
Age || Count
10 || 100
11 || 50
and so on.....
Here is what I did with lambda:
usersAge = map(lambda x: calculate_age(x[0]), User_Data.objects.values_list('date_of_birth'))
users_age_data_source = [[x, usersAge.count(x)] for x in set(usersAge)]
users_age_data_source = sorted(users_age_data_source, key=itemgetter(0))
There's a few ways of doing this. I've had to do something very similar recently. This example works in Postgres.
Note: I've written the following code the way I have so that syntactically it works, and so that I can write between each step. But you can chain these together if you desire.
First we need to annotate the queryset to obtain the 'age' parameter. Since it's not stored as an integer, and can change daily, we can calculate it from the date of birth field by using the database's 'current_date' function:
ud = User_Data.objects.annotate(
age=RawSQL("""(DATE_PART('year', current_date) - DATE_PART('year', "app_userdata"."date_of_birth"))::integer""", []),
)
Note: you'll need to change the "app_userdata" part to match up with the table of your model. You can pick this out of the model's _meta, but this just depends if you want to make this portable or not. If you do, use a string .format() to replace it with what the model's _meta provides. If you don't care about that, just put the table name in there.
Now we pick the 'age' value out so that we get a ValuesQuerySet with just this field
ud = ud.values('age')
And then annotate THAT queryset with a count of age
ud = ud.annotate(
count=Count('age'),
)
At this point we have a ValuesQuerySet that has both 'age' and 'count' as fields. Order it so it comes out in a sensible way..
ud = ud.order_by('age')
And there you have it.
You must build up the queryset in this order otherwise you'll get some interesting results. i.e; you can't group all the annotates together, because the second one for count depends on the first, and as a kwargs dict has no notion of what order the kwargs were defined in, when the queryset does field/dependency checking, it will fail.
Hope this helps.
If you aren't using Postgres, the only thing you'll need to change is the RawSQL annotation to match whatever database engine it is that you're using. However that engine can get the year of a date, either from a field or from its built in "current date" function..providing you can get that out as an integer, it will work exactly the same way.
I have two models:
Base_Activity:
some fields
User_Activity:
user = models.ForeignKey(settings.AUTH_USER_MODEL)
activity = models.ForeignKey(Base_Activity)
rating = models.IntegerField(default=0) #Will be -1, 0, or 1
Now I want to query Base_Activity, and sort the items that have the most corresponding user activities with rating=1 on top. I want to do something like the query below, but the =1 part is obviously not working.
activities = Base_Activity.objects.all().annotate(
up_votes = Count('user_activity__rating'=1),
).order_by(
'up_votes'
)
How can I solve this?
You cannot use Count like that, as the error message says:
SyntaxError: keyword can't be an expression
The argument of Count must be a simple string, like user_activity__rating.
I think a good alternative can be to use Avg and Count together:
activities = Base_Activity.objects.all().annotate(
a=Avg('user_activity__rating'), c=Count('user_activity__rating')
).order_by(
'-a', '-c'
)
The items with the most rating=1 activities should have the highest average, and among the users with the same average the ones with the most activities will be listed higher.
If you want to exclude items that have downvotes, make sure to add the appropriate filter or exclude operations after annotate, for example:
activities = Base_Activity.objects.all().annotate(
a=Avg('user_activity__rating'), c=Count('user_activity__rating')
).filter(user_activity__rating__gt=0).order_by(
'-a', '-c'
)
UPDATE
To get all the items, ordered by their upvotes, disregarding downvotes, I think the only way is to use raw queries, like this:
from django.db import connection
sql = '''
SELECT o.id, SUM(v.rating > 0) s
FROM user_activity o
JOIN rating v ON o.id = v.user_activity_id
GROUP BY o.id ORDER BY s DESC
'''
cursor = connection.cursor()
result = cursor.execute(sql_select)
rows = result.fetchall()
Note: instead of hard-coding the table names of your models, get the table names from the models, for example if your model is called Rating, then you can get its table name with Rating._meta.db_table.
I tested this query on an sqlite3 database, I'm not sure the SUM expression there works in all DBMS. Btw I had a perfect Django site to test, where I also use upvotes and downvotes. I use a very similar model for counting upvotes and downvotes, but I order them by the sum value, stackoverflow style. The site is open-source, if you're interested.