I banged my head on this one:
I have 2 models, and I am trying to design a custom manager/queryset which will allow me to annotate to each Series the id of the linked puzzles satisfying certain conditions in the format '2,13,26'.
The simplified models:
class Series(models.Model):
puzzles = models.ManyToManyField(
Puzzle, through='SeriesElement', related_name='series')
is_public = models.BooleanField(null=False, blank=False, default=False)
class Puzzle(models.Model):
pass
my custom aggregator:
from django.db.models.aggregates import Aggregate
from django.db.models.functions import Coalesce
from django.db.models.fields import CharField
from django.db.models.expressions import Value
class GroupConcat(Aggregate):
"""
according to https://stackoverflow.com/questions/10340684/group-concat-equivalent-in-django
according to https://stackoverflow.com/a/55216659
would be compatible with MySQL and SQLite
"""
function = 'GROUP_CONCAT'
def __init__(self, expression, distinct=False, ordering=None, **extra):
super(GroupConcat, self).__init__(expression,
distinct='DISTINCT ' if distinct else '',
ordering=' ORDER BY %s' % ordering if ordering is not None else '',
output_field=CharField(),
**extra)
def as_sqlite(self, compiler, connection, **extra):
return super().as_sql(compiler,
connection,
template='%(function)s(%(distinct)s%(expressions)s%(ordering)s)',
**extra)
one tentative to achieve my goal:
pzl_sub = apps.get_model('puzzles', 'Puzzle').objects.filter(series__id= OuterRef('id'))
pzl_sub = pzl_sub.filter(series_elements__isnull=False).add_nb_public_series().filter(nb_public_series=5)
pzl_ids= pzl_sub.order_by().values('id')
qs = Series.objects.annotate(id_str_pzl = GroupConcat(pzl_ids))
I obtain only one puzzle.id that fit the specified conditions, instead of the concat of all of the puzzle.ids that fit the conditions
Any clue on what I'm doing wrong?
After days of fighting, and testing everything, I finally figured it out:
pzl_sub = apps.get_model('puzzles', 'Puzzle').objects.filter(series__id= OuterRef('id'))
pzl_sub = pzl_sub.filter(series_elements__isnull=False).add_nb_public_series().filter(nb_public_series=5)
pzl_ids= pzl_sub.order_by().values('series__id')
qs = Series.objects.annotate(id_str_pzl = Subquery(pzl_ids.annotate(result=GroupConcat('id')).order_by().values('result')))
in pzl_ids, we need to single out values('series__id')
and only in a second step, we need to annotate with the aggregator, and single-out again the resulting value ...
Related
I have an Order model like so:
class Order(models.Model):
created_at = models.DateTimeField(...)
An order can be created at any time, but all orders get shipped out on the following Monday.
How can I add an extra field to my orders queryset called assembly_date that reflects the next Monday (date the order should be shipped)?
I tried creating a custom OrderManager like so, but am not sure how to correctly set the assembly_date:
from django.db.models import F, ExpressionWrapper, DateField
from django.db.models.functions import ExtractWeekDay
class OrderManager(models.Manager):
def get_queryset():
# need help with logic here:
return self.super().get_queryset().annotate(
assembly_date = ExpressionWrapper(
F("created_at") - ExtractWeekDay("created_at"),
output_field = DateField(),
)
)
But this operation results in the following error:
django.db.utils.ProgrammingError: operator does not exist: timestamp with time zone - double precision
LINE 1: ...E NULL END) * 2628000.0) * INTERVAL '1 seconds')) - EXTRACT(...
^
HINT: No operator matches the given name and argument types. You might need to add explicit type casts.
Keep in mind, I want to be able to filter all orders based on their assembly_date.
Basically you need to dynamically generate timedelta inside of annotate. But as far as I know, there is no way you can apply isoweekday() to a datetimefield inside of annotate.
You can have another field as assembly_date in your model, and use it directly to query.
from datetime import timedelta, date
class Order(models.Model):
created_at = models.DateTimeField(...)
assembly_date = models.DateTimeField(...)
def save(self, *args, **kwargs):
weekday = self.created_at.isoweekday() # 1 is Monday, 2 is Tuesday.
daysdiff = 8 - weekday
self.assembly_date = self.created_at + timedelta(days= daysdiff)
super(Order, self).save(*args, **kwargs)
I am working on creating a cocktail recipe app as a learning exercise.
I am trying to create a filter through Django's Rest Framework that accepts a string of ingredient IDs through a query parameter (?=ingredients_exclusive=1,3,4), and then searches for all recipes that have all of those ingredients. I would like to search for “All cocktails that have both rum and grenadine” and then also, separately “All cocktails that have rum, and all cocktails that have grendaine.”
The three models in my app are Recipes, RecipeIngredients, and IngredientTypes. Recipes (Old Fashioned) have multiple RecipeIngredients (2oz of Whiskey), and RecipeIngredients are all of a Ingredient Type (Whiskey). I will eventually change the RecipeIngredient to a through model depending on how far I decide to take this.
The list can be of a variable length, so I cannot just chain together filter functions. I have to loop through the list of ids and then build a Q().
However, I'm having some issues. Through the Django Shell, I have done this:
>>> x = Recipe.objects.all()
>>> q = Q(ingredients__ingredient_type=3) & Q(ingredients__ingredient_type=7)
>>> x.filter(q)
<QuerySet []>
>>> x.filter(ingredients__ingredient_type=3).filter(ingredients__ingredient_type=7)
<QuerySet [<Recipe: Rum and Tonic>]>
So here's my question: Why is the Q object that ANDs the two queries different than the chained filters of same object?
I've read through the "Complex lookups with Q objects" in the Django documentation and it doesn't seem to help.
Just for reference, here are my filters in Filters.py.
The "OR" version of this command is working properly:
class RecipeFilterSet(FilterSet):
ingredients_inclusive = django_filters.CharFilter(method='filter_by_ingredients_inclusive')
ingredients_exclusive = django_filters.CharFilter(method='filter_by_ingredients_exclusive')
def filter_by_ingredients_inclusive(self, queryset, name, value):
ingredients = value.split(',')
q_object = Q()
for ingredient in ingredients:
q_object |= Q(ingredients__ingredient_type=ingredient)
return queryset.filter(q_object).distinct()
def filter_by_ingredients_exclusive(self, queryset, name, value):
ingredients = value.split(',')
q_object = Q()
for ingredient in ingredients:
q_object &= Q(ingredients__ingredient_type=ingredient)
return queryset.filter(q_object).distinct()
class Meta:
model = Recipe
fields = ()
I've also included my models below:
# -*- coding: utf-8 -*-
from __future__ import unicode_literals
from django.db import models
class IngredientType(models.Model):
name = models.CharField(max_length=256)
CATEGORY_CHOICES = (
('LIQUOR', 'Liquor'),
('SYRUP', 'Syrup'),
('MIXER', 'Mixer'),
)
category = models.CharField(
max_length=128, choices=CATEGORY_CHOICES, default='MIXER')
def __str__(self):
return self.name
class Recipe(models.Model):
name = models.CharField(max_length=256)
def __str__(self):
return self.name
class RecipeIngredient(models.Model):
ingredient_type = models.ForeignKey(IngredientType, on_delete=models.CASCADE, related_name="ingredients")
quantity = models.IntegerField(default=0)
quantity_type = models.CharField(max_length=256)
recipe = models.ForeignKey(Recipe, on_delete=models.CASCADE, related_name="ingredients")
#property
def ingredient_type_name(self):
return self.ingredient_type.name
#property
def ingredient_type_category(self):
return self.ingredient_type.category
def __str__(self):
return f'{self.quantity}{self.quantity_type} of {self.ingredient_type}'
Any help would be very much appreciated!
The difference between the two approaches to filter() is described in Spanning multi-valued relationships:
Everything inside a single filter() call is applied simultaneously to filter out items matching all those requirements.... For multi-valued relations, they apply to any object linked to the primary model, not necessarily those objects that were selected by an earlier filter() call.
The example in the documentation makes it more clear. I'll rewrite it in terms of your problem:
To select all recipes that contain an ingredient with both type 3 and type 7, we would write:
Recipe.objects.filter(ingredients__ingredient_type=3, ingredients__ingredient_type=7)
That is of course impossible in your model, so this would return an empty queryset, just like your Q example with AND.
To select all recipes that contain an ingredient with type 3 as well as an ingredient with type 7, we would write:
Recipe.objects.filter(ingredients__ingredient_type=3).filter(ingredients__ingredient_type=7)
It's not especially intuitive, but they needed a way to distinguish these two cases and this is what they came up with.
Back to your problem, the OR case can be made simpler by using the in operator:
Recipe.objects.filter(ingredients__ingredient_type__in=[3, 7]).distinct()
The AND case is complicated because it's a condition that involves multiple rows. A simple approach would be to just take the OR version above and further process it in Python to find the subset that has all the ingredients.
A query approach that should work involves annotation with Count. This is untested, but something like:
Recipe.objects.annotate(num_ingredients=Count("ingredients",
filter=Q(ingredients__ingredient_type__in=[3, 7]))
.filter(num_ingredients=2)
Another approach to the AND case for Django 1.11+ would be to use the relatively new QuerySet intersection() method. As per the docs, this method:
Uses SQL’s INTERSECT operator to return the shared elements of two or more QuerySets.
So given an arbitrary list of IngredientType primary keys, you could create a filter() query for each pk (let's call these subqueries) and then spread that list (the * operator) into the intersection() method.
Like so:
# the base `QuerySet` and `IngredientType` pks to filter on
queryset = Recipe.objects.all()
ingredient_type_pks = [3, 7]
# build the list of subqueries
subqueries = []
for pk in ingredient_type_pks:
subqueries.append(queryset.filter(ingredients__ingredient_type__pk=pk))
# spread the subqueries into the `intersection` method
return queryset.intersection(*subqueries).distinct()
I added distinct() in there just to be safe and avoid duplicate results, but I am actually not certain whether it's necessary. Will have to test and update this post later.
I am trying to filter in view my queryset based on relation between 2 fields .
however always getting the error that my field is not defined .
My Model has several calculated columns and I want to get only the records where values of field A are greater than field B.
So this is my model
class Material(models.Model):
version = IntegerVersionField( )
code = models.CharField(max_length=30)
name = models.CharField(max_length=30)
min_quantity = models.DecimalField(max_digits=19, decimal_places=10)
max_quantity = models.DecimalField(max_digits=19, decimal_places=10)
is_active = models.BooleanField(default=True)
def _get_totalinventory(self):
from inventory.models import Inventory
return Inventory.objects.filter(warehouse_Bin__material_UOM__UOM__material=self.id, is_active = true ).aggregate(Sum('quantity'))
totalinventory = property(_get_totalinventory)
def _get_totalpo(self):
from purchase.models import POmaterial
return POmaterial.objects.filter(material=self.id, is_active = true).aggregate(Sum('quantity'))
totalpo = property(_get_totalpo)
def _get_totalso(self):
from sales.models import SOproduct
return SOproduct.objects.filter(product__material=self.id , is_active=true ).aggregate(Sum('quantity'))
totalso = property(_get_totalpo)
#property
def _get_total(self):
return (totalinventory + totalpo - totalso)
total = property(_get_total)
And this is line in my view where I try to get the conditional queryset
po_list = MaterialFilter(request.GET, queryset=Material.objects.filter( total__lte = min_quantity ))
But I am getting the error that min_quantity is not defined
What could be the problem ?
EDIT:
My problem got solved thank you #Moses Koledoye but in the same code I have different issue now
Cannot resolve keyword 'total' into field.Choices are: am, author, author_id, bom, bomversion, code, creation_time, description, id, inventory, is_active, is_production, itemgroup, itemgroup_id, keywords, materialuom, max_quantity, min_quantity, name, pomaterial, produce, product, slug, trigger_quantity, uom, updated_by, updated_by_id, valid_from, valid_to, version, warehousebin
Basically it doesn't show any of my calculated fields I have in my model.
Django cannot write a query which is conditioned on a field whose value is unknown. You need to use a F expression for this:
from django.db.models import F
queryset = Material.objects.filter(total__lte = F('min_quantity'))
And your FilterSet becomes:
po_list = MaterialFilter(request.GET, queryset = Material.objects.filter(total__lte=F('min_quantity')))
From the docs:
An F() object represents the value of a model field or annotated
column. It makes it possible to refer to model field values and
perform database operations using them without actually having to pull
them out of the database into Python memory
In a Django app, I keep daily scores of users in such a model:
class Score(models.Model):
user = models.ForeignKey(User)
score = models.PositiveIntegerField(default=0)
date = models.DateField(auto_now_add=True)
I want to find out the days when a user's score has changed drastically compared to a consequent day. That is, if for example, the user scores 10 times higher than the previous day.
How can I include such a condition in a query filter using Django ORM? Is it possible with a single query using conditional expressions as described here: https://docs.djangoproject.com/en/1.9/ref/models/conditional-expressions/
Thanks.
If you change your Score class slightly to include the previous day's score (which is admittedly pretty wasteful), you can pack the query into one line using F expressions.
Your new class:
class Score(models.Model):
user = models.ForeignKey(User)
score = models.PositiveIntegerField(default=0)
lastscore = models.PositiveIntegerField(default=0)
date = models.DateField(auto_now_add=True)
Then the filter becomes:
from django.db.models import F
daily_chg = 10
big_changes = Score.objects.filter(score__gt=daily_chg*F('lastscore'))
Instead of using timedeltas to search for and set the previous day's score field, I'd look into establishing an ordering via a Meta class and calling latest() when saving the current day's score.
Using timedelta we can test for the last week's days for a given user as such:
from my_app.models import Score
import datetime
def some_view(request):
days_with_score_boost = []
today = datetime.date.today()
for each_day in xrange(0,7):
day_to_test, day_before_to_test = today - datetime.timedelta(days=each_day), today - datetime.timedelta(days=each_day + 1)
day_before_score = Score.objects.get(user=request.user,date=today - datetime.timedelta(days=each_day)).score # will need to catch the exception raised if .get return None and process appropriately
if Score.objects.filter(user=request.user,score__gte=days_before_score * 10,date=day_before_to_test).count() > 0:
days_with_score_boost.append(day_to_test)
days_with_score_boost will be a list of datetime.date objects where the score increased by 10 or more from the day before.
In reponse to your comment, I'd make a measure that checks at save time whether the score boost has occured. However, I would get rid of auto_now_add in favor of writing in the save method.
from django.utils import timezone
from django.core.exceptions import ObjectDoesNotExist
class Score(models.Model):
user = models.ForeignKey(User)
score = models.PositiveIntegerField(default=0)
date = models.DateField(null=True,blank=True)
increased_ten_over_previous_day = models.BooleanField(null=True,blank=True)
def save(self, *args, **kwargs):
self.date = timezone.now().today()
try:
yesterday_score = Score.objects.get(date=self.date-timezone.timedelta(days=1)).score
self.increased_ten_over_previous_day = (yesterday_score * 10) <= self.score
except ObjectDoesNotExist: # called if Score.object.get returns no object; requires you only have one score per user per date
self.increased_ten_over_previous_day = False
super(self, Score).save(*args, **kwargs)
Then you could filter objects for a date_range where increased_ten_over_previous_day is True.
My question is about creating a QuerySet Mixin which provides identical QuerySet methods for both a model and a related model. Here is example code, and the first class ByPositionMixin is what I am focused on:
from django.db import models
from django.db.models.query import QuerySet
from django.core.exceptions import FieldError
class ByPositionMixin(object):
def batters(self):
try:
return self.exclude(positions=1)
except FieldError:
return self.exclude(position=1)
class PlayerQuerySet(QuerySet, ByPositionMixin):
pass
class PlayerPositionQuerySet(QuerySet, ByPositionMixin):
pass
class PlayerManager(models.Manager):
def get_query_set(self):
return PlayerQuerySet(self.model, using=self._db)
class PlayerPositionManager(models.Manager):
def get_query_set(self):
return PlayerPositionQuerySet(self.model, using=self._db)
class Position(models.Model):
# pos_list in order ('P', 'C', '1B', '2B', '3B', 'SS', 'LF', 'CF', 'RF')
# pos id / pk correspond to index value of pos_list(pos)
pos = models.CharField(max_length=2)
class Player(models.Model):
name = models.CharField(max_length=100)
positions = models.ManyToManyField(Position, through='PlayerPosition')
objects = PlayerManager()
class PlayerPosition(models.Model):
player = models.ForeignKey(Player)
position = models.ForeignKey(Position)
primary = models.BooleanField()
objects = PlayerPositionManager()
Inside ByPositionMixin, I try exclude(positions=1) which queries against PlayerQuerySet and if that generates a FieldError, I try exclude(position=1) which queries against PlayerPositionQuerySet. The difference in field names is precise, a Player() has positions, but a PlayerPosition() has only one position. So the difference it the exclude() query is 'positions' / 'position'. Since I will have many custom queries (e.g. batters(), pitchers(), by_position() etc.), do I have to write out try / except code for each one?
Or is there a different approach which would let me write custom queries without having to try against one model and then against the other one?
UPDATE: basically, I have decided to write a kwarg helper function, which provides the correct kwargs for both Player and PlayerPosition. It's a little elaborate (and perhaps completely unnecessary), but should be able to be made to simplify the code for several custom queries.
class ByPositionMixin(object):
def pkw(self, **kwargs):
# returns appropriate kwargs, at the moment, only handles one kwarg
key = kwargs.keys()[0] # e.g. 'positions__in'
value = kwargs[key]
key_args = key.split('__')
if self.model.__name__ == 'Player':
first_arg = 'positions'
elif self.model.__name__ == 'PlayerPosition':
first_arg = 'position'
else:
first_arg = key_args[0]
key = '__'.join([first_arg] + key_args[1:])
return {key: value}
def batters(self): # shows how pkw() is used
return self.exclude(**self.pkw(positions=1))