I am trying to write a custom PostgreSQL function in Django that will coerce datetimes to a specified timezone inside of a queryset. My first pass at the db function looks like this:
from django.db.models.expressions import Func
class DateTimeInTimezone(Func):
template="%(expressions)s AT TIME ZONE %(tz_info)s"
This function works in the simple case where I pass a timezone string into the function directly like so:
MyModel.objects.filter(timefield__gte=DateTimeInTimezone(Now(), tz_info='EST'))
However it doesn't work in the more complex case, where the timezone is defined on some field on the model. Consider the following contrived example:
class User(models.Model):
time_zone = models.CharField()
class Meeting(models.Model):
users = models.ManyToManyField(User, related_name='meetings')
start_time = models.DateTimeField() # in UTC
end_time = models.DateTimeField() # in UTC
To answer the question "What users will be in a meeting at 12pm local time today?", I'd need some variation of this queryset:
noon_utc = ...
User.objects.filter(
meetings__start_time__lte=DateTimeInTimezone(noon_utc, tz_info=F('time_zone')),
meetings__end_time__gt=DateTimeInTimezone(noon_utc, tz_info=F('time_zone'))
)
As currently written, however, DateTimeInTimezone will simply inject the string F('time_zone') into the sql rather than resolving the field.
Is it possible to add support for F Expressions to this function? Is there some other approach that I should consider?
A simple solution is possible with parameter arg_joiner:
class DateTimeInTimezone(Func):
function = ''
arg_joiner = ' AT TIME ZONE '
def __init__(self, timestamp, tz_info):
super(DateTimeInTimezone, self).__init__(timestamp, tz_info)
The method __init__ is used only for the purpose of readability with clear names of parameters. Then arity is not important if parameters are declared by __init__.
A oneliner function is useful for fast development if readability is not important:
...filter(
meetings__start_time__lte=Func(noon_utc, tz_info=F('time_zone'), arg_joiner=' AT TIME ZONE ', function=''),
)
Verified:
>>> qs = User.objects.filter(...)
>>> print(str(qs.query))
SELECT ... WHERE ("app_meeting"."start_time" <= ((2017-10-03 08:18:12.663640 AT TIME ZONE "app_user"."time_zone")) AND ...)
Found an acceptable solution. I overrode the as_sql method for function like so, allowing the django internals to resolve the F expression and then separating it back out into a kwarg I could use in a different part of the template.
class DateTimeInTimezone(Func):
'''
Coerce a datetime into a specified timezone
'''
template="%(expressions)s AT TIME ZONE %(tz_info)s"
arity = 2
def as_sql(self, compiler, connection, function=None, template=None, arg_joiner=None, **extra_context):
connection.ops.check_expression_support(self)
sql_parts = []
params = []
for arg in self.source_expressions:
arg_sql, arg_params = compiler.compile(arg)
sql_parts.append(arg_sql)
params.extend(arg_params)
data = self.extra.copy()
data.update(**extra_context)
# Use the first supplied value in this order: the parameter to this
# method, a value supplied in __init__()'s **extra (the value in
# `data`), or the value defined on the class.
if function is not None:
data['function'] = function
else:
data.setdefault('function', self.function)
template = template or data.get('template', self.template)
arg_joiner = arg_joiner or data.get('arg_joiner', self.arg_joiner)
data['expressions'] = data['field'] = arg_joiner.join(sql_parts)
parts = data['expressions'].split(', ')
data['expressions'] = parts[0]
data['tz_info'] = parts[1]
return template % data, params
I added the three lines between the assignment of data['expressions'] and the final return template % data, params. This isn't a great long term solution as the django internals for this method could change in the next version, but it suits my needs for the time being.
Related
I would like to allow a bunch of my users to run arbitrary and periodical queries.
So far, I've managed to create this model:
class BasicQuery(models.Model):
id = models.AutoField(primary_key=True)
target_entity = models.CharField(max_length=255)
target_field = models.CharField(max_length=255)
operator = models.CharField(max_length=10)
compare_value = models.CharField(max_length=10)
active = models.BooleanField()
run_every = models.ForeignKey(QueryFrequency)
and the function which executes it :
def rule_runner(rule):
target_entity = rule.target_entity
fieldname = rule.target_field
val = rule.compare_value
if rule.operator:
cmd = '{}.objects.filter({}__{}={})'.format(rule.target_entity,
rule.target_field,
rule.operator,
rule.compare_value)
else:
cmd = '{}.objects.filter({}={})'.format(rule.target_entity,
rule.target_field,
rule.compare_value)
return eval(cmd)
But because of the eval(cmd), the code is way too dangerous to go to production, no matter how many tests I'll do before calling this function I'll never feel safe about it. Any suggestions on how I can achieve this purpose in a clean way?
You could unpack a dict containing the values, so they can be used dynamically with the filter method. And about the rule.target_entity, you can use importlib.import_module(str) and getattr(module, str) to get it:
import importlib
def rule_runer(rule):
model = getattr(importlib.import_module('app.models'), 'Model')
if rule.operator:
field = rule.target_field
else:
field = '{}__{}'.format(rule.target_field, rule.operator)
return model.objects.filter(**{field: rule.compare_value})
To unpack a dict is to put ** before it. This can be used to dynamically set the function's arguments, so the dict keys will be the arguments names and the dict values will be these arguments values.
importlib.import_module will allow you to import any module within your PYTHONPATH, which usually includes all your django apps paths (so app.models shouldn't be a problem), by using a string.
This is definitely safer than using eval and you do have to check if the module's path or the model exists to avoid throwing a ModuleNotFoundError (ImportError if < Python 3.6) or AttributeError.
While creating a front end for a Django module I faced the following problem inside Django core:
In order to display a link to the next/previous object from a model query, we can use the extra-instance-methods of a model instance: get_next_by_FIELD() or get_previous_by_FIELD(). Where FIELD is a model field of type DateField or DateTimeField.
Lets explain it with an example
from django.db import models
class Shoe(models.Model):
created = models.DateTimeField(auto_now_add=True, null=False)
size = models.IntegerField()
A view to display a list of shoes, excluding those where size equals 4:
def list_shoes(request):
shoes = Shoe.objects.exclude(size=4)
return render_to_response(request, {
'shoes': shoes
})
And let the following be a view to display one shoe and the corresponding
link to the previous and next shoe.
def show_shoe(request, shoe_id):
shoe = Shoe.objects.get(pk=shoe_id)
prev_shoe = shoe.get_previous_by_created()
next_shoe = shoe.get_next_by_created()
return render_to_response('show_shoe.html', {
'shoe': shoe,
'prev_shoe': prev_shoe,
'next_shoe': next_shoe
})
Now I have the situation that the show_shoe view displays the link to the previous/next regardless of the shoes size. But I actually wanted just shoes whose size is not 4.
Therefore I tried to use the **kwargs argument of the get_(previous|next)_by_created() methods to filter out the unwanted shoes, as stated by the documentation:
Both of these methods will perform their queries using the default manager for the model. If you need to emulate filtering used by a custom manager, or want to perform one-off custom filtering, both methods also accept
optional keyword arguments, which should be in the format described in Field lookups.
Edit: Keep an eye on the word "should", because then also (size_ne=4) should work, but it doesn't.
The actual problem
Filtering using the lookup size__ne ...
def show_shoe(request, shoe_id):
...
prev_shoe = shoe.get_previous_by_created(size__ne=4)
next_shoe = shoe.get_next_by_created(size__ne=4)
...
... didn't work, it throws FieldError: Cannot resolve keyword 'size_ne' into field.
Then I tried to use a negated complex lookup using Q objects:
from django.db.models import Q
def show_shoe(request, shoe_id):
...
prev_shoe = shoe.get_previous_by_created(~Q(size=4))
next_shoe = shoe.get_next_by_created(~Q(size=4))
...
... didn't work either, throws TypeError: _get_next_or_previous_by_FIELD() got multiple values for argument 'field'
Because the get_(previous|next)_by_created methods only accept **kwargs.
The actual solution
Since these instance methods use the _get_next_or_previous_by_FIELD(self, field, is_next, **kwargs) I changed it to accept positional arguments using *args and passed them to the filter, like the **kwargs.
def my_get_next_or_previous_by_FIELD(self, field, is_next, *args, **kwargs):
"""
Workaround to call get_next_or_previous_by_FIELD by using complext lookup queries using
Djangos Q Class. The only difference between this version and original version is that
positional arguments are also passed to the filter function.
"""
if not self.pk:
raise ValueError("get_next/get_previous cannot be used on unsaved objects.")
op = 'gt' if is_next else 'lt'
order = '' if is_next else '-'
param = force_text(getattr(self, field.attname))
q = Q(**{'%s__%s' % (field.name, op): param})
q = q | Q(**{field.name: param, 'pk__%s' % op: self.pk})
qs = self.__class__._default_manager.using(self._state.db).filter(*args, **kwargs).filter(q).order_by('%s%s' % (order, field.name), '%spk' % order)
try:
return qs[0]
except IndexError:
raise self.DoesNotExist("%s matching query does not exist." % self.__class__._meta.object_name)
And calling it like:
...
prev_shoe = shoe.my_get_next_or_previous_by_FIELD(Shoe._meta.get_field('created'), False, ~Q(state=4))
next_shoe = shoe.my_get_next_or_previous_by_FIELD(Shoe._meta.get_field('created'), True, ~Q(state=4))
...
finally did it.
Now the question to you
Is there an easier way to handle this? Should shoe.get_previous_by_created(size__ne=4) work as expected or should I report this issue to the Django guys, in the hope they'll accept my _get_next_or_previous_by_FIELD() fix?
Environment: Django 1.7, haven't tested it on 1.9 yet, but the code for _get_next_or_previous_by_FIELD() stayed the same.
Edit: It is true that complex lookups using Q object is not part of "field lookups", it's more part of the filter() and exclude() functions instead. And I am probably wrong when I suppose that get_next_by_FIELD should accept Q objects too. But since the changes involved are minimal and the advantage to use Q object is high, I think these changes should get upstream.
tags: django, complex-lookup, query, get_next_by_FIELD, get_previous_by_FIELD
(listing tags here, because I don't have enough reputations.)
You can create custom lookup ne and use it:
.get_next_by_created(size__ne=4)
I suspect the method you've tried first only takes lookup arg for the field you're basing the get_next on. Meaning you won't be able to access the size field from the get_next_by_created() method, for example.
Edit : your method is by far more efficient, but to answer your question on the Django issue, I think everything is working the way it is supposed to. You could offer an additional method such as yours but the existing get_next_by_FIELD is working as described in the docs.
You've managed to work around this with a working method, which is OK I guess, but if you wanted to reduce the overhead, you could try a simple loop :
def get_next_by_field_filtered(obj, field=None, **kwargs):
next_obj = getattr(obj, 'get_next_by_{}'.format(field))()
for key in kwargs:
if not getattr(next_obj, str(key)) == kwargs[str(key)]:
return get_next_by_field_filtered(next_obj, field=field, **kwargs)
return next_obj
This isn't very efficient but it's one way to do what you want.
Hope this helps !
Regards,
Using Django's ORM, I am trying to find instances of myModel based on two of its datetime variables; specifically, where the months of these two datetimes are not equal. I understand to filter by the value of a modelfield, you can use Django's F( ) expressions, so I thought I'd try something like this:
myModel.objects.filter(fixed_date__month=F('closed_date__month'))
I know this wouldn't find instances where they aren't equal, but I thought it'd be a good first step since I've never used the F expressions before. However, it doesn't work as I thought it should. I expected it to give me a queryset of objects where the value of the fixed_date month was equal to the value of the closed_date month, but instead I get an error:
FieldError: Join on field 'closed_date' not permitted. Did you misspell 'month' for the lookup type?
I'm not sure if what I'm trying to do isn't possible or straightforward with the ORM, or if I'm just making a simple mistake.
It doesn't look like django F objects currently support extracting the month inside a DateTimeField, the error message seems to be stating that the F object is trying to convert the '__' inside the string 'closed_date__month' as a Foreignkey between different objects, which are usually stored as joins inside an sql database.
You could carry out the same query by iterating across the objects:
result = []
for obj in myModel.objects.all():
if obj.fixed_date.month != obj.closed_date.month:
result.append(obj)
or as a list comprehension:
result = [obj for obj in myModel.objects.all() if obj.fixed_date.month != obj.closed_date.month]
Alternatively, if this is not efficient enough, the months for the two dates could be cached as IntegerFields within the model, something like:
class myModel(models.Model):
....other fields....
fixed_date = models.DateTimeField()
closed_date = models.DateTimeField()
fixed_month = models.IntegerField()
closed_month = models.IntegerField()
store the two integers when the relevant dates are updated:
myModel.fixed_month = myModel.fixed_date.month
myModel.save()
Then use an F object to compare the two integer fields:
myModel.objects.filter(fixed_month__ne=F('closed_month'))
The ne modifier will do the not equal test.
Edit - using raw sql
If you are using an sql based database, then most efficient method is to use the .raw() method to manually specify the sql:
myModel.objects.raw('SELECT * FROM stuff_mymodel WHERE MONTH(fixed_date) != MONTH(close_date)')
Where 'stuff_mymodel' is the correct name of the table in the database. This uses the SQL MONTH() function to extract the values from the month fields, and compare their values. It will return a collection of objects.
There is some nay-saying about the django query system, for example: http://charlesleifer.com/blog/shortcomings-in-the-django-orm-and-a-look-at-peewee-a-lightweight-alternative/. This example could be taken as demonstrating another inconsistency in it's query api.
My thinking is this:
class myModel(models.Model):
fixed_date = models.DateTimeField()
closed_date = models.DateTimeField()
def has_diff_months(self):
if self.fixed_date.month != self.closed_date.month:
return True
return False
Then:
[x for x in myModel.objects.all() if x.has_diff_months()]
However, for a truly efficient solution you'd have to use another column. It makes sense to me that it'd be a computed boolean field that is created when you save, like so:
class myModel(models.Model):
fixed_date = models.DateTimeField()
closed_date = models.DateTimeField()
diff_months = models.BooleanField()
#overriding save method
def save(self, *args, **kwargs):
#calculating the value for diff_months
self.diff_months = (self.fixed_date.month != self.closed_date.month)
#aaand... saving:
super(Blog, self).save(*args, **kwargs)
Then filtering would simply be:
myModel.objects.filter(diff_months=True)
EDITED:
How can I set a Django field's default to a function that gets evaluated each time a new model object gets created?
I want to do something like the following, except that in this code, the code gets evaluated once and sets the default to the same date for each model object created, rather than evaluating the code each time a model object gets created:
from datetime import datetime, timedelta
class MyModel(models.Model):
# default to 1 day from now
my_date = models.DateTimeField(default=datetime.now() + timedelta(days=1))
ORIGINAL:
I want to create a default value for a function parameter such that it is dynamic and gets called and set each time the function is called. How can I do that? e.g.,
from datetime import datetime
def mydate(date=datetime.now()):
print date
mydate()
mydate() # prints the same thing as the previous call; but I want it to be a newer value
Specifically, I want to do it in Django, e.g.,
from datetime import datetime, timedelta
class MyModel(models.Model):
# default to 1 day from now
my_date = models.DateTimeField(default=datetime.now() + timedelta(days=1))
The question is misguided. When creating a model field in Django, you are not defining a function, so function default values are irrelevant:
from datetime import datetime, timedelta
class MyModel(models.Model):
# default to 1 day from now
my_date = models.DateTimeField(default=datetime.now() + timedelta(days=1))
This last line is not defining a function; it is invoking a function to create a field in the class.
In this case datetime.now() + timedelta(days=1) will be evaluated once, and stored as the default value.
PRE Django 1.7
Django [lets you pass a callable as the default][1], and it will invoke it each time, just as you want:
from datetime import datetime, timedelta
class MyModel(models.Model):
# default to 1 day from now
my_date = models.DateTimeField(default=lambda: datetime.now() + timedelta(days=1))
Django 1.7+
Please note that since Django 1.7, usage of lambda as default value is not recommended (c.f. #stvnw comment). The proper way to do this is to declare a function before the field and use it as a callable in default_value named arg:
from datetime import datetime, timedelta
# default to 1 day from now
def get_default_my_date():
return datetime.now() + timedelta(days=1)
class MyModel(models.Model):
my_date = models.DateTimeField(default=get_default_my_date)
More information in the #simanas answer below
[1]: https://docs.djangoproject.com/en/dev/ref/models/fields/#default
Doing this default=datetime.now()+timedelta(days=1) is absolutely wrong!
It gets evaluated when you start your instance of django. If you are under apache it will probably work, because on some configurations apache revokes your django application on every request, but still you can find you self some day looking through out your code and trying to figure out why this get calculated not as you expect.
The right way of doing this is to pass a callable object to default argument. It can be a datetime.today function or your custom function. Then it gets evaluated every time you request a new default value.
def get_deadline():
return datetime.today() + timedelta(days=20)
class Bill(models.Model):
name = models.CharField(max_length=50)
customer = models.ForeignKey(User, related_name='bills')
date = models.DateField(default=datetime.today)
deadline = models.DateField(default=get_deadline)
There's an important distinction between the following two DateTimeField constructors:
my_date = models.DateTimeField(auto_now=True)
my_date = models.DateTimeField(auto_now_add=True)
If you use auto_now_add=True in the constructor, the datetime referenced by my_date is "immutable" (only set once when the row is inserted to the table).
With auto_now=True, however, the datetime value will be updated every time the object is saved.
This was definitely a gotcha for me at one point. For reference, the docs are here:
https://docs.djangoproject.com/en/dev/ref/models/fields/#datetimefield
Sometimes you may need to access model data after creating a new user model.
Here is how I generate a token for each new user profile using the first 4 characters of their username:
from django.dispatch import receiver
class Profile(models.Model):
auth_token = models.CharField(max_length=13, default=None, null=True, blank=True)
#receiver(post_save, sender=User) # this is called after a User model is saved.
def create_user_profile(sender, instance, created, **kwargs):
if created: # only run the following if the profile is new
new_profile = Profile.objects.create(user=instance)
new_profile.create_auth_token()
new_profile.save()
def create_auth_token(self):
import random, string
auth = self.user.username[:4] # get first 4 characters in user name
self.auth_token = auth + ''.join(random.SystemRandom().choice(string.ascii_uppercase + string.digits + string.ascii_lowercase) for _ in range(random.randint(3, 5)))
You can't do that directly; the default value is evaluated when the function definition is evaluated. But there are two ways around it.
First, you can create (and then call) a new function each time.
Or, more simply, just use a special value to mark the default. For example:
from datetime import datetime
def mydate(date=None):
if date is None:
date = datetime.now()
print date
If None is a perfectly reasonable parameter value, and there's no other reasonable value you could use in its place, you can just create a new value that's definitely outside the domain of your function:
from datetime import datetime
class _MyDateDummyDefault(object):
pass
def mydate(date=_MyDateDummyDefault):
if date is _MyDateDummyDefault:
date = datetime.now()
print date
del _MyDateDummyDefault
In some rare cases, you're writing meta-code that really does need to be able to take absolutely anything, even, say, mydate.func_defaults[0]. In that case, you have to do something like this:
def mydate(*args, **kw):
if 'date' in kw:
date = kw['date']
elif len(args):
date = args[0]
else:
date = datetime.now()
print date
Pass the function in as a parameter instead of passing in the result of the function call.
That is, instead of this:
def myfunc(date=datetime.now()):
print date
Try this:
def myfunc(date=datetime.now):
print date()
I am making some view functions to calculate the rank of one user in a community.
My problem is that i want to display the rank afferent to each user, in its profile, and i don't know how, since i don;t have a request and a render_to_response (cause i guessed they are not needed)
my code:
def calculate_questions_vote(request):
useranswer = Answer.objects.filter (answer_by = request.user)
positive_votes = VoteUpAnswer.objects.filter(answer = useranswer)
negative_votes = VoteDownAnswer.objects.filter(answer = useranswer)
question_vote_rank = sum(positive_votes) - sum(negative_votes.count)
return question_vote_rank
def calculate_replies(request):
the_new = News.objects.filter(created_by = request.user)
reply = Reply.objects.filter(reply_to = the_new)
reply_rank = sum(reply)
return reply_rank
def calculate_votes(request):
the_new = News.objects.filter(created_by = request.user)
vote = Vote.objects.filter(voted = the_new)
vote_rank = sum(vote)
return vote_rank
def personal_rank(request):
personal_rank = question_vote_rank + reply_rank + vote_rank
return personal_rank
and in UserProfiles:
user = request.user
personal_rank = calculate_questions_vote(user) + calculate_replies(user) + personal_rank(user)
but my error is:
Error binding parameter 0 - probably unsupported type.
Is mt approach correct? How should i call the rank function in the profile_view def ?
Thanks!
YOu can call the function in your view like rank = personal_rank(reuest.user) and add rank to your context then.
I wouldnt call this functions "view"-functions, since they are not dealing with a request nor are they returning a HttoResponce; they are more "helper" functions and also don't belong to a model if they are dealing with seperate entities (eg. News & Vote). A descent place for them would be eg. utils.py. You import them from there in your views.py and call them with the user as parameter (if it's the actual user it's request.user).
It makes sense that you can't access request from everywhere, because this forces you to keep more to a mvc-like design, if you need request somewhere you need to pass it on from your original view function!
You should change your last function to:
def personal_rank(user):
personal_rank = calculate_questions_vote(user) + \
calculate_replies(user) + \
calculate_votes(user)
return personl_rank
You could also add this last function to your User or UserProfile model class if you have something like that and then call eg. my_user.personal_rank() or my_user.get_profile().personal_rank().
lazerscience gave good explanation for your guestion. I just want to point some possible performance issues.
Please note that calling this method for each User yields with 7 queries. If your case will be showing rank for all users it might take a lot of time. I suggest make some caching (like in StackOverflow). There are many options here, for example you can assign special rank field to a User model (or UserProfile if User is auth.User) in which you could store precalaculated rank (calculated just after some operation, like News saving or in some scheduled intervals).