I have faced this problem in other projects, but right now we're working in Django / Python with Vue.js and PostgreSQL.
Nearly every 'date' field in our model is truly a date, not a datetime. The business users don't care about a time-part, in fact storing any such value is misleading. A good example is the effective date on a tax rate. From the user point of view, it takes effect at midnight on the specified date.
If we store this as a Django DateTimeField or DateField, it requires a time-part. It could be 0000h, but the value has no meaning to the business, because they will never need to see or compare anything with the time-part. The Django functionality is great if we want to store a real datetime, put it into the database as UTC, and then display it to users in their local time through the automatic time zone conversion. But we don't. If the effective date is March 1, 2019, it should display as such regardless of the user's timezone. If the date is stored as a datetime (2019, 3, 1, 0, 0, 0) and entered by someone in Vancouver, it will appear as the next calendar day for another user in Toronto. Definitely not what we want. We could kludge it by setting the time-part to 1200h, but really?
We also have potential problems, depending on the internal representation in the database, when using SQL or tools that access the schema directly (e.g. BI tools). How do we know what time zone applies to the datetime value?
So, we're thinking of using Django CharField with ISO 8601 format (YYYY-MM-DD) instead. It will sort properly, it can be compared easily (or directly in some tools like SQL), and can be displayed without reformatting if the client is willing to use the standard. If we need to do date arithmetic, we can use the Python Standard Libraries datetime and calendar to convert from/to string. We'll need to use those to catch SQL injection attacks anyway.
We will also need to deal with date entry through a Datepicker, converting to the ISO 8601 string before storing and back again when displaying for edit.
It appears to be a better way to represent what the business needs, and it gets rid of any timezone conversion issues.
There is certainly a lot of comment on datetime and time zone handling, but I haven't found anyone taking this approach to storing true dates. Am I missing an important 'gotcha'? We're early enough in the project that we can go either way, so I'm hoping to confirm that this will work before refactoring becomes a big job.
Have you considered using DateField?
This will only store the date part and not the time.
Related
Our company has been a long-time user of Django -- for more than 10 years. We are currently running Django 1.11.26. We've dealt with a lot of datetime-related issues over the years, but our current issue is proving challenging.
Our goal is to do the following:
use Django's timezone support (setting USE_TZ=True) with UTC as default time zone
for a couple of denormalized fields (daily rollups of energy data) store naive datetimes
As is documented in many Stack Overflow questions, Django will issue a warning with datetime field values set to a naive datetime:
RuntimeWarning: DateTimeField received a naive datetime...while time zone support is active.
We think our use case for storing naive datetime is a reasonable one. While for almost all our datetime fields it make sense to use UTC, for certain kinds of daily rollups we want to store a naive/agnostic datetime indicating the start datetime of the day in the local time zone (multiple time zones for different objects). By using a naive datetime, we are able to use datetime-related filters. Therefore if we are summarizing certain kinds of energy rollups for a given datetime, we can find the rollup for the same local datetime for buildings in any time zone (Los Angeles, Chicago, Boston) by filtering in various ways on that naive local datetime.
Here are some things we've tried:
Saving the local/naive datetime as an aware datetime with the given local timezone. This doesn't work because Django will always convert to the UTC equivalent. And we can understand why that happens, in part because MySQL 5.6 DATETIME fields are not timezone-aware.
Overriding a rollup model's save method with override_settings(USE_TZ=False). This does suppress the runtime warnings about a naive datetime, but it forces all datetimes in the model to be saved as naive. And it seems hackish, since override_settings is intended for testing.
The only workaround we have right now is to tell Django that this loca/naive datetime field is UTC -- so no time conversion is done when saving and no runtime warnings are emitted because it is an aware datetime. But it ultimately is wrong. These are not UTC datetimes. They are intentionally naive, denormalized datetimes. And we are concerned that could bite us, especially as Django evolves its time zone support.
Are there other possible solutions for this issue we haven't considered? Thanks in advance for considering the question.
To filter or suppress warnings, you can use warnings.filterwarnings:
You can try adding something like this to settings.py to filter the warning:
from warnings import filterwarnings
filterwarnings('ignore', message=r'.*received a naive datetime')
You may need to fiddle with the filterwarnings call to suppress the warning as needed.
I want to work on a django project that work with time(track time of user), but I don't know to choose datetime field or timezone field or epoch time in an integer field for project model. I worked with all of them and I know we can convert them to each other. My question is for saving time in db, process on time and represent which one is recomended and why? What I need to consider for choosing between them?
The DateTimeField class would be my choice, since it supports timezones, something a simple IntegerField will not be able to do. If you don't handle timezones, there's a possibility that your computations will be wrong, and you will not be able to display dates and times that are correctly localized for your users.
Also, since a DateTimeField gets you a standard Python datetime, it works well for date computations, especially if you use a library like arrow to simplify their handling.
I have Django 1.9.5 installed with MariaDB 10.1.13 (also tested with MySQL 5.6.30) (which was updated from MySQL 5.5) and I am trying to get fractional seconds support in a date time field.
I have created a test model to try and get this working, here is the definition
class History(models.Model):
date = models.DateTimeField(null=True)
then in the shell I have run the following
History(date=datetime.now()).save()
and then when I query it
type(History.objects.get(id=1).date)
I get
<type 'NoneType'>
even though the entry appears in the database.
I can also use the field in a query
History.objects.all().order_by('date')
I know it works because inspecting the data shows that the order has changed
But I need to be able to return the date so I can compare it with another.
I was using the MySQL 5.5 without fractional support but there are records in my database that have the same datetime field and thus the order_by didnt work, I was using order_by id and that worked whilst the records were entered chronologically, but now this isnt the case I need fractional support.
Any ideas?
For this sort of situation it might be better to store the date and time as a unix timestamp in the database. There you millisecond accuracy. You can convert to python datetime objects at anytime with fromtimestamp.
Last but not least, you can easily convert unix timestamps with javascript to date times in the user's own timezone. A process that's far more complex with django or any other server side tech.
In my application I use time zones (USE_TZ=True) and all the dates I create in my code are aware UTC datetime objects (I use django.util.timezone.now for the current date and the following helper function to ensure all the dates in my instances are what I expect:)
#classmethod
def asUTCDate(cls, date):
if not timezone.is_aware(date):
return timezone.make_aware(date, timezone.utc)
return date.replace(tzinfo=timezone.utc)
I also enforced the check of naive/aware dates using this snippet (like suggested in the doc):
import warnings
warnings.filterwarnings(
'error', r"DateTimeField .* received a naive datetime",
RuntimeWarning, r'django\.db\.models\.fields')
As I understood so far, this is the right way to proceed (this is a quote from the django documentation: "The solution to this problem is to use UTC in the code and use local time only when interacting with end users."), and it seems that my app is handling dates very well… but I have just implemented a filter against a model that makes use of the Django 1.6 __hour and it force the extraction based on the user timezone, the result is something like:
django_datetime_extract('hour', "object"."date", Europe/Rome) = 15
but this breaks my query, since some results I was expecting are not included in the set, but when I use a __range to search between dates it seems to work as expected (objects with a date in the range are returned)… so it seems to me that Django takes into account timezones in queries only for the __hour filter… but I don't understand why… I was supposing that UTC is used everywhere except in templates where the displayed dates are formatted according to user tz, but maybe that's not true.
So my questions are: is the way I'm working with time zones right? Is __hour filter wrong or what?
It seems as though you're doing the right thing with dates. However, all the documentation for any date related functionality, such as filtering by hour include this note:
When USE_TZ is True, datetime fields are converted to the current time
zone before filtering.
For the range filter, this note doesn't exist because range can be used to not only filter on dates, but other types as well such as integers and characters. ie It is not necessarily datetime aware.
In essence the problem comes down to this: where do you draw the line between 'interacting with users' where times are in a local timezone, and what is internal where times are in UTC? In your case, you could imagine a user entering in a search box to search for hour==3. Does that mean for example that your form code should do the conversion between hour==3 and the UTC equivalent? This would then require a special forms.HourField. Or perhaps the value (3) should be fed directly to the query where we know that we're searching on an hour field and so a conversion is required.
We really have to follow the documentation on this one.
Values which are going to be filtered against date/time fields using any of the specialised date/time filtering functions will be treated as being in the user's local time zone.
If using the range filter for dates no time conversions occur so you are expected to convert the user's entered local time value to UTC.
I've had this question for a long time. The problem is this: most SQL servers I've worked with (namely MySQL) don't store timezone information with dates, so I assume they simply store dates as relative to the server's local timezone. This creates an interesting problem in the case where I'll have to migrate servers to different timezones, or if I create a cluster with servers spread out over different datacenters, or if I need to properly translate date/times into local values.
For example, if I persist a date like 2011-08-12 12:00:00 GMT-7, it is persisted as 2011-08-12 12:00:00. If I have an event which happens across the world at a specific time, I have to assume that my server is storing dates in GMT-0700, (let's not even add daylight savings time into the mix) then translate them into dates depending on each user's local timezone. If I have multiple servers storing dates in their own timezones, all of this fails miserably.
For frameworks like Hibernate and Django, how do they deal with this problem, if at all? Am I missing something, or is this a significant problem?
As I see it, the best choices are:
Convert all times to UTC when storing them in the database, and localize them from UTC for display
Store the UTC offset in minutes (at least one modern time zone is a multiple of ten minutes offset from UTC) in a separate column from the date/time
Store the timestamp as a string
In my current project we encountered this issue (we use Postgres) and decided to store all times in UTC and convert as needed in the application. No separate storage of the time zone for us. We also decided that all client-server interaction using timestamps would be in UTC, and that local time zones would ONLY be considered for user interaction. This has worked out well so far.
Since you tagged this question with Django, I'll add that the pytz module is extremely useful for dealing with locale timezone conversion.
Your answer for MySQL lies on this page MySQL Server Time Zone Support
Basically MySQL offers automatic timezone support for any fields that use UTC (timestamp) field but not for fields that don't (date, time, and datetime fields). For UTC fields you can set the timezone from the client using SET time_zone = timezone;. For nonUTC fields you must calculate it yourself.
You are so right I've often run into this and tend to look the TZ up and store it in a static "VARS" table so at least I can move it later.
Side notes:
Interesting the DATETIME manual doesn't even mention it
It is affected by NOW() and CURTIME() as shown here
HTH