RuntimeWarning: DateTimeField received a naive datetime - django

I m trying to send a simple mail using IPython. I have not set up any models still getting this error. What can be done?
Error :
/home/sourabh/Django/learn/local/lib/python2.7/site-packages/django/db/models/fields/init.py:827: RuntimeWarning: DateTimeField received a naive datetime (2013-09-04 14:14:13.698105) while time zone support is active.
RuntimeWarning)
Tried : The first step is to add USE_TZ = True to your settings file and install pytz (if possible).
Error changed:
(learn)sourabh#sL:~/Django/learn/event$ python manage.py shell
/home/sourabh/Django/learn/local/lib/python2.7/site-packages/django/db/backends/sqlite3/base.py:53: RuntimeWarning: SQLite received a naive datetime (2013-09-05 00:59:32.181872) while time zone support is active.
RuntimeWarning)

The problem is not in Django settings, but in the date passed to the model. Here's how a timezone-aware object looks like:
>>> from django.utils import timezone
>>> import pytz
>>> timezone.now()
datetime.datetime(2013, 11, 20, 20, 8, 7, 127325, tzinfo=pytz.UTC)
And here's a naive object:
>>> from datetime import datetime
>>> datetime.now()
datetime.datetime(2013, 11, 20, 20, 9, 26, 423063)
So if you are passing email date anywhere (and it eventually gets to some model), just use Django's now(). If not, then it's probably an issue with an existing package that fetches date without timezone and you can patch the package, ignore the warning or set USE_TZ to False.

Use django.utils.timezone.make_aware function to make your naive datetime objects timezone aware and avoid those warnings.
It converts naive datetime object (without timezone info) to the one that has timezone info (using timezone specified in your django settings if you don't specify it explicitly as a second argument):
import datetime
from django.conf import settings
from django.utils.timezone import make_aware
naive_datetime = datetime.datetime.now()
naive_datetime.tzinfo # None
settings.TIME_ZONE # 'UTC'
aware_datetime = make_aware(naive_datetime)
aware_datetime.tzinfo # <UTC>

Just to fix the error to set current time
from django.utils import timezone
import datetime
datetime.datetime.now(tz=timezone.utc) # you can use this value

Quick and dirty - Turn it off:
USE_TZ = False
in your settings.py

make sure settings.py has
USE_TZ = True
In your python file:
from django.utils import timezone
timezone.now() # use its value in model field

One can both fix the warning and use the timezone specified in settings.py, which might be different from UTC.
For example in my settings.py I have:
USE_TZ = True
TIME_ZONE = 'Europe/Paris'
Here is a solution; the advantage is that str(mydate) gives the correct time:
>>> from datetime import datetime
>>> from django.utils.timezone import get_current_timezone
>>> mydate = datetime.now(tz=get_current_timezone())
>>> mydate
datetime.datetime(2019, 3, 10, 11, 16, 9, 184106,
tzinfo=<DstTzInfo 'Europe/Paris' CET+1:00:00 STD>)
>>> str(mydate)
'2019-03-10 11:16:09.184106+01:00'
Another equivalent method is using make_aware, see dmrz post.

If you are trying to transform a naive datetime into a datetime with timezone in django, here is my solution:
>>> import datetime
>>> from django.utils import timezone
>>> t1 = datetime.datetime.strptime("2019-07-16 22:24:00", "%Y-%m-%d %H:%M:%S")
>>> t1
datetime.datetime(2019, 7, 16, 22, 24)
>>> current_tz = timezone.get_current_timezone()
>>> t2 = current_tz.localize(t1)
>>> t2
datetime.datetime(2019, 7, 16, 22, 24, tzinfo=<DstTzInfo 'Asia/Shanghai' CST+8:00:00 STD>)
>>>
t1 is a naive datetime and t2 is a datetime with timezone in django's settings.

You can also override settings, particularly useful in tests:
from django.test import override_settings
with override_settings(USE_TZ=False):
# Insert your code that causes the warning here
pass
This will prevent you from seeing the warning, at the same time anything in your code that requires a timezone aware datetime may give you problems. If this is the case, see kravietz answer.

In the model, do not pass the value:
timezone.now()
Rather, remove the parenthesis, and pass:
timezone.now
If you continue to get a runtime error warning, consider changing the model field from DateTimeField to DateField.

If you need to convert the actual date string to date object, I have got rid of the warning by simply using astimezone:
>>> from datetime import datetime, timezone
>>> datetime_str = '2013-09-04 14:14:13.698105'
>>> datetime_object = datetime.strptime(datetime_str, "%Y-%m-%d %H:%M:%S.%f")
>>> datetime_object.astimezone(timezone.utc)
datetime.datetime(2013, 9, 4, 6, 14, 13, 698105, tzinfo=datetime.timezone.utc)

I encountered this warning when using the following model.
from datetime import datetime
class MyObject(models.Model):
my_date = models.DateTimeField(default=datetime.now)
To fix it, I switched to the following default.
from django.utils import timezone
class MyObject(models.Model):
my_date = models.DateTimeField(default=timezone.now)

Related

Django Infinity as datetime default

I'm struggling to see how I can add a datetime field with an infinity end date.
Setting the default to 'infinity' results in a Django.core exception
django.core.exceptions.ValidationError: ["'infinity' value has an invalid format. It must be in YYYY-MM-DD HH:MM[:ss[.uuuuuu]][TZ] format."]
NB: This is not the same as defining it Null (None). It's specially supported in postgres as just a string condition check for example
SELECT * FROM table WHERE dt = 'infinity'; // or
SELECT * FROM table WHERE NOT isfinite(dt);
Nulls won't show in this, nor can you do a between condition on a Null value unless you COALESCE it, but that will result in a sequential scan.
Any ideas?
Django ORM converts infinity to datetime.max, so consider using datetime.max instead:
$ psql -d yourdb
yourdb=# UPDATE app_yourmodel SET last_login = 'infinity' WHERE id = 1;
$ python3 manage.py shell
>>> from app.models import YourModel
>>> from datetime import datetime
>>>
>>> YourModel.objects.get(id=1).last_login
datetime.datetime(9999, 12, 31, 23, 59, 59, 999999, tzinfo=<UTC>)
>>>
>>> datetime.max
datetime.datetime(9999, 12, 31, 23, 59, 59, 999999)

How to Change the time zone in Python logging?

I would like to change the timestamp in the log file so that it reflects my current time zone so that i can debug errors at a faster rate,
is it possible that i can change the time zone in the log file ?
currently my config is:
logging.basicConfig(filename='audit.log',
filemode='w',
level=logging.INFO,
format='%(asctime)s %(message)s',
datefmt='%m/%d/%Y %I:%M:%S %p')
How to log the timezone
%Z from strftime format
Windows
>>> import logging
>>> logging.basicConfig(format="%(asctime)s %(message)s", datefmt="%m/%d/%Y %I:%M:%S %p %Z")
>>> logging.error('test')
11/03/2017 02:29:54 PM Mountain Daylight Time test
Linux
>>> import logging
>>> logging.basicConfig(format="%(asctime)s %(message)s", datefmt="%m/%d/%Y %I:%M:%S %p %Z")
>>> logging.error('test')
11/03/2017 02:30:50 PM MDT test
If the question is
How do I log in a different timezone than the local time on the server?
part of the answer is logging.Formatter.converter, however, you have to understand naive and aware datetime objects. Unless you want to write your own timezone module, I highly suggest the pytz library (pip install pytz). Python 3 includes a UTC and UTC offset timezone, but there's rules you'll have to implement for daylight saving or other offsets, so I would suggest the pytz library, even for python 3.
For example,
>>> import datetime
>>> utc_now = datetime.datetime.utcnow()
>>> utc_now.isoformat()
'2019-05-21T02:30:09.422638'
>>> utc_now.tzinfo
(None)
If I apply a timezone to this datetime object, the time won't change (or will issue a ValueError for < python 3.7ish).
>>> mst_now = utc_now.astimezone(pytz.timezone('America/Denver'))
>>> mst_now.isoformat()
'2019-05-21T02:30:09.422638-06:00'
>>> utc_now.isoformat()
'2019-05-21T02:30:09.422638'
However, if instead, I do
>>> import pytz
>>> utc_now = datetime.datetime.now(tz=pytz.timezone('UTC'))
>>> utc_now.tzinfo
<UTC>
now we can create a properly translated datetime object in whatever timezone we wish
>>> mst_now = utc_now.astimezone(pytz.timezone('America/Denver'))
>>> mst_now.isoformat()
'2019-05-20T20:31:44.913939-06:00'
Aha! Now to apply this to the logging module.
Epoch timestamp to string representation with timezone
The LogRecord.created attribute is set to the time when the LogRecord was created (as returned by time.time()), from the time module. This returns a timestamp (seconds since the epoch). You can do your own translation to a given timezone, but again, I suggest pytz, by overriding the converter.
import datetime
import logging
import pytz
class Formatter(logging.Formatter):
"""override logging.Formatter to use an aware datetime object"""
def converter(self, timestamp):
dt = datetime.datetime.fromtimestamp(timestamp)
tzinfo = pytz.timezone('America/Denver')
return tzinfo.localize(dt)
def formatTime(self, record, datefmt=None):
dt = self.converter(record.created)
if datefmt:
s = dt.strftime(datefmt)
else:
try:
s = dt.isoformat(timespec='milliseconds')
except TypeError:
s = dt.isoformat()
return s
Python 3.5, 2.7
>>> logger = logging.root
>>> handler = logging.StreamHandler()
>>> handler.setFormatter(Formatter("%(asctime)s %(message)s"))
>>> logger.addHandler(handler)
>>> logger.setLevel(logging.DEBUG)
>>> logger.debug('test')
2019-05-20T22:25:10.758782-06:00 test
Python 3.7
>>> logger = logging.root
>>> handler = logging.StreamHandler()
>>> handler.setFormatter(Formatter("%(asctime)s %(message)s"))
>>> logger.addHandler(handler)
>>> logger.setLevel(logging.DEBUG)
>>> logger.debug('test')
2019-05-20T22:29:21.678-06:00 test
Substitute America/Denver with America/Anchorage for the posix timezone as defined by pytz
>>> next(_ for _ in pytz.common_timezones if 'Alaska' in _)
'US/Alaska'
US/Alaska is deprecated
>>> [_ for _ in pytz.all_timezones if 'Anchorage' in _]
['America/Anchorage']
Local
If you got to this question and answers looking for how to log the local timezone, then instead of hardcoding the timezone, get tzlocal (pip install tzlocal) and replace
tzinfo = pytz.timezone('America/Denver')
with
tzinfo = tzlocal.get_localzone()
Now it will work on whatever server runs the script, with the timezone on the server.
Caveat when not logging UTC
I should add, depending on the application, logging in local time zones can create ambiguity or at least confusion twice a year, where 2 AM is skipped or 1 AM repeats, and possibly others.
#!/usr/bin/python
from datetime import datetime
from pytz import timezone
import logging
def timetz(*args):
return datetime.now(tz).timetuple()
tz = timezone('Asia/Shanghai') # UTC, Asia/Shanghai, Europe/Berlin
logging.Formatter.converter = timetz
logging.basicConfig(
format="%(asctime)s %(levelname)s: %(message)s",
level=logging.INFO,
datefmt="%Y-%m-%d %H:%M:%S",
)
logging.info('Timezone: ' + str(tz))
Using pytz to define a timezone relative to UTC.
Based on the example by: secsilm
#!/usr/bin/env python
from datetime import datetime
import logging
import time
from pytz import timezone, utc
def main():
logging.basicConfig(format="%(asctime)s %(message)s",
datefmt="%Y-%m-%d %H:%M:%S")
logger = logging.getLogger(__name__)
logger.error("default")
logging.Formatter.converter = time.localtime
logger.error("localtime")
logging.Formatter.converter = time.gmtime
logger.error("gmtime")
def customTime(*args):
utc_dt = utc.localize(datetime.utcnow())
my_tz = timezone("US/Eastern")
converted = utc_dt.astimezone(my_tz)
return converted.timetuple()
logging.Formatter.converter = customTime
logger.error("customTime")
# to find the string code for your desired tz...
# print(pytz.all_timezones)
# print(pytz.common_timezones)
if __name__ == "__main__":
main()
Ostensibly the pytz package is the blessed way of converting time zones in Python. So we start with datetime, convert, then get the (immutable) time_tuple to match return type of the time methods
Setting the logging.Formatter.converter function is recommended by this answer: (Python logging: How to set time to GMT).
Find your favorite TZ code by uncommenting the end lines
just add this pythonic line to your code (using pytz and datetime):
from pytz import timezone
from datetime import datetime
import logging
logging.Formatter.converter = lambda *args: datetime.now(tz=timezone('tz string name')).timetuple()
# quoting Ryan J McCall: to find the string name for your desired timezone...
# print(pytz.all_timezones)
# or print(pytz.common_timezones)
An alternative solution if you want to use logging configuration function:
import pytz
import logging
import logging.config
from datetime import datetime
tz = pytz.timezone('Asia/Tokyo')
class TokyoFormatter(logging.Formatter):
converter = lambda *args: datetime.now(tz).timetuple()
LOGGING = {
'version': 1,
'disable_existing_loggers': False,
'formatters': {
'Tokyo': {
'()': TokyoFormatter,
'format': '%(asctime)s %(levelname)s: %(message)s',
'datefmt': '%Y-%m-%d %H:%M:%S'
},
},
'handlers': {
'console': {
'class': 'logging.StreamHandler',
'formatter': 'Tokyo'
},
},
'loggers': {
'foo': {
'handlers': ['console'],
'level': 'INFO'
},
}
}
logging.config.dictConfig(LOGGING)
logger = logging.getLogger('foo')
logger.info('Just a test.')
Define the logging formatter, e.g., "TokyoFormatter". It has an attibute "converter", finishing the job of converting the time zone.
For more details, please refer to Customizing handlers with dictConfig().
import logging, time
from datetime import datetime, timedelta
logger = logging.getLogger(__name__)
converter = lambda x, y: (datetime.utcnow() - timedelta(
hours=7 if time.localtime().tm_isdst else 6)
).timetuple()
logging.Formatter.converter = converter
Edited as Elias points out the original answer didn't check for DST.
If you know your utc offset, you can define a function to correct the time and then pass it to logging.Formatter.converter.
For example, you want to convert the time to UTC+8 timezone, then:
import logging
import datetime
def beijing(sec, what):
'''sec and what is unused.'''
beijing_time = datetime.datetime.now() + datetime.timedelta(hours=8)
return beijing_time.timetuple()
logging.Formatter.converter = beijing
logging.basicConfig(
format="%(asctime)s %(levelname)s: %(message)s",
level=logging.INFO,
datefmt="%Y-%m-%d %H:%M:%S",
)
Just change the hours in datetime.timedelta(hours=8) depending on your situation.
Reference: https://alanlee.fun/2019/01/06/how-to-change-logging-date-timezone/

backtest with local data in zipline

I am using zipline to backtest with the local data, but it seems unsuccessful.
from datetime import datetime
import pytz
import pandas as pd
from zipline.algorithm import TradingAlgorithm
import zipline.utils.factory as factory
class BuyApple(TradingAlgorithm):
def handle_data(self, data):
self.order('AAPL', 1)
if __name__ == '__main__':
data = pd.read_csv('AAPL.csv')
simple_algo = BuyApple()
results = simple_algo.run(data)
above is my code, When I run this script, I got the message:
[2015-04-03 01:41:53.712035] WARNING: Loader: No benchmark data found for date range.
start_date=2015-04-03 00:00:00+00:00, end_date=2015-04-03 01:41:53.632300, url=http://ichart.finance.yahoo.com/table.csv?a=3&c=2015&b=3&e=3&d=3&g=d&f=2015&s=%5EGSPC
Traceback (most recent call last):
File "bollinger.py", line 31, in <module>
results = simple_algo.run(data)
File "/home/xinzhou/.local/lib/python2.7/site-packages/zipline-0.7.0-py2.7.egg/zipline/algorithm.py", line 372, in run
source = DataFrameSource(source)
File "/home/xinzhou/.local/lib/python2.7/site-packages/zipline-0.7.0-py2.7.egg/zipline/sources/data_frame_source.py", line 42, in __init__
assert isinstance(data.index, pd.tseries.index.DatetimeIndex)
AssertionError
Then I change my code to below:
from datetime import datetime
import pytz
import pandas as pd
from zipline.algorithm import TradingAlgorithm
import zipline.utils.factory as factory
class BuyApple(TradingAlgorithm):
def handle_data(self, data):
self.order('AAPL', 1)
if __name__ == '__main__':
start = datetime(2000, 1, 9, 14, 30, 0, 0, pytz.utc)
end = datetime(2001, 1, 10, 21, 0, 0, 0, pytz.utc)
data = pd.read_csv('AAPL.csv', parse_dates=True, index_col=0)
sim_params = factory.create_simulation_parameters(
start=start, end=end, capital_base=10000)
sim_params.data_frequency = '1d'
sim_params.emission_rate = '1d'
simple_algo = BuyApple()
results = simple_algo.run(data)
The
assert isinstance(data.index, pd.tseries.index.DatetimeIndex)
AssertionError
is gone. But in my terminal, it keeps in this message:
[2015-04-03 01:44:28.141657] WARNING: Loader: No benchmark data found for date range.
start_date=2015-04-03 00:00:00+00:00, end_date=2015-04-03 01:44:28.028243, url=http://ichart.finance.yahoo.com/table.csv?a=3&c=2015&b=3&e=3&d=3&g=d&f=2015&s=%5EGSPC
How to solve this problem? Thanks.
data.index=pd.to_datetime(data.index)
data.index=data.index.tz_localize(pytz.utc)
The next code works for me.Is a version of the tutorial example "My first Algorithm" (http://www.zipline.io/tutorial/) .Data must be in ascending order by date. Run as a normal python program( python yourfilename.py):
import pytz
from datetime import datetime
from zipline.algorithm import TradingAlgorithm
from zipline.api import order, record, symbol
import pandas as pd
# Load data manually csv
#Date,Open,High,Low,Close,Volume,Adj Close
#1984-09-07,26.5,26.87,26.25,26.5,2981600,3.02
#...
parse = lambda x: pytz.utc.localize(datetime.strptime(x, '%Y-%m-%d'))
data=pd.read_csv('aapl.csv', parse_dates=['Date'], index_col=0,date_parser=parse)
# Define algorithm
def initialize(context):
pass
def handle_data(context, data):
order('Close',10)
record(AAPL=data['Close'])
# Create algorithm object passing in initialize and
# handle_data functions
algo_obj = TradingAlgorithm(initialize=initialize,
handle_data=handle_data)
# Run algorithm
perf_manual = algo_obj.run(data)
# Print
perf_manual.to_csv('output.csv'

Not naive datetime (tzinfo is already set)

I use django celery in my django app and I'm experiencing this error:
ValueError: Not naive datetime (tzinfo is already set) in djcelery.utils in make_aware
How to fix this?
I think it's a bug from django-celery when they upgrade their version. But by the way, put this in your init.py where your settings.py located.
# Patch the djcelery/snapshot cause it's broken
import datetime
from django.utils.timezone import is_aware
import djcelery.snapshot
orig_maybe_make_aware = djcelery.snapshot.maybe_make_aware
def new_maybe_make_aware(value):
if isinstance(value, datetime.datetime) and is_aware(value):
return value
return orig_maybe_make_aware(value)
djcelery.snapshot.maybe_make_aware = new_maybe_make_aware

Unix timestamp to datetime in django with timezone

I have a javascript calendar that is sending me a unixtimestamp. I am in Singapore. I want this timestamp to be interpreted as a Singapore timestamp and then converted to utc for comparisons with the db.
I cant, for the life of myself, figure out how to tell django that this time stamp is from the current timezone, Singapore.
When i do a print statement of the timestamp, it adds 8 hours to the time (which means that django thinks I input the time in utc and is localizing it to the Singaporean context)
Among many other things, I tried:
start=datetime.datetime.fromtimestamp(int(start_date)).replace(tzinfo=get_current_timezone())
The start_date is 1325376000 (which translates to 2012-01-01 00:00:00)
However,when i print the output of this I get 2012-01-01 08:00:00+06:55. I dont even know where +06:55 is coming from when singapore is +08:00. I am SO lost.
Thanks for your help.
settings.py:
TIME_ZONE = 'Asia/Singapore'
USE_TZ = True
all methods above are valide, but not "django like".
Here is a simple example, how a django programmer would do that:
from datetime import datetime
from django.utils.timezone import make_aware
# valid timestamp
value = 1531489250
# you can pass the following obj to a DateTimeField, when your settings.USE_TZ == True
datetime_obj_with_tz = make_aware(datetime.fromtimestamp(value))
See more utilites on the Django github timezone module to get whole overview...
Assuming you've got pytz installed:
from datetime import datetime
import pytz
local_tz = pytz.timezone("Asia/Singapore")
utc_dt = datetime.utcfromtimestamp(timestamp).replace(tzinfo=pytz.utc)
local_dt = local_tz.normalize(utc_dt.astimezone(local_tz))
For example:
>>> from datetime import datetime
>>> import pytz
>>> local_tz = pytz.timezone("Asia/Singapore")
>>> utc_dt = datetime.utcfromtimestamp(1325376000).replace(tzinfo=pytz.utc)
>>> utc_dt
datetime.datetime(2012, 1, 1, 0, 0, tzinfo=<UTC>)
>>> local_dt = local_tz.normalize(utc_dt.astimezone(local_tz))
>>> local_dt
datetime.datetime(2012, 1, 1, 8, 0, tzinfo=<DstTzInfo 'Asia/Singapore' SGT+8:00:00 STD>)
>>> local_dt.replace(tzinfo=None)
datetime.datetime(2012, 1, 1, 8, 0)
Pass the pytz tzinfo object to fromtimestamp() method:
#!/usr/bin/env python
from datetime import datetime
import pytz # $ pip install pytz
tz = pytz.timezone("Asia/Singapore")
print(datetime.fromtimestamp(1325376000, tz))
# -> 2012-01-01 08:00:00+08:00
Note: the result object is timezone-aware: you could compare it with other aware datetime objects i.e., you don't need to convert it to UTC for comparison -- you can use it as is.
I dont even know where +06:55 is coming from when singapore is +08:00.
You see +06:55 due to the invalid .replace() call. get_current_timezone() returns pytz.timezone("Asia/Singapore") that has a variable utc offset (it may have a different utc offset at different dates). When you call .replace() some random (depends on the implementation) tzinfo object is used. The issue is that .replace() method does not allow pytz.timezone("Asia/Singapore") to choose the correct tzinfo for the input date.
>>> list(tz._tzinfos.values())
[<DstTzInfo 'Asia/Singapore' MALT+7:00:00 STD>,
<DstTzInfo 'Asia/Singapore' MALT+7:20:00 STD>,
<DstTzInfo 'Asia/Singapore' JST+9:00:00 STD>,
<DstTzInfo 'Asia/Singapore' SMT+6:55:00 STD>,
<DstTzInfo 'Asia/Singapore' SGT+7:30:00 STD>,
<DstTzInfo 'Asia/Singapore' MALT+7:30:00 STD>,
<DstTzInfo 'Asia/Singapore' MALST+7:20:00 DST>,
<DstTzInfo 'Asia/Singapore' LMT+6:55:00 STD>,
<DstTzInfo 'Asia/Singapore' SGT+8:00:00 STD>]
i.e., both +06:55 and +0800 are valid (at different dates) for Singapore. That is why you should use .replace() only with timezones that have a constant utc offset such as the utc timezone itself (the offset is zero, always for any date).
fromtimestamp(,tz) method calls tz.fromutc() internally that allows tz to choose the correct offset for a given utc time.