I'm building few dashboards in Apache superset. All my available timestamp fields are in UTC timezone. (for example fields are, class_start_time & class_end_time).
I want that in the timezone the dashboard is opened all the timestamp fields will be automatically converted.
For example, I'm opening dashboard in Norway , so the UTC data should be converted to CET timezone of Norway.
I have tried to add some value here in Hours offset but its not working.
Can you please guide how we can achieve this.?
Just for reference :
In Kibana dashboards (ELK stack) have feature to automatically convert the timezone into which it is being opened. So I need same thing in Superset.
Normally you would be able to set this with environment variables when you start the program or container. In Apache Superset, this is not possible. There is an ongoing discussion on Github about this issue. One GitHub user posts the problem and workaround, which is far from workable:
Daylight savings causes issues where users have to update datasource
timezone offset for each datasource twice per year.
So the only thing you can do is update the hours offset twice a year. To make matters even worse, if you use Postgresql, this may not even be possible due to a bug as described here.
Version:
Django 1.11
PostgreSQL 9.6 (AWS RDS)
Django settings.py:
TIME_ZONE = 'Asia/Seoul'
USE_TZ = False
My Problem:
In last year, My django app migrated database from MySQL to PostgreSQL. It was fine. But I found big trouble.
Tables that was generated in MySQL have DateTimeField, which correspond to naive timestamp field (without timezone) in PostgreSQL. But tables that was generated in PostgreSQL have DateTimeField, which correspond to aware timestamp field (with timezone). It stores all timestamp in UTC.
So My DB tables was messy with naive and aware timestamp field.
How do I fix this issue at best?
The Followings are solutions I think,
Convert naive timestamp fields to aware timestamp fields.
I think It's best practice. But naive timestamp fields are too many. and I'm afraid that I lost data due to converting.
Convert aware timestamp fields to naive timestamp fields.
I think It's one of easy ways. But I heard naive timestamp fields are not practice.
Nothing changed. Actually, I didn't know timestamp field is messy until I examine DB. Djano application has worked well now. But I worry It's big trouble in the future.
Change timezone setting in AWS RDS. Is it possible and solution?
Sorry for my poor english. Please advice.
Convert naive timestamp fields to aware timestamp fields. I think It's best practice. But naive timestamp fields are too many. and I'm afraid that I lost data due to converting.
You're correct, the best result will be to get all the naive values to timezone-aware values. Perhaps you want to ask a new question about the problem that prevents you from doing this?
I am currently developing a server using Flask/SqlAlchemy. It occurs that when an ORM model is not present as a table in the database, it is created by default by SqlAlchemy.
However when an ORM class is changed with for instance an extra column is added, these changes do not get saved in the database. So the extra column will be missing, every time I query. I have to adjust my DB manually every time there is a change in the models that I use.
Is there a better way to apply changes in the models during development? I hardly think manual MySql manipulation is the best solution.
you can proceed as the following:
new_column = Column('new_column', String, default='some_default_value')
new_column.create(my_table, populate_default=True)
you can find more details about sqlalchemy migration in: https://sqlalchemy-migrate.readthedocs.org/en/latest/changeset.html
NOTE: I deleted the question as it existed previously and providing only the relevant info here.
Our database server (RH) has TIME_ZONE = "Europe/London" specified. And, within the Django settings.py, we specify TIME_ZONE = "America/New_York".
And, in my Model class I have specified:
created = models.DateTimeField(editable=False,auto_now=False, auto_now_add=True)
modified = models.DateTimeField(editable=False,auto_now=True, auto_now_add=True)
When I then go look at the data in the admin site, I get UTC/GMT time instead of Eastern.
I thought that all time is adjusted automagically by Django since I specified "America/New_York" as Django's Time Zone.
Any help/clarification is appreciated.
Thanks
Eric
Relying on date/time 'automagic' is dangerous and these auto_add model parameters are a trap. Always understand the timezone(s) you are dealing with. Python makes this easier by attaching a tzinfo member to its datetime objects. While these objects are 'naive' by default, I encourage you to always attach tzinfo detail. Still Python needs some extra help with either python-dateutil or pytz (what I use). Here's a universal rule though - always store your datetimes in a database as UTC.
Why? Your users may be in different locals, mobile phones and laptops travel, servers are misconfigured or mirrored in different timezones. So many headaches. Datetimes should never be naive and if they are (as in a database) and you need the context, also include a timezone field in the table.
So in your case.
Don't use the auto_now fields, use a custom save() instead.
Store UTC in the database
If you need to know the timezone - for say a user event - store the timezone in the database as well.
Convert to the necessary/requested timezone
If you are using pytz, the localize() method is great. Python's datetime object has the useful replace() and astimezone().
One more note, if your database is timezone naive (like MySQL) make sure your datetimes are in UTC and then use replace(tzinfo=None) because the database connector can't handle tz-aware objects.
Here is a thread with detail on Django's auto_now fields.
The simplest/fastest fix [said above by Ajay Yadav] is this ,
Just add TIME_ZONE attribute to the Database's section in settings.py,
settings.py
# Database
# https://docs.djangoproject.com/en/3.1/ref/settings/#databases
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.sqlite3',
'NAME': BASE_DIR / 'db.sqlite3',
'TIME_ZONE': 'Asia/Tokyo',
}
}
For the available Timezone choices , See the Official documentation linked below ,
DJANGO TIMEZONE CHOICES
First off, I would want to store my data as UTC cause its a good starting point.
So let me ask this, Why do you need the time in EST, is this for the end-user, or do you need to do logic on the server and need it in EST?
If its for the enduser, an easy fix is to let the users browser handle converting to the correct time. On the server convert the datetime object to a timestamp:
timestamp = time.mktime(datetime_obj.timetuple()) * 1000
And then on the web page instantiate a Date object:
var date_obj = new Date({{ timestamp }});
var datetime_string = date_obj.toString();
// the datetime_string will be in the users local timezone
Now, on the other hand, if you want to have the time in the correct zone on the server so you can perform logic on it. I recommend using the help of python-dateutil. It will allow you to easily swap to a different timezone:
from datetime import datetime
from dateutil import zoneinfo
from_zone = zoneinfo.gettz('UTC')
to_zone = zoneinfo.gettz('America/New_York')
utc = created # your datetime object from the db
# Tell the datetime object that it's in UTC time zone since
# datetime objects are 'naive' by default
utc = utc.replace(tzinfo=from_zone)
# Convert time zone
eastern_time = utc.aztimezone(to_zone)
Now if you really wanna store the datetime in EST, you need change the time on the DB server (like Ajay Yadav and gorus said). I don't know why you want to store them as EST, but then again I don't know what your application is.
When you say auto_now_add=True, the value will be added by your database server and not your django server. So you need to set time zone on your database server.
Since you edited the question, I'll edit my answer :) Django cannot control the time zone of your db, so the way to fix this is to update the time zone for your db. For MySql, run this query:
SELECT ##global.time_zone, ##session.time_zone;
This should return SYSTEM, SYSTEM by default, which in your case means "Europe/London", and the cause of your problem. Now that you've verified this, follow the instructions in the first comment on this page:
http://dev.mysql.com/doc/refman/5.5/en/time-zone-support.html
Remember to restart MySql server after you've updated the time zone for the changes to take effect.
I have a model called job and I want to set a datetime attribute (started_time) to MySQL now() value. how can I do that in Django?
I don't want to use the model auto_now or auto_now_add methods, since I have other applications who share the same DB and I don't want to handle timezones, thus I want to delegate that to MySQL
Don't use auto_now/auto_add_now, they are problematic. Instead, do this:
started_time = models.DateTimeField(default=datetime.utcnow)
-- assuming that you're working with timestamps in UTC.
Use MySQL trigger.