I pulled some data from an API in JSON format. The data contains a Unix time stamp and information about the timezone. My question is how do I add the time zone info (in bold) at the end of the datetime object?
Combined date and time in UTC(according to ISO 8601): 2017-07-29T12:48:20+00:00
Install and import python-dateutil. Make sure to convert the time stamp into datetime (divide stamp with 1e3 if it is longer than 10 characters) and then use the dateutil package to add the time zone info at the end with tzoffset like so:
time = datetime.datetime.fromtimestamp(timestamp / 1e3).replace(tzinfo=tzoffset(None, timezone))
Related
Say I have a dataframe with a column name epoch that is seconds from epoch time and I wanted to convert it to UTC time. How to do it in Pyspark?
I dont want to harcode "GMT-7" and I want to make sure I use the right offset for the time, GMT-7 or GMT-8 based on daylight savings. I'm running Spark 2.1
myDF = myDF.withColumn("epoch_tmst",(myDF.epoch.cast(dataType=TimestampType()))
myDF = myDF.withColumn("epoch_tmst_utc",to_utc_timestamp(myDF.epoch_tmst,"GMT-7"))
There is no way to convert unixtime into UTC timestamp directly by pyspark method.
(You can do unix->GMT-7 & GMT-7->UTC separately by pyspark, but looks like that's not what you want)
How about take them into literal string once, and use Python's time module?
https://docs.python.org/3/library/time.html#time.gmtime
After spending some significant amount of time, I could not figure out how to convert UDate from one timezone to another. Here is the problem I am trying to solve :
I have a timestamp which is number of milliseconds since epoch. This is a timestamp in UTC. I want to convert this timestamp to a timestamp at some local time zone (e.g. US/Eastern). I want to extract number of days since epoch and number of milliseconds since from converted timestamp. I want to use icu library to do this.
I tried to create a UDate from number of milliseconds since epoch in UTC. I can create a timezone instance for the given timezone.
TimeZone *tz = TimeZone::createTimeZone("US/Eastern");
How do I convert UDate from UTC to the given timezone and extract the answers I want? Could it be done using icu?
Any help will be greatly appreciated.
I want … number of days since epoch and number of milliseconds since from converted timestamp.
I think these will be the same regardless of time zone, won't they?
Or can you give an example of the result you would like to see (before vs. after)
The time_orign_epoch,should be 1970-01-01 00:00:00,why i am getting 5:30 more?
time_origin_epoch = datetime.datetime.fromtimestamp(0)
print time_origin_epoch
1970-01-01 05:30:00
It's because you live in India!
How did I know that?
Well, a timestamp of 0 means 1970-01-01 00:00:00 UTC. Since your output shows 05:30:00, your time zone is UTC+05:30. And India is one of the few places in the world with a timezone offset which is not a whole number of hours.
When you construct a datetime in Python using fromtimestamp(), the default is to use your local timezone for the conversion. This corresponds to calling the classic C function localtime() rather than gmtime().
Running your code gives me a result that is five hours behind what it should be. This is consistent with the fact that I am in EST and am five hours behind UTC.
If you are simply looking to confirm correct output, then check your time zone, I would guess you are somewhere in Eastern Asia. However, if you need a zeroes value produced by the program itself, try resetting your timezone to UTC from the program, and running that snippet again.
From this answer:
If you are using Unix, you can use time.tzset to change the process's local timezone:
import os
import time
os.environ['TZ'] = tz
time.tzset()
You could then convert the datetime strings to NumPy datetime64's using
def using_tzset(date_strings, tz):
os.environ['TZ'] = tz
time.tzset()
return np.array(date_strings, dtype='datetime64[ns]')
I have a lot of dates with time in this format:
day.mon.year - hour:min:sec
And I need to convert this dates with time into Unix timestamp.
I used tm structure, but I can't fill those fields:
tm_wday
tm_yday
And I don't must I fill those field, because I don't know do this field have any effect to the value of Unix timestamp.
Help me to choose rigth way to calculate Unix timestamp.
P.S. Dates with time aren't current, they can be date of the 20-th century or future dates (to 2038 year).
P.P.S. I use OS Windows.
POSIX has a formula for exactly what you want:
http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap04.html#tag_04_15
tm_sec + tm_min*60 + tm_hour*3600 + tm_yday*86400 +
(tm_year-70)*31536000 + ((tm_year-69)/4)*86400 -
((tm_year-1)/100)*86400 + ((tm_year+299)/400)*86400
This works whenever you have a broken-down time in GMT, even if the underlying system's mktime, etc. functions do not use the same format time_t as "Unix timestamps".
If your original time is in local time, you can use mktime and gmtime to convert it to GMT using the system's notion of timezone rules. If you want to apply your own timezone offset rules, just do that manually before using the above formula.
If you are on unix, mktime() will get the second part of the timestamp. It ignores the tm_wday and tm_yday fields.
I'm reading timestamp fields from a PostgreSQL database. The timestamp column is defined as:
my_timestamp TIMESTAMP WITH TIME ZONE DEFAULT NOW()
When reading from the database, I convert it to a boost timestamp like this:
boost::posix_time::ptime pt( boost::posix_time::time_from_string( str ) );
The problem seems to be that boost::posix_time::time_from_string() ignores the timezone.
For example:
database text string == "2013-05-30 00:27:04.8299-07" // note -07 timezone
boost::posix_time::to_iso_extended_string(pt) == "2013-05-30T00:27:04.829900"
When I do arithmetic with the resulting ptime object, the time is off by exactly 7 hours. Is there something better I should be doing to not lose the timezone information?
I think you should be using boost::local_date_time, which handles time zones. There is an example in the documentation that is very similar to what you're trying to do: http://www.boost.org/doc/libs/1_41_0/doc/html/date_time/examples.html#date_time.examples.seconds_since_epoch
EDIT: Boost supports date parsing with specific formats. http://www.boost.org/doc/libs/1_40_0/doc/html/date_time/date_time_io.html#date_time.format_flags
string inp("2013-05-30 00:27:04.8299-07");
string format("%Y-%m-%d %H:%M:%S%F%Q");
date d;
d = parser.parse_date(inp,
format,
svp);
// d == 2013-05-30 00:27:04.8299-07
I originally asked this question so many years ago, I don't even remember doing it. But since then, all my database date/time code on the client side has been greatly simplified. The trick is to tell PostgreSQL the local time zone when the DB connection is first established, and let the server automatically add or remove the necessary hours/minutes when it sends back timestamps. This way, timestamps are always in local time.
You do that with a 1-time call similar to this one:
SET SESSION TIME ZONE 'Europe/Berlin';
You can also use one of the many timezone abbreviations. For example, these two lines are equivalent:
SET SESSION TIME ZONE 'Asia/Hong_Kong';
SET SESSION TIME ZONE 'HKT';
The full list of timezones can be obtained with this:
SELECT * FROM pg_timezone_names ORDER BY name;
Note: there are over 1000 timezone names to pick from!
I have more details on PostgreSQL and timezones available on this post: https://www.ccoderun.ca/programming/2017-09-14_PostgreSQL_timestamps/index.html