Say I have a dataframe with a column name epoch that is seconds from epoch time and I wanted to convert it to UTC time. How to do it in Pyspark?
I dont want to harcode "GMT-7" and I want to make sure I use the right offset for the time, GMT-7 or GMT-8 based on daylight savings. I'm running Spark 2.1
myDF = myDF.withColumn("epoch_tmst",(myDF.epoch.cast(dataType=TimestampType()))
myDF = myDF.withColumn("epoch_tmst_utc",to_utc_timestamp(myDF.epoch_tmst,"GMT-7"))
There is no way to convert unixtime into UTC timestamp directly by pyspark method.
(You can do unix->GMT-7 & GMT-7->UTC separately by pyspark, but looks like that's not what you want)
How about take them into literal string once, and use Python's time module?
https://docs.python.org/3/library/time.html#time.gmtime
Related
I pulled some data from an API in JSON format. The data contains a Unix time stamp and information about the timezone. My question is how do I add the time zone info (in bold) at the end of the datetime object?
Combined date and time in UTC(according to ISO 8601): 2017-07-29T12:48:20+00:00
Install and import python-dateutil. Make sure to convert the time stamp into datetime (divide stamp with 1e3 if it is longer than 10 characters) and then use the dateutil package to add the time zone info at the end with tzoffset like so:
time = datetime.datetime.fromtimestamp(timestamp / 1e3).replace(tzinfo=tzoffset(None, timezone))
After spending some significant amount of time, I could not figure out how to convert UDate from one timezone to another. Here is the problem I am trying to solve :
I have a timestamp which is number of milliseconds since epoch. This is a timestamp in UTC. I want to convert this timestamp to a timestamp at some local time zone (e.g. US/Eastern). I want to extract number of days since epoch and number of milliseconds since from converted timestamp. I want to use icu library to do this.
I tried to create a UDate from number of milliseconds since epoch in UTC. I can create a timezone instance for the given timezone.
TimeZone *tz = TimeZone::createTimeZone("US/Eastern");
How do I convert UDate from UTC to the given timezone and extract the answers I want? Could it be done using icu?
Any help will be greatly appreciated.
I want … number of days since epoch and number of milliseconds since from converted timestamp.
I think these will be the same regardless of time zone, won't they?
Or can you give an example of the result you would like to see (before vs. after)
The time_orign_epoch,should be 1970-01-01 00:00:00,why i am getting 5:30 more?
time_origin_epoch = datetime.datetime.fromtimestamp(0)
print time_origin_epoch
1970-01-01 05:30:00
It's because you live in India!
How did I know that?
Well, a timestamp of 0 means 1970-01-01 00:00:00 UTC. Since your output shows 05:30:00, your time zone is UTC+05:30. And India is one of the few places in the world with a timezone offset which is not a whole number of hours.
When you construct a datetime in Python using fromtimestamp(), the default is to use your local timezone for the conversion. This corresponds to calling the classic C function localtime() rather than gmtime().
Running your code gives me a result that is five hours behind what it should be. This is consistent with the fact that I am in EST and am five hours behind UTC.
If you are simply looking to confirm correct output, then check your time zone, I would guess you are somewhere in Eastern Asia. However, if you need a zeroes value produced by the program itself, try resetting your timezone to UTC from the program, and running that snippet again.
From this answer:
If you are using Unix, you can use time.tzset to change the process's local timezone:
import os
import time
os.environ['TZ'] = tz
time.tzset()
You could then convert the datetime strings to NumPy datetime64's using
def using_tzset(date_strings, tz):
os.environ['TZ'] = tz
time.tzset()
return np.array(date_strings, dtype='datetime64[ns]')
I have a lot of dates with time in this format:
day.mon.year - hour:min:sec
And I need to convert this dates with time into Unix timestamp.
I used tm structure, but I can't fill those fields:
tm_wday
tm_yday
And I don't must I fill those field, because I don't know do this field have any effect to the value of Unix timestamp.
Help me to choose rigth way to calculate Unix timestamp.
P.S. Dates with time aren't current, they can be date of the 20-th century or future dates (to 2038 year).
P.P.S. I use OS Windows.
POSIX has a formula for exactly what you want:
http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap04.html#tag_04_15
tm_sec + tm_min*60 + tm_hour*3600 + tm_yday*86400 +
(tm_year-70)*31536000 + ((tm_year-69)/4)*86400 -
((tm_year-1)/100)*86400 + ((tm_year+299)/400)*86400
This works whenever you have a broken-down time in GMT, even if the underlying system's mktime, etc. functions do not use the same format time_t as "Unix timestamps".
If your original time is in local time, you can use mktime and gmtime to convert it to GMT using the system's notion of timezone rules. If you want to apply your own timezone offset rules, just do that manually before using the above formula.
If you are on unix, mktime() will get the second part of the timestamp. It ignores the tm_wday and tm_yday fields.
I'm using Qt to parse an XML file which contains timestamps in UTC. Within the program, of course, I'd like them to change to local time. In the XML file, the timestamps look like this: "2009-07-30T00:32:00Z".
Unfortunately, when using the QDateTime::fromString() method, these timestamps are interpreted as being in the local timezone. The hacky way to solve this is to add or subtract the correct timezone offset from this time to convert it to "true" local time. However, is there any way to make Qt realize that I am importing a UTC timestamp and then automatically convert it to local time?
Do it like this:
QDateTime timestamp = QDateTime::fromString(thestring);
timestamp.setTimeSpec(Qt::UTC); // mark the timestamp as UTC (but don't convert it)
timestamp = timestamp.toLocalTime() // convert to local time
try using the setTime_t function.
Note that full time zone support is not available yet in Qt, but probably will in future versions.
http://bugreports.qt-project.org/browse/QTBUG-10219