How to get next minute from current_timestamp in presto/athena
Eg.
2021-07-27 12:29:52.951 UTC -> 2021-07-27 12:30:00.000 UTC
2021-07-27 12:29:25.951 UTC -> 2021-07-27 12:30:00.000 UTC
There's no built-in function for that, but you can do it by adding 1 minute to the timestamp and then using date_trunc to round down to the nearest minute:
WITH data(ts) AS (
VALUES
TIMESTAMP '2021-07-27 12:29:52.951 UTC',
TIMESTAMP '2021-07-27 12:29:25.951 UTC'
)
SELECT ts, date_trunc('minute', ts + INTERVAL '1' MINUTE)
FROM data
=>
ts | _col1
-----------------------------+-----------------------------
2021-07-27 12:29:52.951 UTC | 2021-07-27 12:30:00.000 UTC
2021-07-27 12:29:25.951 UTC | 2021-07-27 12:30:00.000 UTC
(2 rows)
as part of ETL to Redshift, in one of the source tables, there are 2 columns:
original_timestamp - TIMESTAMP: which is the local time when the record was inserted in whichever region
original_timezone_offset - Varchar: which is the offset to UTC
The data looks something like this:
original_timestamp original_timezone_offset
2011-06-22 11:00:00.000000 -0700
2014-11-29 17:00:00.000000 -0800
2014-12-02 22:00:00.000000 +0900
2011-06-03 09:23:00.000000 -0700
2011-07-28 03:00:00.000000 -0700
2011-05-01 01:30:00.000000 -0700
In my target table, I need to convert this to UTC (using the offset). How do I do it?
So far I have tried multiple things but dateadd() seems to be the closest solution. But the problem with dateadd() is, when I say:
SELECT original_timestamp, original_timezone_offset
,dateadd(H, original_timezone_offset, original_timestamp) as original_utc_time
it is adding/subtracting '700'/'800' hours instead of 7/8 hrs to the original timestamp because the offset is a VARCHAR and the values are like: -0700 etc.
Did anyone see this issue before? Appreciate any help/inputs. Thanks.
Just take the 'hours' part of the offset:
WITH t as (
SELECT '2011-06-22 11:00:00.000000'::timestamp as original_timestamp, '-0700' as original_timezone_offset
UNION ALL
SELECT '2014-11-29 17:00:00.000000'::timestamp,'-0800'
UNION ALL
SELECT '2014-12-02 22:00:00.000000'::timestamp,'+0900'
)
SELECT
original_timestamp,
original_timezone_offset,
DATEADD(hour, SUBSTRING(original_timezone_offset, 1, 3)::INT, original_timestamp)
FROM t
2011-06-22 11:00:00 -0700 2011-06-22 04:00:00
2014-11-29 17:00:00 -0800 2014-11-29 09:00:00
2014-12-02 22:00:00 +0900 2014-12-03 07:00:00
You'll need some additional fancy code if you have non-full-hour offsets (eg +0730).
First, recognize that if your timestamps are already in local time of the given offset, then you need to subtract that offset to convert back to UTC. In that first example you gave, 2011-06-22 11:00:00 -0700 is equivalent to 2011-06-22 18:00:00 UTC.
However, rather than try to add or subtract these values yourself, you should let the AT TIME ZONE function do the work for you. It will create a timestamptz that is in your supplied offset, then you can use it again to convert to UTC.
(Note that you could use the CONVERT_TIMEZONE function instead, but that one is only understood by Redshift, where AT TIME ZONE works on regular PostgreSQL also.)
However, you have is that the time zone offsets you have aren't in a format understood by these functions. See time zone usage notes. So, before we try to convert, let's translate your offset strings to an understood format.
We will want -0700 to become +07:00. The colon is required, and the sign must be flipped because it will be interpreted with the POSIX-style time zone format. In that format, positive values lie west of GMT instead of the usual conventions specified in ISO 8601.
concat(translate(substring(original_timezone_offset, 1, 3), '-+', '+-'),':',substring(original_timezone_offset, 4, 2))
Then we will use that with AT TIME ZONE to do the conversion:
(original_timezone AT TIME ZONE <the above mess>) AT TIME ZONE 'UTC' AS utc_timestamp
Putting it all together...
WITH t as (
SELECT '2011-06-22 11:00:00.000000'::timestamp as original_timestamp, '-0700' as original_timezone_offset
UNION ALL
SELECT '2014-11-29 17:00:00.000000'::timestamp,'-0800'
UNION ALL
SELECT '2014-12-02 22:00:00.000000'::timestamp,'+0900'
)
SELECT
original_timestamp,
original_timezone_offset,
concat(translate(substring(original_timezone_offset, 1, 3), '-+', '+-'),':',substring(original_timezone_offset, 4, 2)) as modified_timezone_offset,
(original_timestamp AT TIME ZONE concat(translate(substring(original_timezone_offset, 1, 3), '-+', '+-'),':',substring(original_timezone_offset, 4, 2))) AT TIME ZONE 'UTC' AS utc_timestamptz
FROM t
Output:
2011-06-22 11:00:00 -0700 +07:00 2011-06-22 18:00:00
2014-11-29 17:00:00 -0800 +08:00 2014-11-30 01:00:00
2014-12-02 22:00:00 +0900 -09:00 2014-12-02 13:00:00
SQL Fiddle here.
Portal epochconverter.com converts timestamp 1531423084013 to correct date of Thursday, July 12, 2018 3:18:04.013 PM GMT-04:00 DST. But in Python 2.7.12 I got below which is wrong
>>> timestamp=1531423084013
>>> time.ctime(timestamp).rsplit(' ', 1)[0]
'Wed Nov 12 00:06:53'
How to make it correct ?
1531423084013 is in milliseconds not is seconds.
As you can see from epochconverter.com the hour is : 3:18:04.013, so the seconds part is 4.013, this site handle time in seconds and in milliseconds (it seems when the input has 13 digits instead of 10 for time around nowadays).
But time.ctime() from python handle only time in seconds and this is why you get a wrong answer when you enter a time in milliseconds (in my system it throws an out of range).
So you must divide your time in milliseconds by 1000 :
time.ctime(1531423084)
'Thu Jul 12 21:18:04 2018'
(My time zone is UTC+0200)
My input is input_time = "May 5 2016 11:29:32".
Expected output should be in seconds or milli seconds which is type of integer i.e., output_time = 2424241313113.
The above conversion should be done in python. How to do this conversion?
Here's how to convert date time into epoch seconds (dated starting from 00:00:00 UTC on 1 January 1970)
In Python 3.3+
from datetime import datetime
datetime.strptime('May 5 2016 11:29:32','%b %d %Y %H:%M:%S').timestamp()
In Python 2.7.9
datetime.strptime('May 5 2016 11:29:32','%b %d %Y %H:%M:%S').strftime('%s')
Note that strftime('%s') use your local time zone.
Here is my SQL query I am running:
string theQuery = "UPDATE readings SET chng = 1, time = FROM_UNIXTIME(";
theQuery += boost::lexical_cast<string>(ss.time);
theQuery += ") WHERE id = 1;";
ss.time is a uint32_t that records the number of seconds since 1 Jan 1970. When I attempt to put the value "3586767203" into the brackets of FROM_UNIXTIME, which is the time value on my device, it updates my time field to NULL. If I enter a smaller number it updates the time field fine.
Why is it updating to NULL if I am entering a valid time??
You've exceeded the limits of UNIXTIME. 3586767203 is 'Sun, 29 Aug 2083 12:13:23 GMT' and UNIXTIME can't be bigger than a date that resolves into '2038-01-18 22:14:07' ( FROM_UNIXTIME(2147483647) ) because the time is stored as a signed 32 bit integer of seconds after the epoch (Jan 1, 1970) and 2^31 seconds after the epoch is 'Tue, 19 Jan 2038 03:14:08 GMT'.
See http://en.wikipedia.org/wiki/Year_2038_problem for an explanation of this problem.