I have a question about timestamps.
Here is a test_table:
CREATE TABLE test_table(
ts TIMESTAMP,
timestamp TIMESTAMP
) timestamp(timestamp) PARTITION BY DAY;
If I save:
test_table ,ts=1626652800000000000i 1626745200000000000
Both timestamps are in nanoseconds.
I the db I get two different dates saved:
ts = 53522-01-20T00:00:00.000000Z (wrong)
timestamp = 2021-07-22T01:40:00.000000Z (correct)
Why are they different?
If I save them as microseconds:
ts = 2021-07-19T00:00:00.000000Z (correct)
timestamp = 1970-01-19T19:52:25.200000Z (wrong)
Related
Noticed the other day that there are some significant differences in query performance when running two nearly identical queries.
QUERY 1:
SELECT * FROM "table"
WHERE (badge = 'xyz' or badge = 'abc')
and ((year = '2021' and month = '11' and day = '1')
or (year = '2021' and month = '10' and day = '31'))
ORDER BY timestamp
Runtime: 40.751 sec
Data scanned: 94.06 KB
QUERY 2:
SELECT * FROM "table"
WHERE (badge = 'xyz' or badge = 'abc')
and ((year = '2021' and month = '10' and day = '30')
or (year = '2021' and month = '10' and day = '31'))
ORDER BY timestamp
Runtime: 1.78 sec
Data scanned: 216.86 KB
The only major difference between the two is that one query looks at 11/1 & 10/31 and the other looks at 10/31 & 10/30. So there is an additional month partition being looked at in QUERY 1.
When running both queries
with EXPLAIN I
noticed that
QUERY 2 uses a TableScan while QUERY1 uses a ScanFilter.
Anyone know why this might be the case between these two queries?
Additional Details:
Time in queue for both queries was sub 1 second.
In s3, the data is structured as follows:
badge=%s/year=%s/month=%s/day=%s/hour=%s
badge,year,month,day & hour are all partitions defined via Partition Projection.
I have a simple table contains the node, message, starttime, endtime details where starttime and endtime are in unix timestamp. The query I am running is:
select node, message, (select from_unixtime(starttime)), (select from_unixtime(endtime)) from table1 WHERE try(select from_unixtime(starttime)) > to_iso8601(current_timestamp - interval '24' hour) limit 100
The query is not working and throwing the syntax error.
I am trying to fetch the following information from the table:
query the table using start time and end time for past 'n' hours or 'n' days and get the output of starttime and endtime in human readable format
query the table using a specific date and time in human readable format
You don't need "extra" selects and you don't need to_iso8601 in the where clasue:
WITH dataset AS (
SELECT * FROM (VALUES
(1627409073, 1627409074),
(1627225824, 1627225826)
) AS t (starttime, endtime))
SELECT from_unixtime(starttime), from_unixtime(endtime)
FROM
dataset
WHERE from_unixtime(starttime) > (current_timestamp - interval '24' hour) limit 100
Output:
_col0
_col1
2021-07-27 18:04:33.000
2021-07-27 18:04:34.000
to search last week you can use
WHERE your_date >= to_unixtime(CAST(now() - interval '7' day AS timestamp))
I have a table named adata in redshift which contains a column timestamp with integer type data variable. The timestamp column holds the timestamp as epoch values. I have added a new column called rdate with date variable type. I am trying to insert date value into rdate column by converting the timestamp column.
What is working;
select timestamp 'epoch' + timestamp * interval '1 second' AS rdate from adata;
This prints the rdate column with timestamp as date and time nicely.
what is not working
insert into adata (rdate) select timestamp 'epoch' + timestamp * interval '1 second' AS rdate from adata;
I tried with an update statement:
update adata set rdate = (select timestamp 'epoch' + timestamp * interval '1 second' AS rdate from adata);
ts rdate
1611306839 need date here
1611469226 need date here
1611399334 need date here
1611373685 need date here
another option which I tried:
update adata set rdate = (select (timestamp 'epoch' + timestamp * interval '1 second') AS rdate from adata);
It appears that you want to update existing rows in a table. In that case, you would use an UPDATE command.
Since the rows will be updated based upon the value of other columns in the same table, you could simply use:
UPDATE adata SET rdate = timestamp 'epoch' + timestamp * interval '1 second'
where timestamp is the column that contains the existing value.
(Using INSERT would add additional rows, which probably isn't what you are seeking.)
Currently , I have date column in time format ,I want to change it to date time stamp format I.e ( I want the date column to look like 12nov 2020 12:03:45:00 )
Could someone help me on this ?
According to #KurtBremser:
SAS dates are counts of days, SAS datetimes are counts of seconds.
datetime = dhms(date,0,0,0);
will convert a date to a datetime. Or multiply by 86400.
A column showing a time representation hh:mm:ss can be one of three things:
A character column type containing digit characters 0-9 and :
A number column type containing a SAS time value being displayed as hh:mm:ss with the time format TIME8.
A number column type containing a SAS datetime value being displayed as hh:mm:ss with the datetime format TOD.
This sample program demonstrates how different kinds of values can all look the same when viewed.
data have;
v1 = '12:34:56';
v2 = hms(12,34,56);
v3 = dhms(today(),12,34,56);
put v1= / v2= time8. / v3=tod. / v3=datetime18.;
run;
------ LOG ------
v1=12:34:56
v2=12:34:56
v3=12:34:56
v3=25NOV20:12:34:56
Only #3 has enough information in the raw value to be formatted as ddmmmyyy:hh:mm:ss
format myDate datetime18.;
#2 requires computing a new value assuming something about the date part
* supposing myDate contains only time values (00:00:00 to 23:59:59) for today;
myNewDate = dhms(today(),0,0,0) + myDate;
format myNewDate datetime18.;
#1 requires interpretation through INPUT and a date assumption
* supposing myDate contains "hh:mm:ss" for today
myNewDate = dhms(today(),0,0,0) + input(myDate,time8.);
format myNewDate datetime18.;
I found similar posts but had issues with them.
The problem I have is:
I have a date in the format year, month, day example - 2019:01:21
and I want to use a loop likely something like
for i in range(365):
# add to the date by a single day and return something in the form 2019:01:22
You could use datetime.strptime() to parse the given date in string format to a python date format. Then you could use datetime.timedelta() to specify an interval of n days and shift the parsed date by this delta. Here is a short demo:
from datetime import datetime, timedelta
date = datetime.strptime("2019:01:21", "%Y:%m:%d")
for i in range(365):
delta = timedelta(days=1 + i)
print(date + delta)
Output:
2019-01-22 00:00:00
2019-01-23 00:00:00
2019-01-24 00:00:00
...