I have a table in Amazon Redshift where the time column is a string type and stores values as HH:MM form. I need to convert this to time format to do some further processing. I tried ::time which raised "Unknown data type in extract result set" error. To_date(my_field, 'HH:MM') didn't work either.
Char HH:MM to time HH:MM
Thanks for your help :)
According to Datetime format strings - Amazon Redshift:
To convert a 24-hour 2-digit hour, use HH24
To convert minutes, use MI
Therefore, try: TO_TIMESTAMP(field, 'HH24:MI')
See also: TO_TIMESTAMP function - Amazon Redshift
Related
I would like to convert some dates stored as a string to a date. The data looks as follows: 26APR2017:06:01:44. I would prefer it to be in any regular date format.
Can someone please help me to convert this to a date format? I'm sure the dateparse function is useful here and have used it before but am unsure how to adjust it for this particular data.
You can use date_parse() to parse and date() to cast timestamp retunred from parse function to date.
select date_parse('26APR2017:06:01:44','%d%b%Y:%H:%i:%s') will give you
2017-04-26 06:01:44.000
select date(date_parse('26APR2017:06:01:44','%d%b%Y:%H:%i:%s')) will give you
2017-04-26
I'm looking to use Athena Partition Projection to analyze log files from AWS application load balancers and firehose emitted logs. The data in S3 is prefixed with year/month/day and potentially hour as well. I've been able to accomplish using the Firehose Example; however this example uses a string formatted partition column.
I'm looking to see if it's possible to use a date formatted partition column instead (with partition project and the firehose emitted s3 prefix format), as our query writers are already used to most of our queries involving date columns and it avoids the need to string format for relative date queries. Is this possible or would the s3 prefixes need to be changed to accomplish?
Table Properties for String column: WORKS
PARTITIONED BY (
`logdate` string)
TBLPROPERTIES (
'projection.enabled'='true',
'projection.logdate.format'='yyyy/MM/dd',
'projection.logdate.interval'='1',
'projection.logdate.interval.unit'='DAYS',
'projection.logdate.range'='NOW-2YEARS,NOW',
'projection.logdate.type'='date',
'storage.location.template'='s3://bucket/prefix/${logdate}')
Table Properties for Date Partition column Does Not Work
PARTITIONED BY (
`logdate` date)
TBLPROPERTIES (
'projection.enabled'='true',
'projection.logdate.format'='yyyy/MM/dd',
'projection.logdate.interval'='1',
'projection.logdate.interval.unit'='DAYS',
'projection.logdate.range'='NOW-2YEARS,NOW',
'projection.logdate.type'='date',
'storage.location.template'='s3://bucket/prefix/${logdate}')
HIVE_INVALID_PARTITION_VALUE: Invalid partition value '2018/11/13' for DATE partition key: logdate=2018%2F11%2F13
I think the only thing you need to do is make sure the type of the logdate partition key to be string:
PARTITIONED BY (logdate string)
This is not the same as projection.logdate.type, which should continue to be date.
Partition keys with type date are just dates within the calculations partition projection performs. For all other purposes they are strings. PP will parse values using the date format you specify, do its calculations, then output strings using the same date format. This happens during query planning, before the Presto engine is involved.
Presto's schema-on-read approach means that you can say that a column has type date if its format matches the expected format of dates; yyyy-MM-dd in Java format. The format that you get from Firehose's S3 keys, yyyy/MM/dd, can't be cast to date automatically, it needs to be explicitly parsed:
parse_datetime(logdate, 'yyyy/MM/dd')
I think it would have been great if PP would have been aware of the types of partition keys so that you could have done what you have tried to do, but I assume that since PP happens during query planning and most likely not anywhere near where the types of things are known it's probably too difficult to achieve.
I'm casting TIMESTAMP column type into DATE on AWS Athena with parquet files. For some reasons 2049-12-10 00:00:00 throughs an error. Is there any max date limits for this type?
Presto (which Athena is based on) uses 32-bit signed integer to represent a date. The value is interpreted as "epoch day" (number of days since 1970-01-01), so max date value is 5881580-07-11.
I am getting data from an excel file which include a column which shows duration as mm:ss:ms
But when I import the data to Power BI Desktop with Power Query, it converts this column to date/time format.
I don't know how can I solve this issue.
Any idea?
I already tried the code below:
format(((TableName[Duration] / 60)/60)/24, "HH:mm:ss")
and already tried to change the source column type from the excel to general.
PowerBI does not currently support milliseconds in date time datatype. You can convert the excel doc to general, load it into PowerBI, and multiply the value by 86400 to get the total number of seconds. Otherwise you'll need to bring it in as a string and do string manipulations to pull out the information you want.
Hope it helps.
I am using AWS DynamoDB for storing time series data.
Accuracy is very important for this application. But, DynamoDB does not support native Date type to store.
I have two options:
Use UNIX Epoch Time
Use Date as string and store
Both options seem to have their problems:
The Epoch Time has the problem of leap seconds and losing accuracy.
The String Time type may have problems for sorting or querying between range of dates or times.
Which one should I use when accuracy is a key factor?
Use ISO format.
Stores milliseconds
Stores an optional timezone
Supports range queries for dates and date/time
Sample date formatted according to ISO 8601:
2016-12-02T00:52:34.256Z