I have a CSV file with an attribute having epoch values like '1517334599.906'.
I want to convert/update the Epoch values into ISO timestamp 'yyyy-MM-dd HH:mm:ss.SSS' via NiFi.
That conversion is for Kibana to recognize the field as Timestamp. Is there a way to do this? If there is can anyone help me with the configuration?
Using NiFi's record capabilities you can use UpdateRecord with a CsvReader and CsvWriter.
See the "format" function in expression language for converting an epoch to a date string:
https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html#format
In UpdateRecord you would do something like:
/eventDate = ${field.value:format("yyyy-MM-dd HH:mm:ss.SSS")}
This says take the value of /eventDate (change this to your field name) and set the value of that field to the result of the format function on the right.
The only thing I am not sure about is whether an epoch can have a decimal portion as shown in your example. I would expect it to be converted to a long which would be a whole number.
Related
i have a bigint data type value 10062019192751 it is said to me that it is a datetime formated as ddmmyyyyhhmmss (10-06-2019 19:27:51)
how can i convert or parse it to datetime in AWS Athena
using syntax from_unixtime, is giving me different value
Amazon Athena is based on Presto, so you can use Date and Time Functions and Operators — Presto.
The date_parse() command can convert a string into a date by defining the format of the string (consult the above link to see the syntax).
Here is a solution, which first converts the number into a string (varchar) and then converts it into a date:
select date_parse(cast(10062019192751 as varchar),'%d%c%Y%k%i%s')
The output is:
2019-06-10 19:27:51.000
I have the date string in the form: 2019-02-18 09:17:31.260000+00:00 and I am trying to convert it into date in Athena.
I have tried converting into timestamp as suggested in the SO answers but failed.
There is a discussion in https://github.com/prestodb/presto/issues/10567 but no answer to this particular date format.
I tried several format like 'YYYY-MM-dd HH:mm:ss.SSSSSSZ' but doesn't work and get error like INVALID_FUNCTION_ARGUMENT: Invalid format:..is malformed at "+00:00".
Been stuck for a while, any help is appreciated!
Athena is based on a very old version of Presto, and there is no straightforwad way of doing that with some string manipulation trick. For instance, you can use regexp_replace to extract the part of the string that's compatible with the built-in timestamp with timezone type and do:
SELECT cast(regexp_replace('2019-02-18 09:17:31.260000+00:00','(\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}\.\d{3})\d{3}(.*)', '$1$2') AS timestamp with time zone)
Recent versions of Trino (formerly known as PrestoSQL) introduced support for variable-precision temporal types with up to nanosecond precision (12 decimals).
With that feature, you can just do:
trino> select cast('2019-02-18 09:17:31.260000+00:00' as timestamp(6) with time zone);
_col0
--------------------------------
2019-02-18 09:17:31.260000 UTC
(1 row)
A shorter version to Martin Traverso's answer is to sub string the extra characters:
select cast(substr('2019-02-18 09:17:31.260000+00:00',1,23) as timestamp);
In my application, server side date validations were done through IsDate which is very inconsistent in behavior. I used isValid("USdate",DateVar), that works fine with incoming dates, but when DateVar is a date time stamp it fails. Values coming in DateVar could be anything, a date, a time, a date & time or even some invalid data. If I use Date mask with isValid, that behaves like isDate and of no use. How I can accomplish this.
All "dates" that will be arriving via a request - be they via a URL parameter, a form submission, a cookie, etc - will be strings, not dates.
What you need to do is to work out what string formats you will allow, and validate them accordingly.
EG: you might decide that yyyy-mm-dd is OK, but you won't accept m/d/yy. You might pass them as three separate components for y, m and d. But you really oughtn't try to accept any old format, as you will need to have a validator for each format, and there's a law of diminishing returns there: people won't expect to use any format they like; they'll be expecting you to guide them. You also need to be mindful that if you ask me to type in today's date, I'd give you 4/5/2015. But to you that might represent April 5.
Given various month-length and leap-year rules, really the easiest and most reliable way to see if and input string represents a date in an acceptable format do this:
Validate the format mask, eg: if you're accepting yyyy-mm-dd, then the input needs to be \d{4}-\d{2}-\d{2}. Then at least you know the string has been formed properly.
Then extract the components from the string, and attempt to create a date object with them. If it doesn't error: it's OK.
Last: check any date boundaries within / outwith which the date needs to fall.
I am trying to use df.iloc[] to locate a particular cell in dataframe. But we are unable to do it bcoz the cell contains date in timestamp format. I guess we need to convert that into int or string or float type. How do we do this?
that depends on the condition that that row is supposed to answer.
if you are looking for a specific range of dates try this :
dates=pd.date_range(A,Z)
day1DF= LWeekDF.ix[(LWeekDF['ReportTime']>=dates[A]) & (LWeekDF['ReportTime']<=dates[1])]
you now have a data frame with rows that answer to that condition.
this will work with other conditions as well.
if the dates are giving you problems try this
A=pd.datetime.date(a)
I'm using Qt to parse an XML file which contains timestamps in UTC. Within the program, of course, I'd like them to change to local time. In the XML file, the timestamps look like this: "2009-07-30T00:32:00Z".
Unfortunately, when using the QDateTime::fromString() method, these timestamps are interpreted as being in the local timezone. The hacky way to solve this is to add or subtract the correct timezone offset from this time to convert it to "true" local time. However, is there any way to make Qt realize that I am importing a UTC timestamp and then automatically convert it to local time?
Do it like this:
QDateTime timestamp = QDateTime::fromString(thestring);
timestamp.setTimeSpec(Qt::UTC); // mark the timestamp as UTC (but don't convert it)
timestamp = timestamp.toLocalTime() // convert to local time
try using the setTime_t function.
Note that full time zone support is not available yet in Qt, but probably will in future versions.
http://bugreports.qt-project.org/browse/QTBUG-10219