I'm developing a data fusion pipeline. It contains a wrangler node where I'm trying to create a new field that will contain the system date in timestamp format (yyyy-MM-dd'T'HH-mm-ss).
I've tried using the sentence:
set-column :sysdate (${logicalStartTime(yyyy-MM-dd'T'HH-mm-ss)})
But I receive the error:
Caused by: io.cdap.wrangler.api.DirectiveParseException: Error encountered while parsing 'set-column' : Error encountered while compiling '( 2022 -12-01T16-29-32 ) ' at line '1' and column '14'. Make sure a valid jexl transformation is provided.
Which would be the correct sentence?
I've tried:
set-column :sysdate (${logicalStartTime(yyyy-MM-ddHH-mm-ss)})
Which will result in something like "1877", as it substracts the numbers, and also tried:
set-column :sysdate (${logicalStartTime(yyyyMMddHHmmss)})
but the format isn't correct and can only be written if the field is a String.
You have the correct method, just incorrect syntax. The syntax you are looking for is set-column :sysdate ${logicalStartTime(yyyy-MM-dd'T'HH-mm-ss)}, you have to remove (). Then you can convert the string in datetime pattern in this format parse-as-datetime :sysdate "yyyy-MM-dd'T'HH-mm-ss".
Related
I am trying to populate TIMESTAMP_TZ column in Snowflake using the Informatica IICS from the Taskflow. In the taskflow I am using function fn:current-dateTime() which I am trying then in the mapping to convert to TIMESTAMP_TZ using the to_date() function but I am getting error
MAPPING> TE_7002 [2023-01-17 11:07:50.987] Transformation stopped due to a fatal error in the mapping. The expression [To_Date($$inp_load_dttm)] contains the following errors [<<PM Parse Error>> [$$inp_load_dttm)]: Invalid date value string: >>>2023-01-17T16:06:58.019Z<<<.
... To_Date(>>>>$$inp_load_dttm<<<<)].
MANAGER> PETL_24005 [2023-01-17 11:07:50.988] Starting post-session tasks. : (Tue Jan 17 11:07:50 2023)
fn:current-dateTime() returns date (ref 1). to_date expects a string (ref 2). Here in this case you should not try to conver a date to date - it should simply work. If what you need is just the date, without the time, try using fn:current-date().
IDENTIFIER
31-03-2022_13636075
01-04-2022_13650262
04-04-2022_13663174
05-04-2022_13672025
20220099001
11614491_R
10781198
00000000000
11283627_P
11614491_R
-1
how can i remove (only) the "XX-XX-XXXXX_" Part in certain values of a column in SSIS but WITHOUT affecting values that doesn't have this format? For example "21-05-2022_12345678" = "12345678" but the other values i don't want them affected. This are just examples of many rows from this column so i want only the ones that have this format to be affected.
SELECT REVERSE(substring(REVERSE('09-03-2022_13481330'),0,CHARINDEX('_',REVERSE('09-03-2022_13481330'),0)))
result
13481330
but this also affects others values.Also this is in ssms not ssis because i am not sure how to transform this expression in ssis code.
Update : Corrected code in SSIS goes as following:
(FINDSTRING(IDENTIFIER,"__-__-____[_]",1) == 1) ? SUBSTRING(IIDENTIFIER,12,LEN(IDENTIFIER) - 11) : IDENTIFIER
Do you have access to the SQL source? You can do this on the sql by using a LIKE and crafting a match pattern using the single char wildcard _ please see below example
DECLARE #Value VARCHAR(50) = '09-03-2022_13481330'
SELECT CASE WHEN #Value LIKE '__-__-____[_]%' THEN
SUBSTRING(#Value,12,LEN(#Value)-11) ELSE #Value END
Please see the Microsoft Documentation on LIKE and using single char wildcards
If you don't have access to the source SQL it gets a bit more tricky as you might need to use regex in a script task or maybe there is a expression you can apply
I'm brand new to using PBI but as far as I can tell, I should be able to substitute a parameter as part of a Direct Query in place of a hard-coded variable...ie
let
Source = Sql.Database("NAMEOFDB", "CMUtility", [Query="sp_get_residentsinfo "& home_name]),.....
instead of
let
Source = Sql.Database("NAMEOFDB", "CMUtility", [Query="sp_get_residentsinfo 'NAME OF HOME'"]),...
However, the parameter-included version just says
DataSource.Error: Microsoft SQL: Incorrect syntax near 'House'.
Details:
DataSourceKind=SQL
DataSourcePath=NAMEOFDB;CMUtility
Message=Incorrect syntax near 'House'.
Number=102
Class=15
"House" is the currently - assigned last word of the home_name variable. What have I done wrong?
PS - I have surmised that I shouldn't need the extra & at the end of the parameter, as I'm not adding anything else to the query, but even with both &s it still doesn't work.
The type of your parameters is text. In SQL, text literals must be quoted, i.e. sp_get_residentsinfo 'NAME OF HOME', but the statement build by you is sp_get_residentsinfo NAME OF HOME.
You should use Text.Replace to escape single quotes in the parameter's value and append a quote before and after it.
So I've looked through documentation and previous answers on here, but can't seem to figure this out.
I have a STRING that represents a date. A normal output looks as such:
2018-09-19 17:47:12
If I do this, I get it to return in this format 2018-09-19 17:47:12.000:
SELECT
date_parse(click_time,'%Y-%m-%d %H:%i:%s') click_time
FROM
table.abc
But that's not the output I need. I was just trying to show that I'm close, but clearly missing something. When I change click_time to date_parse(click_time,'%Y-%m-%d'), it sends back INVALID_FUNCTION_ARGUMENT: Invalid format: "2018-09-19 17:47:12" is malformed at " 17:47:12"
So there's clearly something I'm not doing correctly to get it to simply return 2018-09-19.
date_parse converts a string to a timestamp. As per the documentation, date_parse does this:
date_parse(string, format) → timestamp
It parses a string to a timestamp using the supplied format.
So for your use case, you need to do the following:
cast(date_parse(click_time,'%Y-%m-%d %H:%i:%s')) as date )
For your further reference, you can go to the below link for prestodb online documentation https://prestodb.github.io/docs/current/functions/datetime.html
We have a file that looks like this:
Col1,Col2,Col3,Col4,Col5
"Hello,",I,",am",some,data!
It therefore has the following 'properties':
Comma-separated
Double-quote column delimiter
Commas in some of the columns
Now, I am not sure if it's actually possible to ingest this with PolyBase, but wondered if there was a way?
The error we are seeing at present is "Could not find a delimiter after quote".. which i guess is because after the double quote it is hitting what is an expected delimiter..
Here is our current file format, for completeness:
CREATE EXTERNAL FILE FORMAT Comma
WITH (FORMAT_TYPE = DELIMITEDTEXT,
FORMAT_OPTIONS(
FIELD_TERMINATOR = ',',
STRING_DELIMITER = '"',
)
)
Specify it in hex instead.
STRING_DELIMITER = '0x22'
(Based on the problem that someone described at the end of https://msdn.microsoft.com/en-au/library/dn935026.aspx )
Sorted this out in the end by adding an intermediary step to convert the file from csv to ORC format..
It's a bit clunky (as it leaves a mess of a copy behind), but the PolyBase then does work with the fileformat:
CREATE EXTERNAL FILE FORMAT Orc
WITH (FORMAT_TYPE = ORC)
works for now, until it is addressed by the product team: https://feedback.azure.com/forums/307516-sql-data-warehouse/suggestions/10600132-polybase-allow-field-row-terminators-within-strin