Pentaho Transformation Executor not passing parameters successfully - kettle

Please refer to 2 Examples attached on this link
Example1 : TEMP1.ktr & TEMP2.ktr
Example2 : PARENT.ktr & CHILD.ktr
I am trying to pass few parameters to ktr patched in Transformation Executor and while having less number of field it is being able to pass those parameters successfully (Refer to Example1) but when i am increasing number of fields in the parent KTR it fails to send parameters to ktr patched in Transformation Executor (Refer to Example2)
Any help would be highly appreciated.

All the results are successfully transferred to CHILD.ktr. You can see the result with adding "Get rows from result"->"Text file Output" in CHILD.Ktr.
But if you want to see result in 'Write to log' step then you need to add all the result field as a parameter one by one in 'Transformation executor' step like you added for 'city' and 'state' in your example1. Then show in log file like ${tmp1} ...... ... as a parameter.
You can download updated parent and child ktr
Link

Related

Override field in the input before passing to the next state in AWS Step Function

Say I have 3 states, A -> B -> C. Let's assume inputs to A include a field called names which is of type List and each element contains two fields firstName and lastName. State B will process the inputs to A and and return a response called newLastName. If I want to override every element in names such that names[i].lastName = newLastName before passing this input to state C, is there an built-in syntax to achieve that? Thanks.
You control the events passed to the next task in a Step Function with three defintion attributes: ResultPath and OutputPath on leaving one task and InputPath on entering the next one.
You have to first understand how the event to the next task is crafted by a State Machine, and each of the 3 above parameters changes it.
You have to at least have Result Path. This is the key in the event that the output of your lambda will be placed under. so ResultPath="$.my_path" would result in a json object that has a top level key of my_path with the value equal to whatever is outputted from the lambda.
If this is the only attribute, it is tacked onto whatever the input was. So if your Input event was a json object with keys original_key1 and some_other_key your output with just the above result path would be:
{
"original_key_1": some value,
"some_other_key": some other value,
"my_path": the output of your lambda
}
Now if you add OutputPath, this cuts off everything OTHER than the path (AFTER adding the result path!) in the next output.
If you added OutputPath="$.my_path" you would end up with a json of:
{ output of your lambda }
(your output better be a json comparable object, like a python dict!)
InputPath does the same thing ... but for the Input. It cuts off everything other than the path described, and that is the only thing sent into the lambda. But it does not stop the input from being appeneded - so InputPath + ResultPath results in less being sent into the lambda, but everything all together on the exit
There isn't really a loop logic like the one you describe however - Task and State Machine definitions are static directions, not dynamic logic.
You can simply handle it inside the lambda. This is kinda the preferred method. HOWEVER if you do this, then you should use a combination of OutputPath and ResultPath to 'cut off' the input, having replaced the various fields of the incoming event with whatever you want before returning it at the end.

AWS Systems Manager `GetParametersByPath` API returns outdated results

I am trying to utilize SSM's GetParametersByPath API and I am getting outdated results which looks like a ~3 seconds caching by Path parameter.
I am testing the following scenario:
Get parameters by path recursively
Put a parameter under the tested path
Get parameters by path again using same arguments as in (1)
I am getting the same response in step (3) as in step (1) regardless of the changes in step (2).
I am thinking that this is related to caching because the following 2 scenarios work as expected.
Correct behavior scenario 1:
Get parameters by path recursively
Put a parameter under the tested path
Sleep for 3 seconds
Get parameters by path again using same parameters as in (1)
Correct behavior scenario 2:
Put a parameter under the tested path
Get parameters by path recursively
This behavior is consistent across different SDKs which I tried: .Net, Python (boto3) and CLI, so this is not an SKD issue.
Here is a code snippet in Python with boto3 that replicates incorrect behavior:
import boto3
client = boto3.client('ssm')
first = client.get_parameters_by_path(
Path='/param-store-test/1',
Recursive=True,
WithDecryption=True,
MaxResults=10)
print(first['Parameters'][0]['Version'] if first['Parameters'] else None)
put_response = client.put_parameter(
Name='/param-store-test/1/2',
Value='test',
Type='SecureString',
KeyId='alias/aws/ssm',
Overwrite=True,
Tier='Standard')
print("v{}".format(put_response['Version']))
second = client.get_parameters_by_path(
Path='/param-store-test/1',
Recursive=True,
WithDecryption=True,
MaxResults=10)
print(second['Parameters'][0]['Version'] if second['Parameters'] else None)
This code gives me the following output when run for the first time:
None
v1
None
And when run for the second time:
1
v2
1
You can see the pattern - the first request is being cached.
According to API docs: Request results are returned on a best-effort basis.
So is this behavior considered to be correct?
Does this mean that I don't have any way to get all parameters by path in a reliable way?
I have noticed that get_parameters_by_path() does not work as I expected.
You might think that (simplified code):
put_parameter('/abc/def', "123")
get_parameters_by_path('/abc/def')
would return "123", whereas it seems to return [] nothing.
However,
get_parameter('/abc/def')
correctly returns "123".
Also,
get_parameters_by_path('/abc')
will return '/abc/def' => "123"
So it seems to work, but not quite as one might expect.

Activating Proxy service consumer from ABAP

I've just created a service consumer and generated its code from ABAP in order to send to the target WSDL system XML file which is created successfully.
The code should fill in the XML_FILE_INFO fields but have no idea if I did it OK:
The code to activate it:
" Call method, and receive a response
try.
lo_proxy->SET_MESSAGE(
exporting
INPUT = gs_input
importing
OUTPUT = ls_output
).
catch CX_AI_SYSTEM_FAULT INTO lo_AI_SYSTEM_FAULT. "lo_ai_system_fault_info.
catch CX_AI_APPLICATION_FAULT INTO LO_AI_APPLICATION_FAULT.
ENDTRY.
How should I fill the XML file's content in the GS_INPUT structure? IS putting the file's path in the FILE_NAME field enough?
If someone had already done something like the mentioned requirement I'd appreciate any help.
Thanks in advance.

How do I retrieve AWS Batch job parameters?

How do I retrieve parameters from an AWS Batch job request? Suppose I have a job submitter app that sends a job request with the following code (in C#):
SubmitJobRequest submitJobRequest = new SubmitJobRequest()
{
JobName = "MyJobName",
JobQueue = "MyJobQueue",
JobDefinition = "MyJobDefinition:1",
Parameters = new Dictionary<string, string>() { {"Foo", "Bar" } },
};
SubmitJobResponse submitJobResponse = AWSBatchClient.SubmitJob(submitJobRequest);
What I want to be able to do now is retrieve what's in the Parameters field in submitJobRequest in my docker app that gets launched. How do I do that? It's not passed in as program args, as I've tested that (the only args I see are those were statically defined for 'Command' my job definition). I know that I can set environment variables via container overrides and then retrieve them via Environment.GetEnvironmentVariable (in C#). But I don't know how to get the parameters. Thanks.
Here is an example using yaml cloudformation.(or you can use json for same properties).You can declare the parameters using a Ref in the command section. I am using user_name but you can add more. We might have a limit of 30KB payload
ContainerProperties:
Command:
- "python"
- "Your command here"
- "--user_name"
- "Ref::user_name"
Now you can submit your job to the queue like this. I am using python and boto3 client:
# Submit the job
job1 = client.submit_job(
jobName=jobName,
jobQueue=jobQueue,
jobDefinition=jobDefinition,
parameters={
'user_name':user_name
}
)
To retrieve the parameters use this(I am using argparse):
parser = argparse.ArgumentParser(description='AWS Driver Batch Job Runner')
parser.add_argument('--user_name', dest='user_name', required=True)
args = parser.parse_args()
print(args.user_name)
Found the answer I was looking for. I just had to add a ref to the parameter for Command in the job definition. In the question example, I would've needed to specify Ref::Foo for Command in the job definition and then "Bar" would've gotten passed as program args to my container app.
To expand on my example, in my specific case, my program uses the CommandLineParser package for passing parameters. Suppose one of the CommandLine options is called Foo. If I were running the program from a command line, I'd set a value for Foo with something like "--Foo Bar". To effectively do the same for my batch job, in my job definition, for Command, I would specify "--Foo Ref::Foo" (without quotes). Then for the Parameters field in my SubmitJobRequest object, I would set Foo exactly as per my original example and then my batch program would see "Bar" for the Foo CommandLine option (just like as if it was run with "--Foo Bar"). Hope that helps.

OTRS Webservice as Requestor Test

I'm new to OTRS (3.2) and also new to PERL but I have been given the task of setting up OTRS so that it will make a call to our remote webservice so a record can be created on our end when a ticket is set as "Closed".
I set up various dynamic fields so the customer service rep can fill in additional data that will be passed into the webservice call along with ticket details.
I couldn't get the webservice call to trigger when the ticket was "Closed" but I did get it to trigger when the "priority" was changed so I'm just using that now to test the webservice.
I'm just using the Test.pm and TestSimple.pm files that were included with OTRS.
When I look at the Debugger for the Webserice, I can see that the calls were being made:
$VAR1 = {
'TicketID' => '6'
};
My webservice currently just has one method "create" which just returns true for testing.
however I get the following from the Test.pm
"Got no TicketNumber (2014-09-02 09:20:42, error)"
and the following from the TestSimple.pm
"Error in SOAP call: 404 Not Found at /TARGET/SHARE/var/otrs/Kernel/GenericInterface/Transport/HTTP/SOAP.pm line 578 (2014-09-02 09:20:43, error)
I've spent countless hours on Google but couldn't find anything on this. All I could find is code for the Test.pm and TestSimple.pm but nothing really helpful to help me create a custom invoker for my needs and configure the webservice in OTRS to get it to work.
Does anyone have any sample invokers that I can look at to see how to set it up?
Basically I need to pass the ticket information along with my custom dynamic fields to my webservice. From there I can create the record on my end and do whatever processing.
I'm not sure how to setup the Invoker to pass the necessary ticket fields and dynamic fields and how to make it call a specific method in my remote webservice.
I guess getting the Test.pm and TestSimple.pm to work is the first step then I can modify those for my needs. I have not used PERL at all so any help is greatly appreciated.
I'm also struggling with similar set of requirements too. I've also never programmed in PERL, but I can tell you at least that the "Got no TicketNumber" in the Test.pm is right from the PrepareRequest method, there you can see this block of code:
# we need a TicketNumber
if ( !IsStringWithData( $Param{Data}->{TicketNumber} ) ) {
return $Self->{DebuggerObject}->Error( Summary => 'Got no TicketNumber' );
}
You should change all references to TicketNumber to TicketID, or remove the validation whatsoever (also there is mapping to ReturnedData variable).
Invoking specific methods on your WS interface is quite simple (but poorly documented). The Invoker name that you specify in the "OTRS as requester" section of web service configuration corresponds to the WS method that will be called. So if you have WS interface with a method called "create" just name the Invoker "create" too.
As far as the gathering of dynamic field goes, can't help you on that one yet, sorry.
Cheers