I have successfully wrote a java program sending a ktr file to a remote Kettle platform to execute. The java program will load tasks in database and sending a related ktr to Kettle to execute. My problem is that I do not know how to get the Kettle Trans execution results for this task gracefully, cause I don't know how to filter out the trans logs by a certain task.
My current solution is to read the Ketlle Trans Log, one of the database log tables generate by Kettle itself.
The problem in this solution is since that all the fields and related data are provided by Kettle, I cannot tell which log is related to which task. So I have to rename transname in the transmeta with the task id, and then the task id will then be passed into the Trans Log table in the transname field. Through this way I can get the related log record by the task id.
private void startClusterTrans(String repoConnectStr, String transName, String taskId) throws PlatformException {
Repository repository = null;
TransMeta transMeta = null;
try {
repository = getKettleRepository(repoConnectStr);
// 1: find the trans to execute by transName
transMeta = findTransMeta(transName, repository);
// 2: set the trans name with task id
// later the taksId will be passed to transname field in the trans log table
transMeta.setName(taskId);
TransExecutionConfiguration transExecutionConfiguration = new TransExecutionConfiguration();
transExecutionConfiguration.setLogLevel(LogLevel.BASIC);
transExecutionConfiguration.setExecutingClustered(true);
...
transExecutionConfiguration.setVariables(transMeta);
Trans.executeClustered(transMeta, transExecutionConfiguration);
} catch (Throwable e) {
throw new PlatformException(e);
}
}
I think this method is not gracefully. Firstly, I have to scan the database every some seconds and cause database pressure. Secondly The transname will be changed if the trans is executed in cluster mode.
Would anyone tell me if there is some other common way to filter the trans execution results? I hope the way to fetch the results will be the same when trans is executed in local, remote and cluster mode.
To elaborate my problem, let's take cluster mode for example. I start a java task #1001 , which will execute a foo.ktr in a cluster mode. I have 1 master and 2 slaves in this situation. After the execution, there will be 3 records generated in trans log table. Sadly I don't know how to link the 3 trans logs to my task #1001.
Now my workaround is that before send foo.ktr to cluster, I change the transname of the transmeta to be task #1001, and after execution, the transname field for the 3 trans records in the trans log table will look like 1001 (my-cluster-sth) ; 1001 (my-cluster: slave1) ; 1001 (my-cluster: slave2) . So I have to use "LIKE" key word to link these 3 records to my task. I don't think this is a good way. The 2 concerns are alread mentioned above.
Please upvote http://jira.pentaho.com/browse/PDI-16549 and http://jira.pentaho.com/browse/PDI-10912.
If I do understand what you are aiming at, in a foreseeable future you 'll need to know the values of the parameters with which the transformation was run. So, upvote also http://jira.pentaho.com/browse/PDI-16550.
Related
As far as I can tell there's no identifier being passed with the GT worker metadata (see below from documentation https://docs.aws.amazon.com/sagemaker/latest/dg/sms-data-output.html)? How would I link this information back to the actual labeling task?
sub I believe is a cognito reference to the worker, so not a unique identifier for the submisson. As of right now, I jsut know that one of the tasks took a certian amount of time for a particular worker, but I can't tell which one. I also guess i have to jump through a few hoops via cognito to get the GT worker id from the sub?
I am looking for a way to summarize origina data shown (from input manifest file), the label given, the time it took to complete. As of right now, I have to make one table that has the data with their human submitted label, and a separate table with time it took to complete by task, but no way to link the two...am I missing something?
here's the worker metadata json:
"submissionTime": "2020-12-28T18:59:58.321Z",
"acceptanceTime": "2020-12-28T18:59:15.191Z",
"timeSpentInSeconds": 40.543,
"workerId": "a12b3cdefg4h5i67",
"workerMetadata": {
"identityData": {
"identityProviderType": "Cognito",
"issuer": "https://cognito-idp.aws-region.amazonaws.com/aws-region_123456789",
"sub": "aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee"
}
}
We are facing an MismatchingMessageCorrelationException for the receive task in some cases (less than 5%)
The call back to notify receive task is done by :
protected void respondToCallWorker(
#NonNull final String correlationId,
final CallWorkerResultKeys result,
#Nullable final Map<String, Object> variables
) {
try {
runtimeService.createMessageCorrelation("callWorkerConsumer")
.processInstanceId(correlationId)
.setVariables(variables)
.setVariable("callStatus", result.toString())
.correlateWithResult();
} catch(Exception e) {
e.printStackTrace();
}
}
When i check the logs : i found that the query executed is this one :
select distinct RES.* from ACT_RU_EXECUTION RES
inner join ACT_RE_PROCDEF P on RES.PROC_DEF_ID_ = P.ID_
WHERE RES.PROC_INST_ID_ = 'b2362197-3bea-11eb-a150-9e4bf0efd6d0' and RES.SUSPENSION_STATE_ = '1'
and exists (select ID_ from ACT_RU_EVENT_SUBSCR EVT
where EVT.EXECUTION_ID_ = RES.ID_ and EVT.EVENT_TYPE_ = 'message'
and EVT.EVENT_NAME_ = 'callWorkerConsumer' )
Some times, When i look for the instance of the process in the database i found it waiting in the receive task
SELECT DISTINCT * FROM ACT_RU_EXECUTION RES
WHERE id_ = 'b2362197-3bea-11eb-a150-9e4bf0efd6d0'
However, when i check the subscription event, it's not yet created in the database
select ID_ from ACT_RU_EVENT_SUBSCR EVT
where EVT.EXECUTION_ID_ = 'b2362197-3bea-11eb-a150-9e4bf0efd6d0'
and EVT.EVENT_TYPE_ = 'message'
and EVT.EVENT_NAME_ = 'callWorkerConsumer'
I think that the solution is to save the "receive task" before getting the response for respondToCallWorker, but sadly i can't figure it out.
I tried "asynch before" callWorker and "Message consumer" but it did not work,
I also tried camunda.bpm.database.jdbc-batch-processing=false and got the same results,
I tried also parallel branches but i get OptimisticLocak exception and MismatchingMessageCorrelationException
Maybe i am doing it wrong
Thanks for your help
This is an interesting problem. As you already found out, the error happens, when you try to correlate the result from the "worker" before the main process ended its transaction, thus there is no message subscription registered at the time you correlate.
This problem in process orchestration is described and analyzed in this blog post, which is definitely worth reading.
Taken from that post, here is a design that should solve the issue:
You make message send and receive parallel and put an async before the send task.
By doing so, the async continuation job for the send event and the message subscription are written in the same transaction, so when the async message send executes, you already have the subscription waiting.
Although this should work and solve the issue on BPMN model level, it might be worth to consider options that do not require remodeling the process.
First, instead of calling the worker directly from your delegate, you could (assuming you are on spring boot) publish a "CallWorkerCommand" (simple pojo) and use a TransactionalEventLister on a spring bean to execute the actual call. By doing so, you first will finish the BPMN process by subscribing to the message and afterwards, spring will execute your worker call.
Second: you could use a retry mechanism like resilience4j around your correlate message call, so in the rare cases where the result comes to quickly, you fail and retry a second later.
Another solution I could think of, since you seem to be using an "external worker" pattern here, is to use an external-task-service task directly, so the send/receive synchronization gets solved by the Camunda external worker API.
So many options to choose from. I would possibly prefer the external task, followed by the transactionalEventListener, but that is a matter of personal preference.
I have written a simple one record delete operation job in production as requested by user, in an AX instance while the other instance was stuck and open. However the record was not deleted.
try
{
ttsbegin;
select fotupdate tableBuffer where tableBuffer.recid == 5457735:
tableBuffer.delete();
ttscommit;
}
catch (exception::error)
{
info("Delete operation cancelled.");
}
tableBuffer's delete()function was overridden with code after super() to store the deleted record in another table.
I have done the same operation earlier successfully but no where with a scenario like one today(executed in one instance while the other instance was stuck).
Please suggest the possible reason as I find the record still persist both in sql server and AX.
Thank you.
If you're trying to prevent this from happening you can use pessimistic locking, where you obtain an update lock.
select pessimisticLock custTable
where custTable.AccountNum > '1000'
See these links for more info:
http://dev.goshoom.net/en/2011/10/pessimistic-locking/
https://blogs.msdn.microsoft.com/emeadaxsupport/2009/07/08/about-locking-and-blocking-in-dynamics-ax-and-how-to-prevent-it/
https://msdn.microsoft.com/en-us/library/bb190073.aspx
cred_query = credits_tbl.query(ancestor=user_key).fetch(1)
for q in cred_query:
q.total_credits = q.total_credits + credits_bought
q.put()
I have a task running which is constantly updating a users total_credits in the credits table.
While that task runs the user can also buy additional credits at any point (as shown in the code above) to add to the total. However, when they try to do so, it does not update the total_credits in the credits table.
I guess I don't understand the 'strongly consistent' modelling of appengine (using ndb) as well as I thought.
Do you know why this happens?
EIM job is getting error out while running it. Below is my IFB file -
"[Siebel Interface Manager]
USER NAME = 'SADMIN'
PASSWORD = 'SADMIN'
PROCESS = "PROCESS UPDATE"
[PROCESS UPDATE]
TYPE = IMPORT
BATCH = 30032012 - 30032015
TABLE = EIM_FN_ASSET5
INSERT ROWS = S_ASSET_CON, FALSE
UPDATE ROWS = S_ASSET_CON, TRUE
ONLY BASE TABLES = S_ASSET_CON
ONLY BASE COLUMNS = S_ASSET_CON.ATTRIB_37,S_ASSET_CON.ATTRIB_38,S_ASSET_CON.ATTRIB_50,S_ASSET_CON.ASSET_ID,S_ASSET_CON.CONTACT_ID,\
S_ASSET_CON.RELATION_TYPE_CD"
In application, it shows error --
"SBL-EIM-00426: All batches in run failed."
I have placed IFB in admin folder itself and below is the log file -
"2021 2012-04-03 05:35:25 2012-04-03 05:35:25 -0500 00000002 001 003f 0001 09 srvrmgr 16187618 1 /004fs02/siebel/siebsrvr/log/srvrmgr.log 8.1.1.4 [21225] ENU
SisnapiLayerLog Error 1 0000000c4f7a00e2:0 2012-04-03 05:35:25 258: [SISNAPI] Async Thread: connection (0x204ec5b0), error (1180682) while reading message"
Kindly help.
Async Thread: connection (0x204ec5b0), error (1180682) while reading message
This happens when an object manager lost the connection to the gateway. There can be many reasons for this: Restart the gateway without bouncing the app server. Network issues... etc.
But, this is the error in your Server Manager session, not in the EIM session (Batch Component). For each EIM job that you start (via server manager) you should see a corresponding EIM tasks. The best is to see the error in the EIMxxxx.log file. Also, you can debug your EIM task by setting Event Logs levels:
change evtloglvl %=3 for comp EIM
(set detailed logging)
(run your EIM job) start task ......
list active tasks for comp EIM
(you should see the job running..)
list tasks for comp EIM
(Or you can see the list of jobs)
change evtloglvl %=1 for comp EIM
(use this line to set the log levels back to "normal")
This will give you some detailed info on what the EIM component is doing. Note: Make use of a small batch or your log will be too big to manage.
If you have some connection errors and you recently lost your DB connection, the best is to completely restart the siebel servers and gateway in the correct order.
Have you tried re-runing the EIM Job.
If the scenario continues even after the second run - Please check the batch number you have given in the IFB file with the batch numbers given in the Input Data file for the EIM component - as from the error it seems that the EIM component is not able to fetch the data.
SBL-SVR-01042 is a generic error when this error is encountered while attempting to instantiate a new instance of a given component and is generic. As to why the error has occurred, one needs to review the accompanying error messages which will help provide context and more detailed information
You can ignore SisnapiLayerLog Error. This is generic error and does not have any significance.
You should concentrate on SBL-EIM-00426. before running task can you check if there is any record in your EIM table. This error comes when you have zero record in interface table.you should increase log level to high and try to trache error. There is also fixed released by Oracle. Refer oracle support for same.
https://support.oracle.com/epmos/faces/BugDisplay?parent=DOCUMENT&sourceId=498041.1&id=10469733
I have edited the IFB file code little bit and it worked for me.
Can you please try the below code and let me know.
[Siebel Interface Manager]
USER NAME = 'SADMIN'
PASSWORD = 'SADMIN'
PROCESS = "PROCESS UPDATE"
[PROCESS UPDATE]
TYPE = SHELL
INCLUDE = "Update Records"
[Update Records]
TYPE = IMPORT
BATCH = 30032012 - 30032015
TABLE = EIM_FN_ASSET5
INSERT ROWS = S_ASSET_CON, FALSE
UPDATE ROWS = S_ASSET_CON, TRUE
ONLY BASE TABLES = S_ASSET_CON
ONLY BASE COLUMNS = S_ASSET_CON.ATTRIB_37 \
,S_ASSET_CON.ATTRIB_38 \
,S_ASSET_CON.ATTRIB_50 \
,S_ASSET_CON.ASSET_ID \
,S_ASSET_CON.CONTACT_ID \
,S_ASSET_CON.RELATION_TYPE_CD
Hope this helps!