We are using Spring boot Camunda integration.
And I have bpmn process in camunda modedeler which looks like this:
It works as expected but once I tick "Asynchronous After" on "Receive offer" task
I start to get error on "Attempted signing" message correlation:
Cannot correlate message 'attemptedSigningMessage': No process definition or execution matches the parameters
It doesn't look like unlucky timing or any of the such problems. I tried to wait after completion of receive task for up to 10 seconds, it seems enough to come to the wait state on "Attempted signing", it more looks like that the process just stopped on synchronization after "Receive offer" task and waits for something.
I checked that by such code:
historyService.createHistoricActivityInstanceQuery().finished().list()
It is important for me to have "Asynchronous After" after the receive task, to save the state of message receiving.
Please help how it can be accomplished with "Receive task"?
Thank you.
This is a race condition. Confirm that the async job has completed and the token has surely arrived at "Attempted signing" when you are sending the message.
How are you testing this? Are you using a unit test?
If yes, are you moving the job forward using
execute(job());
before attempting the correlation?
In a unit test the job executor is turned off by default, so you can test transaction boundaries. So the process will not automatically move forward after an async continuation. You have to move the job forward explicitly so the token moves to the next wait state (the message receive event here).
Also see: https://github.com/camunda/camunda-bpm-assert/blob/master/docs/User_Guide_BPMN.md#helpers-execute
If no, I recommend to write one, so you can test the model.
Have you maybe set additional async continuations between "Receive offer' and "Attempted signing"?
If none of this helps, can you isolate the behavior and share the model?
Related
I am trying to implement an async GRPC server, where whenever the client makes a call it gets an indefinite stream of messages. I read through the official documentation. It doesn't cover the scenario where I want to keep the stream open for each RPC. This article - https://www.gresearch.co.uk/article/lessons-learnt-from-writing-asynchronous-streaming-grpc-services-in-c/
addresses the issue of keeping the stream open by basically putting the callback handler object in the completion queue again.
The article suggests:
if (HasData())
{
responder_.Write(reply_, this);
}
else
{
grpc::Alarm alarm;
alarm.Set(completion_queue_, gpr_now(gpr_clock_type::GPR_CLOCK_REALTIME), this);
}
I tried using the Alarm object approach as suggested in this article but for some reason on the next Next function call on completion queue I get ok argument as false - GPR_ASSERT(cq_->Next(&tag, &ok));. As a result, I have to close the server and am unable to wait on a stream till further data is available.
I am able to receive data fine till the else case is not hit.
Could someone please help me identify what I might be doing wrong? I am unable to find much C++ resources on GRPC. Thanks!
When Alarm goes out of scope, it will generate a Cancel() causing you to get !ok in Next().
if you want to use this, you would need to put the Alarm into your class scope and trigger it:
std::unique_ptr<Alarm> alarm_;
alarm_.reset(new Alarm);
alarm_->Set(cq_, grpc_timeout_seconds_to_deadline(10), this);
from the doc on Alarm::Set:
Trigger an alarm instance on completion queue cq at the specified
time.
Once the alarm expires (at deadline) or it's cancelled (see Cancel),
an event with tag tag will be added to cq. If the alarm expired, the
event's success bit will be true, false otherwise (ie, upon
cancellation).
I am currently working on a server application in C++. My main inspirations are these examples:
Windows SDK IOCP Excample
The I/O Completion Port IPv4/IPv6 Server Program Example
My app is strongly similar to these (socketobj, packageobj, ...).
In general, my app is running without issues. The only things which still causes me troubles are half open connections.
My strategy for this is: I check every connected client in a time period and count an "idle counter" up. If one completion occurs, I reset this timer. If the Idle counter goes too high, I set a boolean to prevent other threads from posting operations, and then call closesocket().
My assumption was that now the socket is closed, the pending operations will complete (maybe not instantly but after a time). This is also the behavior the MSDN documentation is describing (hints, second paragraph). I need this because only after all operations are completed can I free the resources.
Long story short: this is not the case for me. I did some tests with my testclient app and some cout and breakpoint debugging, and discovered that pending operations for closed sockets are not completing (even after waiting 10 min). I also already tried with a shutdown() call before the closesocket(), and both returned no error.
What am I doing wrong? Does this happen to anyone else? Is the MSDN documentation wrong? What are the alternatives?
I am currently thinking of the "linger" functionality, or to cancel every operation explicitly with the CancelIoEx() function
Edit: (thank you for your responses)
Yesterday evening I added a chained list for every sockedobj to hold the per io obj of the pending operations. With this I tried the CancelIOEx() function. The function returned 0 and GetLastError() returned ERROR_NOT_FOUND for most of the operations.
Is it then safe to just free the per Io Obj in this case?
I also discovered, that this is happening more often, when I run my server app and the client app on the same machine. It happens from time to time, that the server is then not able to complete write operations. I thought that this is happening because the client side receive buffer gets to full. (The client side does not stop to receive data!).
Code snipped follows as soon as possible.
The 'linger' setting can used to reset the connection, but that way you will (a) lose data and (b) deliver a reset to the peer, which may terrify it.
If you're thinking of a positive linger timeout, it doesn't really help.
Shutdown for read should terminate read operations, but shutdown for write only gets queued after pending writes so it doesn't help at all.
If pending writes are the problem, and not completing, they will have to be cancelled.
is there any way to wait till all messages in mailboxes(and stashes) are processed in whole ActorSystem?
I'm trying to develop architecture of integration tests for system based on Akka. And it would be convenient to have method something like actorSystem.awaitAllWorkDone(), in this case arbitrary test would look like:
Start whole environment consisting of many actors
Send message triggering functionality being tested
Call actorSystem.awaitAllWorkDone(). Which waits till all messages are processed, including message sent from step 2 and all inner communication messages which it causes. For example message from step 2 fires 10 other messages sent to other actors, so it should wait for them too.
Now the system in "consistent" state and I can verify it's state. Like verify that all required changes were made in db or all required side-effects were made.
Actually, step 3 can be replaced with waiting for message specific for this test which indicates that all inner-communication are completed. But whole point is to have one "await" method for any test case, and not to searching which message for this test case is indicator that all inner-communication are completed.
Anybody have an idea how to achieve this?
I've read about graceful shutdowns here using the WEBJOBS_SHUTDOWN_FILE and here using Cancellation Tokens, so I understand the premise of graceful shutdowns, however I'm not sure how they will affect WebJobs that are in the middle of processing a queue message.
So here's the scenario:
I have a WebJob with functions listening to queues.
Message is added to Queue and job begins processing.
While processing, someone pushes to develop, triggering a redeploy.
Assuming I have my WebJobs hooked up to deploy on git pushes, this deploy will also trigger the WebJobs to be updated, which (as far as I understand) will kick off some sort of shutdown workflow in the jobs. So I have a few questions stemming from that.
Will jobs in the middle of processing a queue message finish processing the message before the job quits? Or is any shutdown notification essentially treated as "this bitch is about to shutdown. If you don't have anything to handle it, you're SOL."
If we are SOL, is our best option for handling shutdowns essentially to wrap anything you're doing in the equivalent of DB transactions and implement your shutdown handler in such a way that all changes are rolled back on shutdown?
If a queue message is in the middle of being processed and the WebJob shuts down, will that message be requeued? If not, does that mean that my shutdown handler needs to handle requeuing that message?
Is it possible for functions listening to queues to grab any more queue messages after the Job has been notified that it needs to shutdown?
Any guidance here is greatly appreciated! Also, if anyone has any other useful links on how to handle job shutdowns besides the ones I mentioned, it would be great if you could share those.
After no small amount of testing, I think I've found the answers to my questions and I hope someone else can gain some insight from my experience.
NOTE: All of these scenarios were tested using .NET Console Apps and Azure queues, so I'm not sure how blobs or table storage, or different types of Job file types, would handle these different scenarios.
After a Job has been marked to exit, the triggered functions that are running will have the configured amount of time (grace period) (5 seconds by default, but I think that is configurable by using a settings.job file) to finish before they are exited. If they do not finish in the grace period, the function quits. Main() (or whichever file you declared host.RunAndBlock() in), however, will finish running any code after host.RunAndBlock() for up to the amount of time remaining in the grace period (I'm not sure how that would work if you used an infinite loop instead of RunAndBlock). As far as handling the quit in your functions, you can essentially "listen" to the CancellationToken that you can pass in to your triggered functions for IsCancellationRequired and then handle it accordingly. Also, you are not SOL if you don't handle the quits yourself. Huzzah! See point #3.
While you are not SOL if you don't handle the quit (see point #3), I do think it is a good idea to wrap all of your jobs in transactions that you won't commit until you're absolutely sure the job has ran its course. This way if your function exits mid-process, you'll be less likely to have to worry about corrupted data. I can think of a couple scenarios where you might want to commit transactions as they pass (batch jobs, for instance), however you would need to structure your data or logic so that previously processed entities aren't reprocessed after the job restarts.
You are not in trouble if you don't handle job quits yourself. My understanding of what's going on under the covers is virtually non-existent, however I am quite sure of the results. If a function is in the middle of processing a queue message and is forced to quit before it can finish, HAVE NO FEAR! When the job grabs the message to process, it will essentially hide it on the queue for a certain amount of time. If your function quits while processing the message, that message will "become visible" again after x amount of time, and it will be re-grabbed and ran against the potentially updated code that was just deployed.
So I have about 90% confidence in my findings for #4. And I say that because to attempt to test it involved quick-switching between windows while not actually being totally sure what was going on with certain pieces. But here's what I found: on the off chance that a queue has a new message added to it in the grace period b4 a job quits, I THINK one of two things can happen: If the function doesn't poll that queue before the job quits, then the message will stay on the queue and it will be grabbed when the job restarts. However if the function DOES grab the message, it will be treated the same as any other message that was interrupted: it will "become visible" on the queue again and be reran upon the restart of the job.
That pretty much sums it up. I hope other people will find this useful. Let me know if you want any of this expounded on and I'll be happy to try. Or if I'm full of it and you have lots of corrections, those are probably more welcome!
When there is a "FAULT" and "max.delivery.attempts" processes the configured number of times and even then the process continues in "FAULT" logo in the following section "Message Processor Turns" without manual intervention it activates again? The fact that the "Message Processor" DISABLED can not impact the reading of new messages in JMS queue.
Since the Message store and process story implemented in the way to served as First Come First Out basis it is not possible to skip the message that got fault and continue the message flow.
Nevertheless up coming release has a new improvement where you can drop the message out from the queue after x number of fail attempts. Having said that, it is not good practice while you do the schedule and process.
To understand further about Message-stores and Message-processors read on the given article
In order to avoid this situation you can use the Sampling processor and send the message to back-end. Sample process will immediately remove it from the queue and process further. If the delivery of the message is failed or if you find fault you can re added in to store in Fault sequence.