I have a question regarding the effect of eventually consistent (EC) microservice systems.
Imagine we have a booking system - a user-service A and booking-service B. Each service has its own database. Imagine the system does a concurrent booking of the same resource for distinct users at the same time. Lets assume we have a Runtime Verification System checking the concurrent booking.
Would it be possible that the monitor does not realize the concurrent booking at B, because the update in the database is done delayed because of the EC mechanism?
In your example, the Booking Service is the source of truth (presumably) for whether or not the resource is available to book. So, that service should be pretty clear on allowing the first booking request to happen and rejecting the second.
In a case like this, where "first come first served" is the requirement, you'd want an intermediate state that would wait for a response from the Booking Service and update the User Service only when a response has been received.
If your architecture is set up right, User Service shouldn't be calling Booking Service directly anyway - it should be communicating through a messaging plane. As such, when the User clicks "Book Now," you could generate a resourceBookingRequested message and submit it to the queue. You'd acknowledge this request has been queued to the user and update their UI to "Awaiting Booking Confirmation..." or something similar.
Once the booking is accepted, or rejected, the User Service subscribes to the resulting message and updates the UI (and/or takes other actions like sending an email) to let the user know their request succeeded or didn't.
Related
I am looking into using an actor model framework (akka.net with akka.net persistence, but I am looking for a general case answer) to build an 'widget order processing workflow'.
Pretty standard:
Customer orders widget
Payment is processed
Email confirmation sent to customer
Send picklist message to warehouse
Warehouse sends a 'widget has been shipped' message back
Send a 'your item has shipped' email to customer
Now let's say between 4 and 5 a server deployment/restart happens. This would cause a actor(s) rehydration (let's assume there is no snapshot yet). That means we would process the payment again, and resend the order placed email. However it turns out our customers don't like this 'feature'!
How to I prevent non-idempotent actions from re-occurring when using an actor model framework?
I have thought about having a separate store of 'payment processed for order db table'; but this feels like I am fighting the framework/paradigm and I wonder if there is a 'proper' way of doing this kind of thing!
Ok so it turns out it is pretty simple.
With akka.net persistence, after a system restore, messages are replayed. The correct state can be recreated by (re) processing these messages.
There is however a IsRestoring property, which can be checked to see if this is the first or a subsequent processing. Presumably other actor model framework have something similar.
So you do something like:
private void ProcessPayment (Order message)
{
if(!this.IsRestoring){
//Perform non-idempotent payment process
}
}
To make a robust workflow processor, you have to store ALL data of a workflow process in a permanent storage.
You can employ a database, a messaging system like Kafka, or use ready-made workflow management software.
Since you already use Akka, Akka Persistence also can be an option.
UPDATE
Building a system which continue to work correctly in presence of system failures and restarts is a considerable task, far more complex than developing an actor framework. That is, you cannot just take any actor framework and add fault tolerance to it.
I am confuse about both communication for the given scenario.I feel that every single list item can be synchronous communication.
Order service calling the shipping service to proceed for shipment.
User buying items from User Interface(UI) Service resulting in
invocation of Order Service.
User Interface(UI) service calling catalog service to get information
about all of the items that it needs to render.
All three examples would be considered asynchronous as they prompt a response due to cause and effect - call and respond. While all three of these could happen concurrently, each in and of themselves is not synchronous.
Synchronous communication happens simultaneously, like two people editing the same document online. Each editor reads and writes at the same time, but does not interrupt the other in any way.
The best example of synchronous communication is a telephone conversation. All connected parties can hear (receive) & speak (transmit) at the same time, and although humans have difficulting performing both actions simultaneously, the telephone connection itself has no trouble providing both concurrently.
Asynchronous acts like a two-way radio. You must stop transmitting in order to receive.
Synchronous = in synch
Sender wait for a response from the receiver to continue further.
Both Sender and Receiver should be in active state.
Sender send data to receiver as it requires an immediate response to continue processing.
When you execute something synchronously, you wait for it to finish before moving on to another task.
Asynchronous = out of synch
Sender does not wait for a response from the receiver
Receiver can be inactive.
Once Receiver is active, it will receive and process.
Sender puts data in message queue and does not require an immediate response to continue processing.
When you execute something asynchronously, you can move on to another task before it finishes.
In your case,
Catalog Service <-- UI --> Order Service --> Shipment service
1) UI has to fetch item details from Catalog Service (Synchronous because it needs item immedietly)
2) Once all items selected, UI has to invoke Order service.(synchronous / asynchronous, depends upon user action)
User might add in shopping cart for future use (or) in favourites (or) to immediate process order.
3) Once all items exist in shopping cart collection , it has to invoke shipmentService. (asynchronous)
Payment should be synchronous. You need acknowledgement.
Assuming all payment and other stuff done, it calls shipment delivery service
Delivery is asynchronous because it cant get acknowledge immedietly. It may take 2 days delay etc.
Given that I am a bit confused with CQRS I would like to understand it further in the following scenario.
I have an Actor that charge Users' credit card. To do so it contact a bank external service that does the operation, get a confirmation result. I would like to know how can I apply this with CQRS.
The information that needs to be written here is that a specific user has been charge a certain amount. So the event generated is Charged (UserID, Card, Amount). Something like that.
The problem is that all the examples I have seen especially with AKKA, would only generate the event after a Command is validated, such that it is persisted in a journal, and used to update the state of the actor. The Journal could then be red on the other side, such that to create a Reading view here.
Also usually, in those examples, the update state function has a logic that somewhat execute the command, because the command correspond straightforwardly to a state update at the end of the day. This is the typical BasketShoping example: CreateOrder, AddLineItem. All Of this Command, are directly translated in Event, that correspond to a specific code of the Update state function.
However in this example, one needs to actually contact an external service, charge the user and then generate an event. Contacting the external service can't be done in the update state, or after reading the journal. It would not make sense.
How is that done, and where, and when exactly, in the spirit of CQRS?
I can think of 2 ways of doing this.
First is a simple way. The command is DoCharge(UserId, Card, Amount). Upon reception of this command, you call the external payment service. If this has been successfully completed, you generate an event, Charged(UserId, Card, Amount, TransactionId) and store it in the journal.
Now, of course, it's not completely safe way, because your Actor can crash after it has sent the request to payment service, but before it has received and persisted the confirmation of the successful completion. Then you risk of charging the user twice. To overcome this risk, you have to make your payment operation idempotent. Here's how to do it. This example is based on the classic "RESTify Day trader" article. I'll summarize it here.
You need to split the payment operation in 2 phases. In first one, payment service creates a transaction token. It just identifies the transaction, and no financial operations are performed yet. Upon the creation, the identifier is received by your service and persisted in the journal.
In next phase you perform a payment associated with the identifier from phase one. If your actor now fails in the middle, while operation is performed successfully on the payment service side, the transaction token will already be marked as processed by the payment service, and it won't let you charge the customer twice. Now, if you restart the failed Actor, and it tries to run the payment associated with the existing transaction token, the payment service should return result like "Already executed" or such. Of course, at the end you also persist the result of the operation in the journal.
I have a situation where I have a component with a list of users. I need to send an email to each user, and then update the user's record to indicate the time when the email was sent.
This seems like a good first use-case for Akka. I was thinking the component that gets the list of users would be an Actor, and a separate Actor would be created for each outbound email-sending Actor, which could be multithreaded. The email-sending Actor would be responsible for updating the date in the user record.
If the email-sending Actor failed, the user's record wouldn't be done, and the supervising Actor would be able to reschedule it.
Is that the right model? Or should the supervisor be responsible for updating the user record?
Or is this a bad use-case for Akka completely?
BTW, I could send mail via SMTP or a web service. I don't think it matters much for the discussion, but I thought I'd mention it.
I don't think you should get hung up on transactionality when there's IO involved. You have no guarantees that the emails will be read, received, received by the right person, received within a certain period of time, be understood etc.
Just send the email and then update the database.
Happy hAkking!
Imagine 3 system components:
1. External ecommerce web service to process credit card transactions
2. Local Database to store processing results
3. Local UI (or win service) to perform payment processing of the customer order document
The external web service is obviously not transactional, so how to guarantee:
1. results to be eventually persisted to database when received from web service even in case the database is not accessible at that moment(network issue, db timeout)
2. prevent clients from processing the customer order while payment initiated by other client but results not successfully persisted to database yet(and waiting in some kind of recovery queue)
The aim is to do processing having non transactional system components and guarantee the transaction won't be repeated by other process in case of failure.
(please look at it in the context of post sell payment processing, where multiple operators might attempt manual payment processing; not web checkout application)
Ask the payment processor whether they can detect duplicate transactions based on an order ID you supply. Then if you are unable to store the response due to a database failure, you can safely resubmit the request without fear of double-charging (at least one PSP I've used returned the same response/auth code in this scenario, along with a flag to say that this was a duplicate).
Alternatively, just set a flag on your order immediately before attempting payment, and don't attempt payment if the flag was already set. If an error then occurs during payment, you can investigate and fix the data at your leisure.
I'd be reluctant to go down the route of trying to automatically cancel the order and resubmitting, as this just gets confusing (e.g. what if cancelling fails - should you retry or not?). Best to keep the logic simple so when something goes wrong you know exactly where you stand.
In any system like this, you need robust error handling and error reporting. This is doubly true when it comes to dealing with payments, where you absolutely do not want to accidentaly take someone's money and not deliver the goods.
Because you're outsourcing your payment handling to a 3rd party, you're ultimately very reliant on the gateway having robust error handling and reporting systems.
In general then, you hand off control to the payment gateway and start a task that waits for a response from the gateway, which is either 'payment accepted' or 'payment declined'. When you get that response you move onto the next step in your process and everything is good.
When you don't get a response at all (time out), or the response is invalid, then how you proceed very much depends on the payment gateway:
If the gateway supports it send a 'cancel payment' style request. If the payment cancels successfully then you probably want to send the user to a 'sorry, please try again' style page.
If the gateway doesn't support canceling, or you have no communications to the gateway then you will need to manually (in person, such as telephone) contact the 3rd party to discover what went wrong and how to proceed. To aid this you need to dump as much detail as you have to error logs, such as date/time, customer id, transaction value, product ids etc.
Once you're back on your site (and payment is accepted) then you're much more in control of errors, but in brief if you cant complete the order, then you should either dump the details to disk (such as csv file for manual handling) or contact the gateway to cancel the payment.
Its also worth having a system in place to track errors as they occur, and if an excessive number occur then consider what should happen. If its a high traffic site for example you may want to temporarily prevent further customers from placing orders whilst the issue is investigated.
Distributed messaging.
When your payment gateway returns submit a message to a durable queue that guarantees a handler will eventually get it and process it. The handler would update the database. Should failure occur at that point the handler can leave the message in the queue or repost it to the queue, or post an alternate message.
Should something occur later that invalidates the transaction, another message could be queued to "undo" the change.
There's a fair amount of buzz lately about eventual consistency and distribute messaging. NServiceBus is the new component hotness. I suggest looking into this, I know we are.