Supposed I have a CQRS-based system and my domain needs some data from an external web service to make its decisions. How do I model this correctly?
I can think of two options:
The command handler runs the domain logic and the domain itself calls out to the web service. Once it gets a response, it attaches the appropriate events to the current aggregate and stores them. The domain basically "waits" for the web service to return.
The command handler runs the domain logic and the domain immediately emits a domain-internal more data needed event. A process manager reacts on this, talks to the web service, reacts on the result, and creates another command on the former aggregate, basically something such as continue.
Which approach is "better", or are both wrong, and I should follow a completely separate way? Basically, I'm fine with option 1, because I think this is basically nothing but a long-running computation inside the domain, but somehow the idea of "waiting" irritates me.
What should I do?
I tend to think of my domain as I do about a physical calculator. It takes input and produces output. That output can be either stored or emitted as events. So in goes data, some behaviour takes place, and out comes data. So very much focused on behaviour.
Your option (1) scenario has resulted in a couple of DDD discussions around injecting services or repositories (or, I guess, an anti-corruption layer) into entities. The general concensus is that it should be avoided and one should opt for, say, double-dispatch. The point is that the domain then needs more information and it either needs to be passed in initially or it needs to be fetched. In my calculator analogy fetching more data is like the calculator prompting you for more input.
If you go with option (1) then whatever is calling the domain needs to handle any web-call failure in order to retry.
If you go with option (2) where you use something like a service bus and, possibly, a process engine of sorts (say saga or workflow) it is quite likely that the service bus handler or the process engine is going to be handling the failures and retries.
I don't think one solution is necessarily 'better' than the other but rather 'different'. I'd go with whatever you feel comfortable with and if you have infrastructure dealing with the failure/retry in some way already then I'd go with the option that is most easily supported by that infrastructure.
Hope that helps :)
Related
How do you integrate applications via web services and deal with technical errors like connectivity errors for web service calls which change state?
E.g. when the network connection gets interrupted during a web service call, how does the client know whether the web services has processed its action or not?
Can this issue be solved at the business layer only (e.g. to query a previous call state) or are you aware of some nice frameworks/best practices which can help wrapping transactional guarantees around a web service?
Implementing it all by yourself with some kind of transactional context tracked in the business layer is always an option. You can use some compensation mechanisms to ensure transactions are rolled back if needed, but you'll need to:
have the information on transactions persisted somewhere
use transaction correlation IDs, so you can query when the response has
been lost (having correlation IDs is good idea anyway)
implement the operations needed to read/write/rollback, etc, so it might make your services a bit more complex
Another option I can think of is If you're using SOAP you can go for asynchronous communication and look for some stack implementing WS-Coordination, WS-AtomicTransaction and WS-BusinessActivity specifications, then decide for yourself if it is a good idea in your context or not. For example, I think Axis2 supports these, but of course eventually it depends on technologies and stack you use.
From the article above:
WS-AtomicTransaction defines a coordination type that is most useful
for handling system-generated exceptions, such as an incomplete write
operation or a process terminating abnormally.
Below are the types of 2-Phase Commit that it implements.
Hope this helps!
I am currently working on a Play! project that has the following architecture:
Controllers -> Services (actors) -> Models (Regular case classes)
For each request that comes in, we will issue a call to the service layers like so:
Service ? DoSomething(request, context)
We have a set number of these service actors behind an akka router that are created during app initialization, and is expandable on demand.
And in the service we mostly do modest data manipulation or database calls:
receive = {
case DoSomething(x, y) => {
...
Model.doSometing(...)
sender ! result
}
}
I am having second thoughts on whether we should be using actors for our services or just use Futures only.
We do not have any internal state that needs to be modified in the service actors, whatever message comes in goes to a function and spits out the result. Isn't this the big strength of the actor model?
We are doing a lot of tasks which seem to take a lot away from the actor model
We aren't doing heavy computation and remoting doesn't make sense because most of the work is for the database and roundtriping to a remote actor to make some db call is unnecessary
We do use reactivemongo, so every db call is non blocking. We can make a lot of these calls
It seems to me that removing akka and just use Futures makes our life a lot easier, and we don't really lose anything.
There certainly is no shortage of opinion on the topic of what should and shouldn't be an actor. Like these two posts:
http://noelwelsh.com/programming/2013/03/04/why-i-dont-like-akka-actors/
http://www.chrisstucchio.com/blog/2013/actors_vs_futures.html
I don't think you're going to find an absolute answer to this question other then that it's situational and it's really up to your preferences and your problem. What I can do for you is to offer my opinion that is based on us implementing Akka for about 2 years now.
For me, I like to think of Akka really as a platform. We come for the Actor Model but we stay for all of the other goodness that the platform provides like Clustering/Remoting, FSM, Routing, Circuit Breaker, Throttling and the like. We are trying to build an SOA like architecture with our actors acting as services. We are deploying these services across a cluster, so we are taking advantage of things like Location Transparency and Routing to provide the ability for a service consumer (which itself could be another service) to find and use a service no matter where it is deployed, and in a highly available manner. Akka makes this whole process pretty simple based on the platform tools they offer.
Within our system, we have the concept of what I call Foundation Services. These are really simple services (like basic lookup/management services for a particular entity). These services generally don't call any other services, and in some cases, just perform DB lookups. These services are pooled (router) and don't usually have any state. They are pretty similar to what you are describing some of your services to be like. We then start to build more and more complex services on top of these foundation services. Most of these services are short lived (to avoid asking), sometimes FSM based, that collect data from the foundation services and then crunch and do something as a result. Even though these foundation services are themselves pretty simple, and some would say don't require an actor, I like the flexibility in that when I compose them into a higher level service, I can look them up and they can be anywhere (location transparent) in my cluster with any number of instances available (routing) for using.
So for us, it was really a design decision to baseline around an actor as a sort of micro-like service that is available within our cluster for consumption by any other service, no matter how simple that service is. I like communicating with these services, where ever they are, through a coarse grained interface in an async manner. A lot of those principles are aspects of building out a good SOA. If that's your goal, then I think Akka can be very helpful in achieving that goal. If you are not looking to do something like that, then maybe your are right in questioning your decision to use Akka for your services. Like I said earlier, it's really up to you to figure out what you are trying to do from an architecture perspective and then design your services layer to meet those goals.
I think you're on right tracks.
We do not have any internal state that needs to be modified in the service actors, whatever message comes in goes to a function and spits out the result. Isn't this the big strength of the actor model?
I found Chris Stucchio's blog (referred to by #cmbaxter above) instantly delightful. My case was so simple that architectural considerations were not a valid point. Just Spray routing and lots of database access, like you have. No state. Thus Future. So much simpler code.
Actor should be crated when you need some really long living stuff with modifying state. In other cases there are no any benefits from actors, especially from non-typed ones.
- do pattern matching every time
- control actor's lifecycle
- remember the things which should not be passed between threads
Why do all of this when you may have simple Future?
There are tasks where actors fit very well, but not everywhere
I was wondering the same and what we decide to do was to use Akka for our data access and it works very well, it's very testable (and tested), and very portable.
We created repositories, long living actors, that we bootstrapped in our App : (FYI, we are using slick for our DB Access, but also have a similar design for our MongoDB needs)
val subscriptionRepo = context.actorOf(Props(new SubscriptionRepository(appConfig.db)), "repository-subscription")
Now we are able to send a "Request" Message for data, ex:
case class SubscriptionsRequested(atDate: ZeroMillisDateTime)
that the actor will respond with
case class SubscriptionsFound(users: Seq[UserSubscription])
or Failure(exception)
In our situation (spray apps but also CLI), we wrapped those calls in a short living actor that take the context and complete on reception and closes itself. (You could handle domain specific logic in those actor, have it to extend another actor that manages its lifecycle and exception so you would only have to specify a partial function for your needs and leave the abstract actor to deal with timeouts, common exceptions etc.
We also have situations where we needed more work to be done in the initiating actor, and it very convenient to fire x messages to your repositories and have your actor storing those messages as they arrive, doing something once they are all there, firing back for completion to the sender( for instance) and close itself.
Thanks to this design, we have a very reactive repository living outside our app, completely tested with Akka TestKit and H2, completely DB agnostic, and it's dead easy, to access data from our DBs (and we never do any ASK, only Tell : Tell to repo, tell to sender, complete or x Tells to repos, pattern match on expected results until completion, tell to sender).
The place I'm working at is trying to establish some ground rules, and the debate we're having now is local libraries vs web services for code reuse. Web services seem to be the popular pick in most companies, and that's what most of the developers here are leaning toward.
I just can't see how you can effectively use web services for any serious work. How can I safely execute multiple service calls if I can't use a transaction?
Let's say I have a cron job that grabs customers from our database who meet a certain condition that they need to be notified of. They are sent a fax, an email, and a ticket is created to track the issue internally. That is 3 different service calls that would happen for each customer in a for loop.
If an error occurs anywhere in there, it's possible that, for example, a fax and email is sent to the customer, but a ticket is not created. Or worse, this cron job could contain a bug on that causes it to fail at the same point every time, and it repeatedly emails the same customer. If the libraries were all local, everything could just be wrapped in a transaction, and none of that would happen. But we're using web services in this example.
Note that the email and fax methods actually insert the data into an email queue and a fax queue, which in turn are handled with a cron job. So the call to the "send email" and "send fax" service methods would be safe to rollback.
An option is to put this entire chunk of code in the web service itself, so the web service itself would call the email, fax, and ticket creating methods in a transaction. But then we're creating a web service method just for the use of a transaction; there is no valid reason we would ever actually need to call this method from anywhere except this one cron script.
How would you generally handle this method?
I would handle it by building a SAGA, an abstraction over a long running business process that has internal state, responds to external events, and interacts with external systems.
I would do this because your problem statement is incomplete: what happens when you can't send the email because the server is down? What if the fax system isn't working, but the other two aren't?
When you can't invoke one, should you retry? For how long? What happens if you can't raise a ticket for four hours, should you escalate to someone? Should this get a response, so something needs to track ticket status and escalate after some time? Should you email the original submitter some time, if you can't carry out any notification actions?
Using a saga is a model for when you can't just have a transaction, because it might potentially span hours of real time before the actions are complete - and holding a database lock that long, ouch.
Moving to a SOA means moving away from some of your older assumptions. One of those is that you should write methods and invoke them, to one where you encapsulate the behaviour of the system at a higher level and expose that as services.
If you try and build a web service that is like a local library, your life is going to suck. Approach this from the view that you want services that own data, and own behaviour related to that data, and encapsulate the details inside that.
(As an aside, I suspect that sending those things off from cron is actually part of a bigger business process, right, where cron does stuff, and sends notification as a consequence of that. Your service might well want to expose that entire sequence as a saga instead.)
Anyway, point is: you don't encapsulate things in a service because you want a transaction, and you don't put things in a transaction just to make them atomic. Those are separate concerns, and should be treated separately.
PS: if you use a transaction, don't you email twice if the email sent but the ticket wasn't created? You actually need a finer grained set of updates anyhow.
I've been tasked with creating an intermediate layer which needs to exchange data (over HTTP) between two independent systems (e.g. Receiver <=> Intermediate Layer (IL) <=> Sender). Receiver and Sender both expose a set of API's via Web Services. Everytime a transaction occurs in the Sender system, the IL should know about it (I'm thinking of creating a Windows Service which constantly pings the Sender), massage the data, then deliver it to the Receiver. The IL can temporarily store the data in a SQL database until it is transferred to the Receiver. I have the following questions -
Can WCF (haven't used it a lot) be used to talk to the Sender and Receiver (both expose web services)?
How do I ensure guaranteed delivery?
How do I ensure security of the messages over the Internet?
What are best practices for handling concurrency issues?
What are best practices for error handling?
How do I ensure reliability of the data (data is not tampered along the way)
How do I ensure the receipt of the data back to the Sender?
What are the constraints that I need to be aware of?
I need to implement this on MS platform using a custom .NET solution. I was told not to use any middleware like BizTalk. The receiver is an SDFC instance, if that matters.
Any pointers are greatly appreciated. Thank you.
A Windows Service that orchestras the exchange sounds fine.
Yes WCF can deal with traditional Web Services.
How do I ensure guaranteed delivery?
To ensure delivery you can use TransactionScope to handle the passing of data between the
Receiver <=> Intermediate Layer and Intermediate Layer <=> Sender but I wouldn't try and do them together.
You might want to consider some sort of queuing mechanism to send the data to the receiver; I guess I'm thinking more of a logical queue rather than an actual queuing component. A workflow framework could also be an option.
make sure you have good logging / auditing in place; make sure it's rock solid, has the right information and is easy to read. Assuming you write a service it will execute without supervision so the operational / support aspects are more demanding.
Think about scenarios:
How do you manage failed deliveries?
What happens if the reciever (or sender) is unavailbale for periods of time (and how long is that?); for example: do you need to "escalate" to an operator via email?
How do I ensure security of the messages over the Internet?
HTTPS. Assuming other existing clients make calls to the Web Services how do they ensure security? (I'm thinking encryption).
What are best practices for handling concurrency issues?
Hmm probably a separate question. You should be able to find information on that easily enough. How much data are we taking? what sort of frequency? How many instances of the Windows Service were you thinking of having - if one is enough why would concurrency be an issue?
What are best practices for error handling?
Same as for concurrency, but I can offer some pointers:
Use an established logging framework, I quite like MS EntLibs but there are others (re-using whatever's currently used is probably going to make more sense - if there is anything).
Remember that execution is unattended so ensure information is complete, clear and unambiguous. I'd be tempted to log more and dial it down once a level of comfort is reached.
use a top level handler to ensure nothing get's lost; but don;t be afraid to log deep in the application where you can still get useful context (like the metadata of the data being sent / recieved).
How do I ensure the receipt of the data back to the Sender?
Include it (sending the receipt) as a step that is part of the transaction.
On a different angle - have a look on CodePlex for ESB type libraries, you might find something useful: http://www.codeplex.com/site/search?query=ESB&ac=8
For example ESBasic which seems to be a class library which you could reuse.
I am developing a Windows Phone app where users can update a list. Each update, delete, add etc need to be stored in a database that sits behind a web service. As well as ensuring all the operations made on the phone end up in the cloud, I need to make sure the app is really responsive and the user doesn’t feel any lag time whatsoever.
What’s the best design to use here? Each check box change, each text box edit fires a new thread to contact the web service? Locally store a list of things that need to be updated then send to the server in batch every so often (what about the back button)? Am I missing another even easier implementation?
Thanks in advance,
Data updates to your web service are going to take some time to execute, so in terms of providing the very best responsiveness to the user your best approach would be to fire these off on a background thread.
If updates not taking place (until your app resumes) due to a back press is a concern for your app then you can increase the frequency of sending these updates off.
Storing data locally would be a good idea following each change to make sure nothing is lost since you don't know if your app will get interrupted such as by a phone call.
You are able to intercept the back button which would allow you to handle notifying the user of pending updates being processed or requesting confirmation to defer transmission (say in the case of poor performing network location). Perhaps a visual queue in your UI would be helpful to indicate pending requests in your storage queue.
You may want to give some thought to the overall frequency of data updates in a typical usage scenario for your application and how intensely this would utilise the network connection. Depending on this you may want to balance frequency of updates with potential power consumption.
This may guide you on whether to fire updates off of field level changes, a timer when the queue isn't empty, and/or manipulating a different row of data among other possibilities.
General efficiency guidance with mobile network communications is to have larger and less frequent transmissions rather than a "chatty" or frequent transmissions pattern, however this is up to you to decide what is most applicable for your application.
You might want to look into something similar to REST or SOAP.
Each update, delete, add would send a request to the web service. After the request is fulfilled, the web service sends a message back to the Phone application.
Since you want to keep this simple on the Phone application, you would send a URL to the web service, and the web service would respond with a simple message you can easily parse.
Something like this:
http://webservice?action=update&id=10345&data=...
With a reply of:
Update 10345 successful
The id number is just an incrementing sequence to identify the request / response pair.
There is the Microsoft Sync Framework recently released and discussed some weeks back on DotNetRocks. I must admit I didnt consider this till I read your comment.
I've not looked into the sync framework's dependencies and thus capability for running on the wp7 platform as yet, but it's probably worth checking out.
Here's a link to the framework.
And a link to Carl and Richard's show with Lev Novik, an architect on the project if you're interested in some background info. It was quite an interesting show.