Lets say I have web applicatons/services:
API
Set of Applications
API is used for managing some resources (simple CRUD operations). Now what I need is to subscribe Applications for changes of different API resources. Applications would do some background work on a change.
I came up to idea of callbacks. So that Applications can oauthorise and post to the API a callback config.
I think that this config should look like this:
{
'callback_url': 'http://3rdpartyservice.com/callback',
'resources': ['foo1', 'foo2'],
'ref_data': { 'token': 'abcd1234' }
}
resources is array of the resources that 3rd party service is interested in
ref_data is custom json for 3rd party usage (e.g. for auth)
This way on specified resource change the API would send a request to callback_url. This request would contain resource data, action(create/update/delete) and ref_data.
The intention here is to make this generic enough to allow 3rd party clients configure such callbacks.
So the question are:
Are there any best practices?
What about security potential issues?
Are there any real world examples on the web?
Tx
Sounds very similar as WebHooks or Service Hooks.
Check out the Web Hooks on GitHub, to get a good idea what they are and how they work. See also last alinea Service Hooks, as it explains how github handles these WebHooks. This would be similar for your application. The OAuth explains why and how it is done.
See also Webhooks, REST and the Open Web, from API User Experience.
There is even RestHooks.
The general solution to this requirement is usually called "publish/subscribe". There are dozens of solutions to this - google "publish subscribe REST" for some examples. You can also read "Enterprise Integration Patterns".
They key challenge in this kind of solution is "real-time versus queue".
For instance, if you have an API with a million clients, who are all interested in the same event, you cannot guarantee that in real time you can reach all of those clients within whatever timeframe their application demands. You also have to worry about the network going away, or clients being temporarily down. In this case, you application might define an event queue, and clients look in that queue for events they're interested in. Once you go down that route, you're probably going to use some off-the-shelf software rather than building your own. Apache Camel is a good open source implementation.
In your example, for instance, what happens if you cannot reach 3rdpartyservice.com? Or if http://3rdpartyservice.com/callback throws an error when posting an update to foo1, but not to foo2? Or if http://3rdpartyservice.com/ uses a different flavour of OAuth than you're used to? How do you guarantee http://3rdpartyservice.com/ that it's you who is posting an update, not a hacker?
Your choices really tend to come down to your non-functional requirements, rather than the functional ones - things like uptime, guarantee of notification, guarantee of delivery, etc. are more important than the specifics of how you pass across the parameters, and whether it's "resource-based" or some other protocol.
Related
How do you integrate applications via web services and deal with technical errors like connectivity errors for web service calls which change state?
E.g. when the network connection gets interrupted during a web service call, how does the client know whether the web services has processed its action or not?
Can this issue be solved at the business layer only (e.g. to query a previous call state) or are you aware of some nice frameworks/best practices which can help wrapping transactional guarantees around a web service?
Implementing it all by yourself with some kind of transactional context tracked in the business layer is always an option. You can use some compensation mechanisms to ensure transactions are rolled back if needed, but you'll need to:
have the information on transactions persisted somewhere
use transaction correlation IDs, so you can query when the response has
been lost (having correlation IDs is good idea anyway)
implement the operations needed to read/write/rollback, etc, so it might make your services a bit more complex
Another option I can think of is If you're using SOAP you can go for asynchronous communication and look for some stack implementing WS-Coordination, WS-AtomicTransaction and WS-BusinessActivity specifications, then decide for yourself if it is a good idea in your context or not. For example, I think Axis2 supports these, but of course eventually it depends on technologies and stack you use.
From the article above:
WS-AtomicTransaction defines a coordination type that is most useful
for handling system-generated exceptions, such as an incomplete write
operation or a process terminating abnormally.
Below are the types of 2-Phase Commit that it implements.
Hope this helps!
I am doing some research on SOAP, for a personal project, and I came across a website with a list of pros and cons for using SOAP, and I understood what most of them meant, except for this one under disadvantages:
SOAP is typically limited to pooling, and not event notifications, when leveraging HTTP for transport. What's more, only one client can use the services of one server in typical situations.
From my understanding of pooling, there should be no issue pooling a SOAP Object for re usability. Pooling is simply a way to use the same resources over and over again, like a connection to a database. Also not entirely certain on the context of Event Notifications.
So my two questions here are, what does the above block quoted text actually mean, and is this information correct?
Website: http://searchsoa.techtarget.com/definition/SOAP
SOAP is RPC, and in RPC some local client invokes a method on some remote target and receives a result. That's how it works, so SOAP works that way too. A client invokes a service asking for something and the service just responds.
If you want "events" in this type of communication the most simple approach is to invoke the service more often (i.e. polling). This has the advantage that nothing changes for the server or the client. It's the same RPC call but done more frequently.
These days everyone is connected to the web and everyone is subscribed to all sorts of services. They want to get notified as soon as something happens to the world around them. Pooling becomes inefficient in this sea of users and services because you are wasting resources. You might poll a service a hundred times just to get back one notification. For this reason technology is evolving so that resource use is minimized. And the direction this is moving to is push services.
Now almost everything happens in the browser. Every browser manufacturer rushes to implement the latest technology changes and HTML5 spec. This means actual pages that push notifications to users instead of faking it with Ajax, comet, etc.
SOAP has been around since 1998 and it's not moving as fast as the rest of the web, mainly because SOAP is mostly an enterprise player and because it's a protocol. Because it's a protocol you have to make new technology available to it without breaking that protocol. Things move slower so people have abandoned SOAP in favor of other ways of doing server-client communication.
SOAP is typically limited to pooling, and not event notifications...
That is correct. But be aware that "typically" does not mean "always".
You can have events, but it's harder. It involves using WS-* specifications like WS-Eventing and WS-Addressing. This is a change in the way SOAP clients operate because a client now becomes some sort of a service too because it needs to receive calls too, not just initiate them. If your technology stack implements these specifications then good for you, but if it doesn't, then you have to build it yourself and it's a real pain.
So for these reasons, if you don't have blocking performance or resource usage issues, you "typically" chose doing polling with SOAP and not event notifications.
In my website, I'd like to create a public API that would allow clients (unknown people) to interact with my services. A classic REST API would work well in that case.
However, I need to be able to send events to the clients too. These events are not related to client HTTP requests. I saw "webhooks" are a way to deal with this. If I understood well, with webhooks, my service would send HTTP POST requests to a URL specified by the client, with event data inside this request.
I think websocket can be used too as a solution for this full-duplex communication need.
What I want to know, is which method would be the simplest for clients to implement to talk to my services? Simplicity is the key point here.
The hard thing is that my clients can use various technologies (full websites with HTTP servers, iOS/Android apps without server, etc.)
What are implications for clients if I use REST API + webhooks? Websockets? etc?
How to make a choice?
Hope it's clear (but not sure). Thanks :)
I would consider webhooks a simpler solution. And yes, you understood it well, that with webhooks, a developer using your API would register a URL where your backend would POST event data. It's a common pattern that's used in APIs.
A great benefit of using a webhooks design is that a client/server connection does not need to stay open. After all, if events occur infrequently (i.e. only a few times per hour, or per day) or keeping a consistent connection open is a challenge, establishing a connection only when it's needed is rather efficient.
The challenge of using webhooks for you, the API provider, is designing an evented backend system that deals with change of state detection and reliable webhook calling mechanisms (i.e. dealing with webhook receiver URLs that are unresponsive or throw errors).
The challenge of using webhooks on the developer end is that they need to stand up a reliable web server that listens for the event POST data from your server.
Realtime APIs (i.e. based on Websockets, Bayeux/CometD) are really swell because that live connection means that new connections do not have to be established, which is particularly useful with very chatty sessions. Additionally, there are a lot of projects and companies out there that have taken care of the heavy lifting on the server and client with fully-baked libraries. One of those is Fanout.io which makes pushing messages between the client/server possible with just a few lines of code, utilizing XMPP, Bayeux, and Websockets when possible.
(I am not affiliated with Fanout, but I have used it)
So, to sum it up, webhooks are simple mostly because you are already familiar with the architecture needed to implement them, and the pattern is a well traveled one. If you are leaning toward a persistent connection approach, I would look at tools/platforms like Fanout because it takes care of the heavy lifting (i.e. subscribe/publish, concurrent connection scale, client/server libraries).
I want to enable event notifications for my customers. There are many possible ways to send notifications: emails, sms, XMPP/other IM, pre-recorded voice messages over SIP, phone-specific message push services, REST callbacks etc.
I don't want to develop all these transports myself, so I need a web service that can manage those notifications for customers. Also I don't want to store emails/phones/other personally identifiable information.
The notifications are transactional (i.e. it's not mass delivering same message to everyone). Paid solutions are welcome.
There is http://pagerduty.com but it is
designed to work within enterprise and not with outside customers
focused on full cycle of incident response as opposed to simple message delivery
So it puts more burden on respondents and I want something that requires zero effort for the users to setup.
Monitis is another example. It has multiple transports including Twitter, but again it's designed for insiders and not for service subscribers coming in bulk numbers.
Amazon SNS seems to be too low-level as it only manages delivery of push notifications, but for diplaying them I have to write a mobile app which I don't want.
XMPP servers as described in How best to deliver notifications to various IM / notification services? have traditionally supported the idea of different transports, but I'd like a third-party hosted service.
Twilio has only 2 transports: SMS and voice call and more oriented on full 2-side communications.
I cannot even find the right google keywords to search for the service/SaaS I want.
The question is, are there any such services? A sample of a few would give me an idea of what to look for.
This comes very late, perhaps too late but...
You should not need to implement any of the transport but you may be required to build some of the gateways and you will most likely need to assembly the application which talks to each of the gateways. You are not likely going to find a single service for this.
You've already outlined the strategy. You basically have these pieces:
transports
gateways
application
Each of the transports is accessed through some client via either an API or a CLI - so you'll need to figure out what your environment is. Java is probably a good choice but other cross-platform environments would likely work. Existing infrastructure like Apache ServiceMix has support for some of these transports:
https://cwiki.apache.org/confluence/display/SM/Components+list
and there may be other middle-ware with similar, distinct transports.
You will likely want a gateway for each provider for each transport type. You may be able to find a provider which adequately services multiple transports, e.g. Twilio's SMS and voice, but that will likely be the exception. You may also find that because of the differences in transports (and therefore functionality), it's more convenient to build a gateway for each transport type. So, you might have two configured providers in your SMS gateway, one for Twilio and one for Kannel, and you might have your Twilio account used in the SMS gateway and in the SIP gateway.
The final step is assembling your application into something meaningful. This might be something like:
sent.......: "Thanks for your purchase, ${username}!"
sent to the channels (i.e., provider-transport pair) configured, perhaps, by the user and being able to collect the response from the user:
response...: "It was a pleasure! --Bob"
You will need to store the basics of the each transport's endpoint, e.g., phone number for SMS, username for chat, etc., so if you have PII security issues to address you'll need to think though that. One option may be to turn all the PII over to each provider but you'll still need to keep each account for your users in each provider, and you will likely need to know something about the user, like "${username}" above, to personalize your notification appropriately within your application. So, removing all PII from your application seems unlikely.
I'm not sure how much this help but perhaps it gives you some ideas.
I would like to use Amazon SQS in my application to queue requests from other external systems that don't belong to me.
What is the better way of doing this, directly expose the SQS Queue and the required messageformat OR publish a web service (WCF) that queues the request.
Also I read that SQS is relative slow for a singe access, but am I right that it can handle easyly a lot of concurrent accesses from different clients?
Best
Thomas
This is largely a matter of preference and depends a bit on your situation. But my recommendation would be to wrap it with your own web-service.
Building your web-service allows you to do things like validation, throttling, schema versioning etc. E.g. you can reject invalid messages with immediate synchronous feedback to the sender. If the external systems are publishing directly to your queue, then invalid messages become your problem not theirs, and if you revise your schema and want to reject old-schema messages then you either have to drop them or set up a separate back-channel to feed back information to the publisher. That adds unnecessary complexity to your system. Having a web-service would even let you switch to other queuing technologies later if you need to.
But building your own web-service has downsides too: will your own service be able to handle the same load as the SQS API with the same low latency? It won't scale infinitely like SQS, so how responsive will you need to be to changes in load? Have you got the resources to manage a separate service? And it's more work than just giving a client's AWS account permission to publish to your queue.
If you're happy with the extra work involved, and you want a more future-proof system, IMHO it's worth building the web-service wrapper.