I'm looking to build my first web service and I would like to be pointed in the right direction right from the start.
Here are the interactions that must take place:
First, a smartphone or computer will send a chunk of data to my web service.
The web service will persist the information to the database.
Periodically, an algorithm will access and modify the database.
The algorithm will periodically bundle data and send it out to smartphones or computer (how?)
The big question is: What basic things do I need to learn to in order to implement something like this?
Now here are the little rambling questions that I've also got rolling around in my head. Feel free to answer these if you wish. (...maybe consider them extra credit?)
I've heard a lot of good things about RESTful services, I've read the wiki article, and I've even played around with the Twitter's webservice which is RESTful. Is this the obvious way to go? Or should I still consider something else?
What programming language do I use to persist things to the database? I'm thinking php will be the first choice for this.
What programming language do I use to interact with the database? I'm thinking anything is probably acceptable, right?
Do I have to worry about concurrent access to the database, or does MySQL handle that for me? (I'm fairly new to databases too.)
How on earth do I send information back? Obviously, if it's a reply to an HTTP request that's no problem, but there will be times when the reply may take quite a long time to compute. Should I just let the HTTP request remain outstanding until I have the answer?
There will be times when I need to send information to a smartphone regardless of whether or not information has been sent to me. How can I push information to my users?
Other information that may help you know where I'm coming from:
I am pretty familiar with Java, C#, C++, and Python. I have played around with PHP, Javascript, and Ruby.
I am relatively new to databasing, but I get the basic idea.
I've already set up my server and I'm using a basic LAMP stack. My understanding of L, A, M and P is fairly rudimentary.
Language: Python for it's ease of use assuming the GIL is not a particular concern for your requirements (e.g. multi-threading). It has drivers for most databases and supports numerous protocols. There are several web frameworks for it - the most popular probably being Django.
Protocols: if you are HTTP focused study SOAP and REST. Note, SOAP tends to be verbose, which causes problems moving volumes of data. On the other hand, if you are looking at other options study socket programming and perhaps some sort of binary format such as Google's protocol buffers. Flash is also a possibility (see: Flash Remoting). Note, binary options require users install something onto their machine (e.g. applet or standalone app).
Replies: if the process is long running, and the client should be notified when it's done, I would recommend developing an app for the client. Browser's can be programmed with JavaScript to periodically poll, or a Flash movie can be embedded to real time updates, but these are somewhat tricky bits of browser programming. If you're dealing with wireless phones, look at SMS. Otherwise I would just provide a way for clients to get status, but not send out notification (e.g. push vs. pull). As #jcomea_ictx wrote, AJAX is an option if it's a browser based solution - study jQuery.
Concurrency: understand what ACID means with regards to databases. Think about what should happen if you receive multiple writes to the same data - database may not necessarily solve this problem the way you'd want.
Please, for the love of programming, don't use PHP if you're already comfortable with Python. The latter makes for far cleaner, more maintainable code. Not that it's impossible to write good code in PHP, but it's a relative rarity. You can use Python for all the server-end stuff including MySQL interaction, with the MySQLdb module. Either with standard CGI, or FCGI, or mod_python.
As for the database, use of transactions will eliminate conflicts. But you can usually design a system in such a way that conflicts will not happen. For example, use of auto-incrementing primary-key IDs on each insert will make sure that every entry is unique.
You can "pull" data with Javascript, perhaps using AJAX methodology, or "push" using SMS or other technologies.
When replies take a while to compute, you can "poll" using AJAX. This is a very common technique. The server just returns "we are working on this" (or equivalent) with a built-in refresh until the results are ready.
I'm no expert on REST, but AJAX, especially when using polling rather than simply responding to user input, can be said to violate RESTful principles. But you can be a purist, or you can do whatever works. It's up to you.
I don't believe I've ever used any "push" technologies other than SMS, and that was years ago when many companies had free SMS gateways. So if you want to "push" data, better hope someone else joins in the conversation!
Use Java. The latest version of Java EE 6 makes coding RESTful and SOAP services a breeze, and it interoperates with databases very easily as well.
The benefits of using a true language instead of a script are: full server-side state, strong typing, multithreading, and countless other things that may or may not come in handy, but knowing they are available makes your project future proof.
Related
I am looking at developing a HTTP-based server for storing and later retrieving internal assets (just files/blobs of data). Clients should be able to connect with a simple HTTP request and retrieve data, or use a RESTful API to store files.
When a file is stored, we will need to make some calls into preexisting C++ code to update our asset database, set up internal linkages, and run various checks (e.g. is the data sane).
We have mountains of existing C++ tech here and it's not going to be practical to rewrite it in a high-level language, so we'd like to reuse as much of that as we can. However, in our brief research, it seems like putting together a performant HTTP server is absolutely trivial in higher-level languages, and we'd like to use as much off-the-shelf tech as possible for the HTTP server part.
Anyone have any experience/insight with this? We looked at Node.js, and it seemed very powerful, and then we looked at its C++ integration and got scared off. We aren't going to be able to rewrite all of our low-level tech to be non-blocking, and Node.js seems to require that. Others suggested Ruby, which we have some experience with, but we've never added C++ to the mix before. Totally open to other ideas as well.
Thanks guys!
I would use http://code.google.com/p/mongoose/ .
Mongoose is written in C and easily integrated into existing code.
Maybe you can try poco:
http://pocoproject.org/features.html
Poco is a good c++ library and they provide a HTTP server framework.
I am in the process of developing a rich internet web application that communicates to a (Java) back-end via web services. I have prototyped a user interface in both Flex/Flash and GWT/Javascript and have noticed that these RIA platforms tend to favor an RPC-style method of back-end communication (AMF for Flex and GWT-RPC for GWT).
In my case, the server also needs to provide web services other clients that I do not author. Because of this, I am leaning towards standards-based web services (e.g. SOAP and REST). I am convinced that the RIA must use the same web service we provide for others. I "get" SOAP because it models the RPC style I am familiar with from experience. I am new to REST, but have prototyped a REST back-end using CXF/Jackson. At this time, however, my REST API still feels like an RPC-style API and I realize it's because I am having trouble getting my head around the idea of HATEOAS.
I have read Roy T. Fieldings helpful blog post about 10 times and I think I am beginning to see the light. For example, it is clear to me that if I were to include links to various state transitions along with my resource I could really reduce the amount of coupling between my client and server. My client could just render buttons that provide the user with access to the legal operations that can be performed on the displayed entity at that time.
But does loose coupling between a RIA and its server application matter?
By their very nature, RIAs are pretty tightly coupled with the server data model. Out of the box they presuppose many things. I am guessing that is why they also prefer an RPC-style application protocol...because loose coupling is not a design goal. But I am beginning to realize that if we took HATEOAS seriously, we could write a much more generic RIA client that would make VERY few assumptions about the data model and operations that can be performed. That could reduce the amount of effort to maintain the client through changes in the back-end, but does this make sense? does the benefit outweigh the cost?
p.s. - Two more details -- This application has an extremely complex, deeply nested data model. Also, I could not care less if somebody tells me we are not a 100% pure REST web-app.
It's an excellent philosophical question. My general response is some coupling should be expected.
Let me explain more. While it's possible to conceive of a fully generic application interface that just exposes the model in a human-usable way, it's actually incredibly difficult to write such a piece of software except for truly miniscule domains (e.g., filling a record that will be used to populate a DB where all the fields are picked from finite simple enumerations). If your application doesn't fit that model, you're going to have to have something in there that is specific to the app. If you do that in a “generic” way, you'll be downloading a complex description of what your generic client app is supposed to do, and that description will itself start to feel more and more like a programming language. Now you're back to square one, except with a (probably badly-designed) new domain-specific language in the mix as well. You might as well cut to the chase and accept that going wholly generic isn't worthwhile.
But that's not to say that you shouldn't take care to think carefully about what resources you expose, what verbs apply to those operations, and how users' software will discover those resources. Following REST and HATEOAS there will help a lot (and it helps if you've got a clear idea about what the application's underlying abstract model is; you should aim to expose that in a natural way).
Given that the GWT app is served by HTTP, having it tightly coupled with the server is not violating HATEOAS. It's" code on demand".
Google, Twitter and Facebook all use specific APIs for their app, different from the one exposed to third parties (Twitter has recently moved to using their public API for their web app, but it wasn't originally the case). Google said they had no plan to move G+ over to its public API, because it allows them to experiment and make breaking changes without breaking third parties.
I've been working on a rails app for a couple of days now that I need an underlying "middle layer" for that connects my rails application to the various services that make up the data.
The basic setup looks like this:
Frontend ("Rails app") -> user requests data to be aggregated -> info goes in database and a JSON request is sent to "middle layer" to retrieve the data from a source of other places, process it, then send it back to the frontend which streams it to the users browser via websockets.
Middle layer -> uses sockets to listen for the frontend making a request. Once request is made, aggregation begins.
Base layer -> load balancing among a scalable network design.
I don't think that this is as efficient as it could be though. I feel like I'll run into concurrency problems or it just being too slow for use. I need to be able to scale it up.
My main question though rests in the area of what language would be more efficient for this to run fast?
It depends. What are your data sources? Are they on the same machine? Are they databases?
My gut tells me that the language you choose won't play much of a role in how well this kind
of application performs, but it it hard to say without knowing the details.
C++ is probably a bad idea for a scalable web-app though. It is very likely that you will end
up with something which is slower than what you would have written in Ruby, because you end up worrying about irrelevant details. With regards to concurrency concurrency problems, C++ is definitely not the easiest language in which to write concurrent code.
Without knowing more, I would recommend that you stick with Ruby or some other high-level language and profile to see where the bottlenecks are. If you find out that there is some tight loop which needs to run really fast you can write that part in C, but you probably won't need that.
Stick with Ruby unless you can prove an actual need for C++. Look into something like delayed_job to handle your background tasks.
Following "A well earned retirement for the SOAP Search API" from Google announcing they have recently removed their SOAP APIs, I'm curious what the community thinks of SOAP in 2009. I can see its use for remoting and more verbose client-server stateless communication, but for more generalised [Ajax] web usage is it now redundent?
Have REST URLs removed the need for SOAP and that kind of web service once and for all?
There is nothing in REST that says you can't use form POST fields to PUT data for when you need to send complex requests over. You can even post big blocks of bulky XML if you want to try and make it as SOAPy as possible.
IMHO SOAP gives you nothing but a wrapper that you never needed to begin with. The thing that killed it for me was the way Axis and other engines compile stubs of your WSDL into their code and then every time you add something to the WSDL it breaks the consumers, even if everything was designed to be backward compatible. REST forever.
SOAP is here to stay - and rightfully so.
In an enterprise environment, things like self-describing services (with the help of WSDL), ability to use transactions and reliable messaging are paramount. They're much more important than the running after the "rave of the day".
REST has its good uses - but it cannot ever replace SOAP totally, nor should it. REST is great for lightweight communcation - twittering and the like. But there's also good reason to have and know about SOAP.
SOAP currently has the much better tooling support in most environments - it'll be some time before REST has something comparable.
SOAP allows for machine-readable service description and service discovery - REST has nothing like that, your REST service might - or might not - be documented, and the quality of the English prose documenting your REST services varies wildly.
Yes, REST is all the rage right now - and it does make a lot of fun scenarios a lot easier to handle. But I don't think it's ready for "prime-time, enterprise-level" use, really. Maybe some day - but not today.
Oh if only SOAP were dead. I can assure you that some companies are still pursuing SOAP-based, RPC strategies as fast as they can.
As you already said -- REST can't deal with the verbose cases.
If you can tell me how I can take an arbitrary number of complex arguments through a RESTful web service, without hitting URL length restrictions, I'd love to hear it.
But for complex queries of scientific data, we need something more than positional parameters or key/value pairs.
My prediction is that SOAP won't die until sometime after COBOL and Fortran.
I have 2 websites(www.mysite1.com and myweb2.com, both sites are in ASP.NET with SQL server as backend ) and i want to pass data from one site to another.Now i am confused whether to use a web service or a form posting (from mysite1 to a page in myweb2)
Can any one tell me the Pros and Cons of both ?
By web service I assume you mean SOAP based web service?
Anyway both are equal with few advantages. Posting is more lightweight, while SOAP is standardized (sort of). I would go with more restful approach, because I think SOAP is too much overhead for simple tasks while not giving much of advantage.
Webservices are SOAP messages (the SOAP protocol uses XML to pass messages back and forth), so your server on both ends must understand SOAP and whatever extensions you want to talk about between them, and they probably (but don't have to) be able to grok WMDL files (that "explains" the various services endpoints and remote functionality available). Usually we call this the SOAP / WS-* stack, with emphasis on 'stack' as there's a few bits of software that needs to be available, and the more complex the SOAP calls, the more of this stack needs to be available and maintained.
Using POST, on the other hand, is mostly associated with RESTful behaviours, and as an example of a protocol of such, look to HTTP. Inside the POST you can of course post complex XML, but people tend to use plain POST to simplify the calling, and use HTTP responses as replies. You don't need any extra software, probably, as most if not all webkits has got HTTP support. My own bias leans towards REST, in case you wonder. Through using HATEOAS you can create really good infrastructure for self-aware systems that can modify themselves with load and availability in real-time as opposed to the SOAP way, and this lies at the centre of the argument for it; HTTP was designed for large distributed networks in mind, dealing with performance and stability. SOAP tends to be a one-stop if-it-breaks-you're-stuffed kinda thing. (Again, remeber my bias. I've written about this a lot on my blog, especially the architecture side and the impact of SOA vs. ROA. :)
There's a great debate as to which is "better", to which I can only say "it depends completely on what you want to do, how you prefer to do it, what you need it to do, your environment, your experience, the position of the sun and the moon(s), and the mood my cat is in." Eh, meaning, a lot.
I'm all for a healthy debate about this, but I tend to think that SOAP is a reinvention; SOAP is an envelope with a header and body, and if that sounds familiar, it is exactly how HTML was designed, a fact very few people tend to see. HTTP as just a protocol for shifting stuff around is well understood and extremely well supported, and SOAP uses it to shift their XML envelopes around. Is there a real difference between shifting SOAP and HTML around? Well, yes, the big difference is that SOAP reinvents all the niceties of HTTP (caching, addressability, state, scaling), and then use HTTP only for delivering the message and nothing else and let the stack itself have to deal with those niceities mentioned earlier. So, a lot of the goodness of HTTP is ignored and recreated in another layer (hence, you need a SOAP stack to deal with it), which to me seems wasteful, ignorant and adding complexity.
Next up is what you want to do. For really complex things, there's lots in the webservices stack of standards (I think it's about 1200 pages combined these days) that could help you out, but if your needs are more modest (ie. not that crazy about seriously complex security, for example) a simple POST (or GET) of a request and an envelope back with results might be good enough. Results in HTTP is, as you probably know, HTTP content-type, so lots is already supported but you can create your own, for example application/xml+myformat (or more correctly, application/x-xml+myformat if I remember correctly). Get the request, if it's a response code 200, and parse.
Both will work. One is heavy (WS-* stack) depending on what your needs are, the other is more lightweight and already supported. The rest is glue, as they say.
I would say the webservice is definitely the best choice. A few pro's:
If in the future you need to add another website, your infrastructure (the webservice) is already there
Cross-site form posting might give you problems when using cookies or
might trigger browser privacy restrictions
If you use form posting you have to
write the same code over and over
again, while with using the
webservice you write the code once,
then use it at multiple locations.
Easier to maintain, less code to
write.
Maintainability (this is related to
the above point) ofcourse, all the
code relevant to exchanging data is
all at one location (your webservice)
There's probably even more. Like design time support/code completion.
From my little experience I'd say that you'd be best using a web service since you can see the methods and structure of the service in your code, once you've created it at the recieving end that is.
Also using the form posting methos would eman you have to fake form submissions which isn't as neat as making a web service call.
Your third way would be to get the databases talking, though I'm guessing they're disparate and can't 'see' each other?
I would suggest a web service (or WCF). As Beanie said, with a service you are able to see the methods and types of the service (that you expose), which would make for much easier and cleaner moving of data.
I agree with AlexanderJohannesen that it is debatable whether SOAP webservices or RESTful apis are better, however if both sites are under your control and done with asp.net, definitely go with SOAP webservices. The tools that Visual Studio provides for creating and consuming webservices are just great, it won't take you more than a few minutes to create the link between the two sites.
In the site you want to receive communication create the web service by selecting add item in VS. Select web service and name it appropriately. Then just create a method with the logic you want to implement and add the attribute [WebMethod], eg.
[WebMethod]
public void AddComment(int UserId, string Comment) {
// do stuff
}
Deploy this on your test server, say tst.myweb2.com.
Now on the consuming side (www.myweb1.com), select Add Web Reference, point the url to the address of the webservice we just created, give it a name and click Add refence. You have a proxy class that you can call just like a local class. Easy as pie.