Related
I'm not the only one with this question, but haven't found a lot of information in my research so far, so help me out.
We are a small IT crowd in an organization. We're looking to build a small, private service that would emulate a heroku/gae workflow. The basics of this: deploy an app as a git repository, and have it scale in a 'cloud' environment. Basically, a platform as a service (Paas).
Pretend we are amateur PM's, programmers, and sysadmins tasked with this. What would you recommend? We know generally what is needed: some sort of routing, database, caching, authentication, etc. What other tools do we need?
We would prefer tools along a ruby/python/haskell/erlang dimension, on a linux/bsd stack, with postgres databases(couchdb or cassandra in the future). We are not touching anything in the ms/.net area, nothing on the JVM (We've looked at Steamcannon, but no; Scala and Clojure tools are not entirely out of the question). We have a basic grasp of bootstrapping a cloud (e.g. Eucalyptus) to build on. We have an understanding of the basics in server admin, and the physical infrastructure limitations aren't a factor right now.
We're not looking into why gaerokuyardspace is the best choice, a list of such services, why we should ditch our plans for one of these services, or an argument against this plan. For this situation the decision has been made that the cost to build privately is more attractive than the cost of deploying elsewhere. We already know why and how for these services. We're looking to emulate and build upon these for private needs.
A short list of tools to be expanded:
Beehive
Steamcannon
Gitosis/Gitolite
?
Basically, I'd like to generate a list of tools for building heroku/gae like service on a small, private, definitely experimental/toy level.
I don't know that it will meet all of your stated needs today, but you should take a look at Cloud Foundry from VMware. You can check the FAQ for the commercial project or look in to the Open Source version that you can host and manage yourself.
Some combination of Cloud Foundry (above) gitolite, and fabric
will probably do well for you. Any such solution will take some time to get right.
(Disclaimer: I'm a lead developer on the AppScale project)
AppScale is pretty much right up your alley, especially if you're looking to run Google App Engine apps in your own private cloud. It's open source, so grab it and extend it if there are other types of apps you want to support (and definitely commit it back to us if you do).
I'm looking to build my first web service and I would like to be pointed in the right direction right from the start.
Here are the interactions that must take place:
First, a smartphone or computer will send a chunk of data to my web service.
The web service will persist the information to the database.
Periodically, an algorithm will access and modify the database.
The algorithm will periodically bundle data and send it out to smartphones or computer (how?)
The big question is: What basic things do I need to learn to in order to implement something like this?
Now here are the little rambling questions that I've also got rolling around in my head. Feel free to answer these if you wish. (...maybe consider them extra credit?)
I've heard a lot of good things about RESTful services, I've read the wiki article, and I've even played around with the Twitter's webservice which is RESTful. Is this the obvious way to go? Or should I still consider something else?
What programming language do I use to persist things to the database? I'm thinking php will be the first choice for this.
What programming language do I use to interact with the database? I'm thinking anything is probably acceptable, right?
Do I have to worry about concurrent access to the database, or does MySQL handle that for me? (I'm fairly new to databases too.)
How on earth do I send information back? Obviously, if it's a reply to an HTTP request that's no problem, but there will be times when the reply may take quite a long time to compute. Should I just let the HTTP request remain outstanding until I have the answer?
There will be times when I need to send information to a smartphone regardless of whether or not information has been sent to me. How can I push information to my users?
Other information that may help you know where I'm coming from:
I am pretty familiar with Java, C#, C++, and Python. I have played around with PHP, Javascript, and Ruby.
I am relatively new to databasing, but I get the basic idea.
I've already set up my server and I'm using a basic LAMP stack. My understanding of L, A, M and P is fairly rudimentary.
Language: Python for it's ease of use assuming the GIL is not a particular concern for your requirements (e.g. multi-threading). It has drivers for most databases and supports numerous protocols. There are several web frameworks for it - the most popular probably being Django.
Protocols: if you are HTTP focused study SOAP and REST. Note, SOAP tends to be verbose, which causes problems moving volumes of data. On the other hand, if you are looking at other options study socket programming and perhaps some sort of binary format such as Google's protocol buffers. Flash is also a possibility (see: Flash Remoting). Note, binary options require users install something onto their machine (e.g. applet or standalone app).
Replies: if the process is long running, and the client should be notified when it's done, I would recommend developing an app for the client. Browser's can be programmed with JavaScript to periodically poll, or a Flash movie can be embedded to real time updates, but these are somewhat tricky bits of browser programming. If you're dealing with wireless phones, look at SMS. Otherwise I would just provide a way for clients to get status, but not send out notification (e.g. push vs. pull). As #jcomea_ictx wrote, AJAX is an option if it's a browser based solution - study jQuery.
Concurrency: understand what ACID means with regards to databases. Think about what should happen if you receive multiple writes to the same data - database may not necessarily solve this problem the way you'd want.
Please, for the love of programming, don't use PHP if you're already comfortable with Python. The latter makes for far cleaner, more maintainable code. Not that it's impossible to write good code in PHP, but it's a relative rarity. You can use Python for all the server-end stuff including MySQL interaction, with the MySQLdb module. Either with standard CGI, or FCGI, or mod_python.
As for the database, use of transactions will eliminate conflicts. But you can usually design a system in such a way that conflicts will not happen. For example, use of auto-incrementing primary-key IDs on each insert will make sure that every entry is unique.
You can "pull" data with Javascript, perhaps using AJAX methodology, or "push" using SMS or other technologies.
When replies take a while to compute, you can "poll" using AJAX. This is a very common technique. The server just returns "we are working on this" (or equivalent) with a built-in refresh until the results are ready.
I'm no expert on REST, but AJAX, especially when using polling rather than simply responding to user input, can be said to violate RESTful principles. But you can be a purist, or you can do whatever works. It's up to you.
I don't believe I've ever used any "push" technologies other than SMS, and that was years ago when many companies had free SMS gateways. So if you want to "push" data, better hope someone else joins in the conversation!
Use Java. The latest version of Java EE 6 makes coding RESTful and SOAP services a breeze, and it interoperates with databases very easily as well.
The benefits of using a true language instead of a script are: full server-side state, strong typing, multithreading, and countless other things that may or may not come in handy, but knowing they are available makes your project future proof.
Web services and web APIs have managed to increase the accessibility of the information stored and catalogued on the internet. They have also opened up a vast array of enterprise power functionality for smaller thin client applications.
By taping into these services developers can provide functionality that would have taken them months perhaps years to set up. They can combine them into single applications that make life generally easier for its users.
Whether displaying information about the music being played, finding items of interest in the locale of the user or just simply tweeting and blogging from the same application - the possibilities are growing everyday.
I want to know about the most interesting or useful services that are out there, especially ones that most of us may not have heard about yet. Do you maintain an API or service? or do you have a clever mash up that provides even more benefits than the originals?
YQL - Yahoo provide a tool that lets you query many different API's across the web, even for sites that don't provide an API as such.
From the site:
The Yahoo! Query Language is an
expressive SQL-like language that lets
you query, filter, and join data
across Web services.
...
With YQL, developers can access and
shape data across the Internet through
one simple language, eliminating the
need to learn how to call different
APIs.
The World Bank API is pretty cool. Google uses it in search results. My favourite implementations are the cartograms at worldmapper.
(source: worldmapper.org)
It's very niche, but I happen to think the OpenCongress API is amazing.
Less niche: Google Translate has an API which will guess the language of something. You'd be AMAZED how frequently this comes in handy (even though it's not as tweakable as you'd like and is not trained on small samples).
I was just about to have a stab at using the SoundCloud API
I know many people who already use for sharing their musical masterpieces and its a pretty good site. Hopefully the api will be as well!
I like the RESTful API for weather.com. It's free and very useful for the new age of location-aware apps: https://registration.weather.com/ursa/xmloap/step1
It does require registration, but they don't spam you or anything - it's just to provide you a key to use the API.
Ah yes - here's another one I've been meaning to check out but haven't tried yet
The BBC offer a bunch of apis/feeds that look very promising
http://ideas.welcomebackstage.com/data
They include apis for accessing schedule data for both TV and Radio listings along with all kinds of news searches. It even looks like they'll be offering some sort of geo-location service soon so it will be interesting to see what that has to offer
Another interesting one for liberal brits! ;)
The Guardian news paper have their own api
http://www.guardian.co.uk/open-platform
MuiscBrainz
Excellent service for music mashups.
Not so many knows that Last.FM initial database was scraped from this service.
The United States Postal Service offers a web service that does address standardization. Quite useful in reducing clutter and cleaning data before it gets put into your database.
Before you start flaming, I'm going to tell you that I am trying to convince myself that this is a bad idea.
Basically, I'm trying to create a website with some basic accounting functions. My friend, a consultant who only knows excel, asked if this could be an excel spreadsheet instead of a web interface.
I found myself thinking, why is excel not the better tool in this case? It has all the tools a web2.0 app has (scripting, access to a db, basic formatting), and is made for accounting. You could basically use excel to write a program that fetches data from an SQL database, populates some cells, and use formulas for the rest.
Is there precedent to using excel as one would use the browser to make an ajax-y web2 app?
Why is this a bad idea?
Update: just to clarify I meant this to be more a "rhetorical" discussion. I'm not sure why the rest of the message didn't clarify that...
Well, if you made it in excel, then it wouldn't really be a web application. It would be an Excel application.
Not that that's a bad thing though. If it makes more sense to build the app in Excel, then go for it.
The problems you'll have doing it in Excel will be the same as the problems you run into with any desktop vs. web application. For example, How do you handle deploying new versions of the app? How do you handle updating the app for new (or older) versions of Excel? etc.
ETA:
If you want to avoid the bugginess of Excel VBA and the headaches of managing a desktop application, you might want to look into the Google Spreadsheets API. You can use it to create/update Google docs spreadsheets on the fly, including formulas and lots of other spreadsheet goodness. Using a hybrid of web application and Google Spreadsheets might give you the best of both worlds, depending on what exactly you need to do.
After years of programming in Excel-VBA, the best answer that I can give you for not doing this:
Excel-VBA is buggy! It is probably the most bug-filled app that Microsoft has produced. It's great for some tasks, but forcing it to do a job for which it was never intended will lead to trouble.
I have a few spreadsheets that do similar things, (things Bill never intended), and without exception, they teeter on the brink of failure, and tend to crash with only the slightest provocation.
Sure, you can do it, but the headache is not worth it.
Excel was not meant to be used that way, so it will be painful.
Two better ideas for web-spreadsheet integration:
Take a look at Resolver One, a programmable Excel-compatible spreadsheet with an integrated web server.
Use Google Docs spreadsheets. There is an API to interact programmatically with them.
It's not a bad idea, but it does come with some limitations. If deployment isn't an issue for you, and you don't need "universal" access to the application, your solution will likely save you some time. A web application would certainly be a more elegant approach, particularly if you want to make the software available publicly.
The web is inherently based on HTML and added to with Databases, CSS and server-side languages and javascript (possibly others). As with most things, you should use the right tool for the job. If you want a website then you should use the tools for creating a website.
Excel is not meant to make websites, forcing it to do so will likely lead to more frustration than happiness.
Well, you're going to run it on the server side, you'd probably run into licencing and performance issues.
If you're delivering .xls to the client, then you'd need for all your clients to have Excel or something compatible. But you also lose control over your "site" - what's the point of visiting if you have everything you need locally?
So, the SQL statements would be on the client? That's rarely a good idea.
It's a bad idea for the same reason that writing a graphics editor with MS Access is bad, or coding a MMORPG using Powerpoint is bad :)
I would also say that once you stop using a browser as the client it ceases to be a "web application" - you are really just talking about an Excel sheet which fetches its data via HTTP.
+1 to everyone who said "an excel app is not a web app."
BUT... If you want to use the spreadsheet metaphor for server-side calculations for a web app, or if you want to access the library of financial functions that come with Excel from server-side code, you can use Excel Services. It's exposed via SOAP, interoperable, callable from any SOAP-capable platform.
It's server-capable. It does not actually load Excel on the server, but a non-GUI runtime of the functions.
I think that using Excel as Browser is bad idea, however i think better idea is to use Excel Control in Desktop App.
This way you can control the sql and connections. you can save data as often as you like. you could also implement some update mechanism. App would be more secure and harder to hack.
I think Excel freezes when you try to connect to some outside resource and this way you would control everything.
Did you start by getting user requirements for this application? It doesn't sound like they wanted a website to begin with. Sounds like they wanted an excel spreadsheet with macros.
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 4 years ago.
Improve this question
I'm building an app that authors would (hopefully) use to help them, uh.. author things.
Think of it like a wiki but just for one person, but cooler. I wish to make it as accessible as possible to my (potential) adoring masses, and so I'm thinking about making it a web-app.
It certainly doesn't have to be, there is no integration with other sites, no social features. It involve typing information into forms however, so for rapid construction the web would probably be the best.
However, I don't really want to host it myself. I couldn't afford it for one, but it's mostly that people who use this may not want their data stored elsewhere. This is private information about what they are writing and I wouldn't expect them to trust me with it, and so I'm thinking about making it a thick-client app.
And therein lies the problem, how to make a application that focuses mainly on form data entry available easily to potential users (yay web apps) but also offline so they know they are in full control of their data (yay thick-client apps).
I see the following solutions:
Build it as a thick-client Java app and run a cutdown version on the net as an applet that people can play with before downloading the full thing.
Build it as a Flex app for online and an Air app for offline (same source different build scripts basically).
Build it as a standard web-app (HTML, JS etc) but have a downloadable version that somehow runs the site totally on their computer. It wouldn't touch the net at all.
Ignoring 1 and 2 (I'm looking into them separately), I think 3 would involve:
Packaging up an install that contains a tiny webserver that has my code on it, ready to run.
Remapping the DB from something like mySQL to something like SQLite.
Creating some kind of convience app that ran the server and opened your browser to the right location, possibly using something like Prism to hide the whole broswer thing.
So, have you ever done something like this before?
If so, what problems did you encounter?
Finally, is there another solution I haven't thought of?'
(also, Joyent Slingshot was a suggestion on another question, but it's RoR (which I have no experience in) and I'm 99% sure it doesn't run under linux, so It's not right for me.)
I think you should look at tiddlywiki for inspiration.
It's a wiki written in JavaScript entirely self-contained in a single html file. You load it into your browser as a file:/// URL, so there is no need for a server.
I use it as a personal wiki to keep notes on various subjects.
Google Gears is used to offer a few of the google apps offline (Google Reader, Gmail, Docs and more).
What is Google Gears?
Gears is an open source browser extension that lets developers create
web applications that can run offline.
Gears provides three key features:
A local server, to cache and serve application resources (HTML,
JavaScript, images, etc.) without
needing to contact a server
A database, to store and access data from within the browser
A worker thread pool, to make web applications more
responsive
by performing expensive operations in
the background
Gears is currently an early-access developers' release. It is not yet intended for use by real users in production applications at this time.
If you're a developer interested in using Gears with your application, visit the Gears Developer Page.
If you wish to install Gears on your computer, visit the Gears Home Page. Please note, however, that Gears is not yet intended for general use.
But as you read it's still in early stages.
There is an additional option, and that is to use the new HTML5 offline application features, namely the Application Cache, Client-Side Databases, and Local Storage APIs.
Currently I believe that Safari is the only shipping browser to support any of these, and i believe it only supports the client side databases and local storage parts. The webkit nightlies support all of these features, the firefox nightlies support many of them (maybe all now?)
[Edit (olliej): Correction, Firefox 3 supports the Application cache, but alas not the client side DB]
We are using something similar to your third option to test our websites locally. Works just fine.
Our packaged webserver is not small enough to accomplish what you need, but then again we've not been trying to keep it small either. If you can package your webserver code into a small enough package I don't see why this approach would'nt work.
I think AIR is the way to go..
Have you checked into google gears?
Some pointers for solution 3:
for the GUI part, ExtJS seems really nice.
for the storage part, there is a nice javascript library that abstracts different storage backends: PersistJS.
Supported backends for PersistJS:
flash: Flash 8 persistent storage.
gears: Google Gears-based persistent storage.
localstorage: HTML5 draft storage.
whatwg_db: HTML5 draft database storage.
globalstorage: HTML5 draft storage (old spec).
ie: Internet Explorer userdata behaviors.
cookie: Cookie-based persistent storage.
Also, I think the moin moin wiki software has a desktop version that includes its own webserver. This stuff is easy in python, since batteries are included.
You might want to check out how they do it?
You could make a dedicated client using Webkit or Firefox's backbone. Some games use that solution for UI for example.
Or you could make a little webserver (I have a little webserver in Lua that I use for similar purposes, just a few megas with libaries and all). However if you take this route the biggest issue to consider is you don't want your webserver to depend on environmental variables, you want it to be totally autonomous. You should try to isolate all variables t o a config file and be done with it (bundle style)
Or you could use a Java client application to display the webpage
Or GoogleGears, but that's the same (almost) as Flex+Air. so choose Flex+Air if that's what you are familiar with
You didn't specify a language but I looked at Karigell a few years ago. It's Python web framework, similar to Django or TurboGears, but it doesn't have the overhead of those frameworks.
From my messing around with it, it seems like it would work for your purposes. It has a built-in web server (though you can use pretty much any server you want) and you can use any database that Python supports.
Plus, Python works well with Linux. :)
If you made the app a regular web app heavily reliant on client-side technologies (using DHTML and the likes of Google Gears to store data offline as already suggested) so once opened, there wasn't much interaction with the server, you could probably host the thing on a basic shared hosting account which wouldn't cost that much. That might be your easiest starting point as you wouldn't have to worry about all the issues with desktop apps such as compatibility with different operating systems, packaging up an install etc, yet you wouldn't need massive server resources behind it either.
You can use HTML, JS and whatever else in Adobe AIR and you'll have plenty of options of saving data locally, too.
in java world you could use jetty for a server, implement web app using your favorite framework and use hsqldb as a database - it lives entirely in your container (jetty). you can deploy preview app on the web and package downloadable offline version.
There's a portable distribution of Apache/MySQL/PHP (to place on USB keys):
http://portableapps.com/apps/development/xampp
This should be easily adapted to your needs.
You could also consider using XULRunner or Prism
They're the opensource technology that FireFox, Thunderbird and Joost are built on, and allows you to develop apps in XML and javascript essentially against the same rich api that FireFox itself has. And of course this is cross platform too, so it'd work on Mac/Linux/Windows...
Check here for more info:
https://developer.mozilla.org/en/XULRunner
I was thinking of doing something like this myself. My plan was to write app using django and write script that starts django's testing server and opens default browser on specified port. My plan was to use SQLite...
Also, it would be nice to pack it into one package, so users without django installed can run app without any dependecies...
My suggestion, as you pointed above, is to use a Wiki system to solve your problem. Now the question could be: Wich one?
You can use Trac, it is very simple and you can customize its GUI. But, if you prefer something more advanced please use MoinMoin. I used it for years, and IMO it is a very good and strong wiki system.
Depiste wich wiki you will choose, forget to write your web-app from scratch. According to yor question the best approach is to pick something that works and customize/modify it to fit your needs.