I'm the only programmer of a pretty small ISP in a rural area with just around 2000 customers. Now I have finished a couple of semesters in university but I only have a couple of years of experience in the field so I'm uncertain of the architectural decisions that I'm making and was hoping somebody could help me pick the right path.
Most of our internal apps were created 8-10 years ago and are severely outdated and I have been given the job to replace those systems. Most of the basic underlying systems are solid but the apps that we use to manage our customers and connecting those to our internal systems are...lacking to say the least.
Most of these applications were created in PHP back in the day and are using mysql databases. I decided that i was going to create a couple of rest APIs using NodeJS on top of these databases and then create a central app that will take care of connecting all those systems together and making sure they stay up to date with one another.
Now for the question. I've been looking a bit into enterprise architecture and from what I've gathered going with this sort of micro service architecture seems to be a solid plan. However I've also seen a couple articles talking about message buses and my question is if i should instead set up a message bus, for example apache activemq so these services can talk together amongst themselves instead of using a central app that would handle managing all of them.
Are there any specific patterns that i should be reading up on or does what I've come up with look solid enough?
An enterprise service bus will add a lot of complexity to your design, so you need to look at the pro/con to see if it's really necessary. Here is an article you can always upgrade your architecture in the future and migrate the services.
I run some complex services on Apache Tomcat and they work great. Supports a user pool of 70,000. If you build in connection pooling and redundancy you should be fine.
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 7 years ago.
Improve this question
I am thinking of using either one of them in building our startup which is like a job portal with validation,verification and includes special features for freelancing and all.
Is meteor or sails good for using as a backend or should we go with more robust backend like Django? Will using javascript on our backend provide the ability to scale in the future moreso than Django/python?
I would really like some opinions in this matter to get to a decision.
sails.js and meteor are both great options for production.
Both frameworks have good real-time (socket.io) support, large/active communities, support a stateless backend design which make horizontal scalability possible, and are great for getting a web application spun up quickly.
sails.js - http://sailsjs.org
broad database support through the Waterline ORM (there are over a dozen supported databases)
concepts should more familiar to most node.js developers (it's built on express)
modeled after rails, grails, and django, so the paradigm is more familiar to developers with experience in those tools
extensible through npm package manager via express middleware and custom modules
meteor - https://www.meteor.com
better integration between the backend and frontend
project is VC-backed with a firmer corporate backing
extensible using custom package manager and extension system
built-in deployment system and hosting on meteor.com
Will using javascript on our backend provide the ability to scale in the future moreso than Django/python?
Probably. As with anything, you just have to do it right.
My overall opinion is that meteor is sort of cult-ish and monolithic, and that once you've chosen it, you're locked in. sails.js is built on express, so it's easy to split out functionality and integrate with other tools.
My disclaimer is that I work for Balderdash (the company that invented sails.js); but on that note, I can also tell you that millions of users are served by sails.js applications. We find that it's quite good, and our business is thriving because of the power of sails.js. I know folks who have used meteor with success as well.
I think this is a primarily opinion-based question, so you're going to receive answers from the very same "type".
But I can tell you one thing: Meteor is robust enough for production use, specially now that they hit the 1.1 release (https://www.meteor.com/blog/2015/03/31/meteor-11-microsoft-windows-mongodb-30).
Meteor is perfectly suited for startups, since it brings everything you need (and more) into a single "pack".
Check this: http://meteorpedia.com/read/Why_Meteor
So yah, that's my answer going for the Meteor side.. (not very technical, I know).
I'm currently working with a team of developers on a company project to create a centralized repository of product and pricing information. This will be built for both internal company use and external client use. On top of the basic features of storing product and pricing information we also need to build up an infrastructure to accommodate:
REST API endpoints
Dev/Staging/Deployment workflows (particularly for performing updates on records in a live environment)
Logging
Analytics
Reporting
Security (authentication and authorization).
Going over the list, it reads like a very common set of requirements for a web application and I doubt my company is breaking any sort of new ground. SO, is there any particular resources (frameworks, technology stacks, articles, books) that can help me understand how other web applications are tackling these problems?
A bit of background on the team. The team has worked on a handful of small-to-medium sized web applications using PHP, Mongo and MySQL for the backend, and basic HTML, CSS, JQuery on the frontend. The team is familiar with design patterns (i.e.Gang of Four) but to date have not worked on anything requiring all of the elements listed above
It's probably worth playing around with a solid Web development framework like Zend, Yii or even Ruby on Rails or Django, which are not PHP frameworks, but are fairly mature and well structured. Even if you do not plan to use that framework for development you'll get some great ideas for how to structure your web applications, how to implement logging and common web security features.
As far as deployment and workflows go, you may want to give Extreme Programming a read if you haven't already. It describes what many developers today considered to be a fairly classic agile project management methodology, but it also gets into important practices such as testing and continuous integration, which in my opinion are incredibly important components of the development workflow. If you're starting from scratch as a team you'll benefit enormously from implementing a solid agile methodology -- or at the very least from a solid foundation in testing and continuous integration.
For examples of REST style applications you may want to see how popular open source implementations work. Some of these frameworks will have a REST structure built in, but there are many open source options, some of which are discussed here.
For analytics, Google has quite a bit of documentation here.
As far as reporting goes, I'm not clear on what you need, but if you're talking about log parsers and bug or downtime reporters there are some excellent tools out there, including continuous integration automation tools such as Atlassian's Bamboo that will provide some reporting assistance. These can help you with part of the reporting process, but from my experience a large, complex web application can benefit from custom reporting elements, considered as part of the development process from the beginning. It's not that difficult to parse logs programmatically, I don't think there's a one size fits all implementation.
As a side note, Atlassian has some excellent development tools if you're willing to pay for them, but open source alternatives shouldn't be difficult to find, such as the ubiquitous Trac for ticket tracking, and basic project management with an integrated wiki.
I can't say I know of a single, comprehensive location that provides you all the information you need (at least not yet!), but hopefully you'll glean something interesting from this answer. Starting on some serious web development projects with a fresh team (if I interpreted your situation correctly) can be a really enjoyable challenge. Good luck!
Closed. This question is off-topic. It is not currently accepting answers.
Want to improve this question? Update the question so it's on-topic for Stack Overflow.
Closed 13 years ago.
Improve this question
I'm trying to build a web application that is similar to Youtube (it's not a knock off), but I guess I don't know how video is served on the internet very well.
I know how to build regular database driven web applications, but nothing like the scalability of Youtube. All of the applications I have built before have all been run on one server with the files stored on the same box as the web server.
How does one decouple the application server from the file storage from the media server?
I would more or less want 4 machines (clusters of machines)
1.) Application servers
-- Present the web page, handle user uploads, link the user's flash player to the correct media server etc.
2.) Database shards
-- Store user information, check favorites, etc.
3.) File storage
-- Store the media files
4.) Media servers
-- Serve the media files
How do I hook all of this together? Which technologies should I leverage? Where do I go to learn more about architecting this?
How does Youtube's embeddable flash stuff work? I want to embed my flash player on other websites and have it tie into my architecture.
Note I have looked into: http://highscalability.com/youtube-architecture
But I still don't get the overall picture of how this stuff ties together.
If someone can explain in high level terms how all of this stuff works?
Are there dedicated client servers running internally to shuffle around all of this stuff between the application servers, file storage, etc. Is it all via HTTP using JSON, what is going on here!
Thanks
Two books I'd recommend are:
Scalable Internet Architectures
Building Scalable Web Sites
The latter is by the director of engineering at flickr. Not youtube, but I think you'll find it enlightening.
Beyond that, the High Scalability blog is a good source of case studies and collected wisdom, all of which provide a good starting point for further exploration.
Start by hiring the right people; if you hire smart people, they'll be able to come up with answers to these questions, and more which will crop up.
Also, start at the scale that you plan to initially operate at. Don't plan for scalability you don't need. You aren't going to be making another Youtube - even if you're very successful within your field.
Scalability is expensive - very expensive - to develop and maintain. If you don't need it, it will drain your resources and restrict your developers needlessly. Just building a credible test environment for high performance systems tends to be a big job, and such a system would require several such environments.
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 4 years ago.
Improve this question
I'm building an app that authors would (hopefully) use to help them, uh.. author things.
Think of it like a wiki but just for one person, but cooler. I wish to make it as accessible as possible to my (potential) adoring masses, and so I'm thinking about making it a web-app.
It certainly doesn't have to be, there is no integration with other sites, no social features. It involve typing information into forms however, so for rapid construction the web would probably be the best.
However, I don't really want to host it myself. I couldn't afford it for one, but it's mostly that people who use this may not want their data stored elsewhere. This is private information about what they are writing and I wouldn't expect them to trust me with it, and so I'm thinking about making it a thick-client app.
And therein lies the problem, how to make a application that focuses mainly on form data entry available easily to potential users (yay web apps) but also offline so they know they are in full control of their data (yay thick-client apps).
I see the following solutions:
Build it as a thick-client Java app and run a cutdown version on the net as an applet that people can play with before downloading the full thing.
Build it as a Flex app for online and an Air app for offline (same source different build scripts basically).
Build it as a standard web-app (HTML, JS etc) but have a downloadable version that somehow runs the site totally on their computer. It wouldn't touch the net at all.
Ignoring 1 and 2 (I'm looking into them separately), I think 3 would involve:
Packaging up an install that contains a tiny webserver that has my code on it, ready to run.
Remapping the DB from something like mySQL to something like SQLite.
Creating some kind of convience app that ran the server and opened your browser to the right location, possibly using something like Prism to hide the whole broswer thing.
So, have you ever done something like this before?
If so, what problems did you encounter?
Finally, is there another solution I haven't thought of?'
(also, Joyent Slingshot was a suggestion on another question, but it's RoR (which I have no experience in) and I'm 99% sure it doesn't run under linux, so It's not right for me.)
I think you should look at tiddlywiki for inspiration.
It's a wiki written in JavaScript entirely self-contained in a single html file. You load it into your browser as a file:/// URL, so there is no need for a server.
I use it as a personal wiki to keep notes on various subjects.
Google Gears is used to offer a few of the google apps offline (Google Reader, Gmail, Docs and more).
What is Google Gears?
Gears is an open source browser extension that lets developers create
web applications that can run offline.
Gears provides three key features:
A local server, to cache and serve application resources (HTML,
JavaScript, images, etc.) without
needing to contact a server
A database, to store and access data from within the browser
A worker thread pool, to make web applications more
responsive
by performing expensive operations in
the background
Gears is currently an early-access developers' release. It is not yet intended for use by real users in production applications at this time.
If you're a developer interested in using Gears with your application, visit the Gears Developer Page.
If you wish to install Gears on your computer, visit the Gears Home Page. Please note, however, that Gears is not yet intended for general use.
But as you read it's still in early stages.
There is an additional option, and that is to use the new HTML5 offline application features, namely the Application Cache, Client-Side Databases, and Local Storage APIs.
Currently I believe that Safari is the only shipping browser to support any of these, and i believe it only supports the client side databases and local storage parts. The webkit nightlies support all of these features, the firefox nightlies support many of them (maybe all now?)
[Edit (olliej): Correction, Firefox 3 supports the Application cache, but alas not the client side DB]
We are using something similar to your third option to test our websites locally. Works just fine.
Our packaged webserver is not small enough to accomplish what you need, but then again we've not been trying to keep it small either. If you can package your webserver code into a small enough package I don't see why this approach would'nt work.
I think AIR is the way to go..
Have you checked into google gears?
Some pointers for solution 3:
for the GUI part, ExtJS seems really nice.
for the storage part, there is a nice javascript library that abstracts different storage backends: PersistJS.
Supported backends for PersistJS:
flash: Flash 8 persistent storage.
gears: Google Gears-based persistent storage.
localstorage: HTML5 draft storage.
whatwg_db: HTML5 draft database storage.
globalstorage: HTML5 draft storage (old spec).
ie: Internet Explorer userdata behaviors.
cookie: Cookie-based persistent storage.
Also, I think the moin moin wiki software has a desktop version that includes its own webserver. This stuff is easy in python, since batteries are included.
You might want to check out how they do it?
You could make a dedicated client using Webkit or Firefox's backbone. Some games use that solution for UI for example.
Or you could make a little webserver (I have a little webserver in Lua that I use for similar purposes, just a few megas with libaries and all). However if you take this route the biggest issue to consider is you don't want your webserver to depend on environmental variables, you want it to be totally autonomous. You should try to isolate all variables t o a config file and be done with it (bundle style)
Or you could use a Java client application to display the webpage
Or GoogleGears, but that's the same (almost) as Flex+Air. so choose Flex+Air if that's what you are familiar with
You didn't specify a language but I looked at Karigell a few years ago. It's Python web framework, similar to Django or TurboGears, but it doesn't have the overhead of those frameworks.
From my messing around with it, it seems like it would work for your purposes. It has a built-in web server (though you can use pretty much any server you want) and you can use any database that Python supports.
Plus, Python works well with Linux. :)
If you made the app a regular web app heavily reliant on client-side technologies (using DHTML and the likes of Google Gears to store data offline as already suggested) so once opened, there wasn't much interaction with the server, you could probably host the thing on a basic shared hosting account which wouldn't cost that much. That might be your easiest starting point as you wouldn't have to worry about all the issues with desktop apps such as compatibility with different operating systems, packaging up an install etc, yet you wouldn't need massive server resources behind it either.
You can use HTML, JS and whatever else in Adobe AIR and you'll have plenty of options of saving data locally, too.
in java world you could use jetty for a server, implement web app using your favorite framework and use hsqldb as a database - it lives entirely in your container (jetty). you can deploy preview app on the web and package downloadable offline version.
There's a portable distribution of Apache/MySQL/PHP (to place on USB keys):
http://portableapps.com/apps/development/xampp
This should be easily adapted to your needs.
You could also consider using XULRunner or Prism
They're the opensource technology that FireFox, Thunderbird and Joost are built on, and allows you to develop apps in XML and javascript essentially against the same rich api that FireFox itself has. And of course this is cross platform too, so it'd work on Mac/Linux/Windows...
Check here for more info:
https://developer.mozilla.org/en/XULRunner
I was thinking of doing something like this myself. My plan was to write app using django and write script that starts django's testing server and opens default browser on specified port. My plan was to use SQLite...
Also, it would be nice to pack it into one package, so users without django installed can run app without any dependecies...
My suggestion, as you pointed above, is to use a Wiki system to solve your problem. Now the question could be: Wich one?
You can use Trac, it is very simple and you can customize its GUI. But, if you prefer something more advanced please use MoinMoin. I used it for years, and IMO it is a very good and strong wiki system.
Depiste wich wiki you will choose, forget to write your web-app from scratch. According to yor question the best approach is to pick something that works and customize/modify it to fit your needs.