Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 1 year ago.
The community reviewed whether to reopen this question 1 year ago and left it closed:
Original close reason(s) were not resolved
Improve this question
What do we actually mean by large scale web application ?
What is the criteria to call it large scale web application
Is it number of code lines of a application or
Number of user per day of a web application. say 10K per day ?
What do we actually mean by large scale web applications? That depends who you ask.
If people have build smaller applications in the past and need to build a larger one now that handles more data and more traffic, they might call it a large scale web application. But if you then compare that site with ones like LinkedIn, Facebook, or Google then it will still be a small application.
People have different notions of what large is, what's large for some might be medium size for others and small for some others. But a large scale web application has characteristics such as these:
performance, being able to handle a large number (millions) of users/request or a large number (thousands) of transactions per second. Both have challenges depending on the type of app (CPU bound, IO bound or both).
scalability, the horizontal type, not only at the web server level but also at the database level. Depending on what you are doing a RDBMS might not cut it anymore so you have to walk the NoSQL path, sometimes using more than one product as NoSQL solutions tend to be specialized systems tackling a specific use cases as opposed to being a general purpose database as RDBMSs are. A lot of integration challenges arise from connecting heterogeneous solutions together and make them behave as a single application.
large scale applications are distributed, taking advantage of CDNs or running the app on servers geographically closer to the user. You can easily have hundreds or thousands of server nodes, with a large sys admin team having to manage the setup. If you don't have your own data centers you can run in the cloud.
besides a large sys admin team, you often have a large development team needing to optimize for performance and scalability, designers and front end developers working on providing a fluid user interface, with mobile support, etc;
having to deal with a large volume of data and with lots of data types. No longer handling just products, customers, orders, etc, but also clicks, page views/hits, events, logs, customer behavior tracking, etc. This is in relation to the previous NoSQL point, but also with this volume of data these apps tend to have a large back-office that offers all kinds of reports, graphs, administrative tools, etc to manage the app itself.
availability 24/7;
some other, miscellaneous keywords to throw into the mix like SOA architecture, microservices, data-warehouses, security, distributed caches, continuous deployment, etc.
etc.
These are some of the characteristics (I think) large web scale web apps have in common and it's important to think about these aspects up front and deal with all challenges that come from them from the beginning, when building the app.
Related
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 5 years ago.
Improve this question
I am looking into purchasing server space with AWS to host what will eventually be over 50 websites. They will have many different ranges of traffic coming in. I would like to know if anyone has a recommendation on what size of server that would be able to handle this many sites.
Also, I was wondering if it's more cost effective/efficient to host an separate EC-2 instance for each site or too purchase a large umbrella server and host all sites on a single instance?
Thanks,
Co-locating services on single/multiple servers is a core architecting decision your firm should make. It will directly impact the performance, security and cost of your systems.
The benefit of having multiple services on the same Amazon EC2 instance is that they can share resources (RAM, CPU) so if one application is busy, it has access to more total resources. This is in contrast to running each one on a separate instance, where there is a smaller, finite quantity of resources. Think of it like car-pooling vs riding motorbikes.
Sharing resources means you can probably lower costs, since you'll need less total capacity.
From a security perspective, running on separate instances is much better because they are isolated from each other. You should also investigate network isolation to prevent potential breaches between instances on the same virtual network.
You should also look at the ability to host all of these services using a multi-tenant system as opposed to 50 completely separate systems. This has further benefits in terms of sharing resources and reducing costs. For example, Salesforce.com doesn't run a separate computer for each customer -- all the customers use the same systems, but security and data is kept separate at the application layer.
Bottom line: There are some major architectural decisions to make if you wish to roll-out secure, performant systems.
The short correct answer:
If those sites are only static(html, css and js). EC2 won't be necessary because you can use S3 and it will be more cheap and you won't have to worry about scaling.
But if those sites have a dynamic part like php, python and similar. Well it is a different story.
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 6 years ago.
Improve this question
Should I use google charts in production environment?
Google charts are very easy to use.
https://google-developers.appspot.com/chart/interactive/docs/quick_start
But is it recommended to be used in a production environment?
The API's are not hosted in house but called form google servers.
There is a risk of google changing them or discontinuing them.
I couldn't find any license agreement to use.
Is the data secure as the data is being sent to google servers.
Are the above real risks or am I over thinking.
I was wondering if anyone has any experience with using google API's in production. Or if anyone can give some recommendations.
The Terms of Service cover some of your questions. Basically, Google's deprecation policy says that the API will be available for 3 years following deprecation (and most of the API - namely, the Interactive Charts API - is not deprecated; the old Image Chart API is, however).
For data security, most charts in the Interactive Charts API do not send any data to Google's servers, though there are exceptions. Each chart's documentation has a Data Policy section which explains what, if any, data is sent to Google (examples: AreaCharts, which do not send any data; and GeoCharts, which may send data if you use the geocoding features). Charts in the Image Chart API do send data to Google's servers, as they generate the images server-side rather than client-side, but this API is deprecated anyway, so you probably shouldn't be using it.
The main risk with using the Visualization API in my experience is that you have (practically) no control over versioning. When the development team releases an update, everyone everywhere gets the update. Usually this is a good thing, as it brings new features, bug fixes, and performance enhancements to everyone. Occasionally, however, a new release may introduce a bug, or change the behavior or appearance of a chart in some way that is undesirable for your application. When this happens, you generally cannot roll back to the previous version. For projects that are under active development for long periods of time, this is generally an acceptable trade-off for the free (as in beer) chart API. For projects that do not have a long-term maintenance budget, this can be problematic.
If your user-base is in an area that has poor connectivity to Google's servers, having the API hosted remotely could be problematic, but in general this is not the case.
I have used it in a production environment. All the questions you have posed are very real possibilities. For use it came down to budget, the money was there to purchase a system so we went with what we could afford at the time. The direction you go really depends on budget and existing systems that might be able to achieve the same thing.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 7 years ago.
Improve this question
I would like to know the differences between Mashery, WSO2 and 3scale. Someone who has used API Managers before can give his opinion? What are advantages and disadvantages of each one
thanks
cheers
Not sure, but this question might end up flagged as off topic - vendor comparison, but anyway I'll jump in. I work at 3scale (full disclosure) but hopefully this is useful anyway - the three are pretty different. Trying to be as neutral as possible!:
3scale uses NGNIX and/or open source code plugins to enforce all of the API traffic rules and limits (rate limits, key security, oauth, analytics, switching apps on and off etc.) and the traffic always flows directly to your servers (not via the cloud) so you don't have additional latency or privacy concerns. Because it's NGNIX it's also widely supported, very fast and flexible. Then it has a SAAS backend that manages all the analytics, rate limits, policies, developer portal, alerts etc. + synchronizes across all the traffic manager nodes. It's free to use up to nearly 5million API calls per month.
WSO2's system is an additional module to the WSO2 ESB so if you're using that it makes a lot of sense. It runs everything locally with no cloud components - a pro or a con depending on how you see it. It's also been around a lot less time and doesn't have such a large userbase.
Mashery has two systems - the main one with which the API traffic flows through Mashery's cloud systems first and has traffic management applied there. So there is always a latency heavy roundtrip between the users of the API and your servers + it means Mashery is in your API traffic critical path. They also have an on premise traffic manager but it's much less widely used. Both solutions have very significant costs and long term commitments.
As 3scale what we see as the main advantage is you have a tons of control as to how you set up all the traffic flow and never have to route through a third party plus you have the benefit if having all the heavy lifting hosted and synchronized across multiple data centers. We're also committed to having a strong free for ever tier of service since we want to see a lot of APIs out there! http://www.3scale.net/
Good luck with your choice!
steve.
Closed. This question is off-topic. It is not currently accepting answers.
Want to improve this question? Update the question so it's on-topic for Stack Overflow.
Closed 13 years ago.
Improve this question
I'm trying to build a web application that is similar to Youtube (it's not a knock off), but I guess I don't know how video is served on the internet very well.
I know how to build regular database driven web applications, but nothing like the scalability of Youtube. All of the applications I have built before have all been run on one server with the files stored on the same box as the web server.
How does one decouple the application server from the file storage from the media server?
I would more or less want 4 machines (clusters of machines)
1.) Application servers
-- Present the web page, handle user uploads, link the user's flash player to the correct media server etc.
2.) Database shards
-- Store user information, check favorites, etc.
3.) File storage
-- Store the media files
4.) Media servers
-- Serve the media files
How do I hook all of this together? Which technologies should I leverage? Where do I go to learn more about architecting this?
How does Youtube's embeddable flash stuff work? I want to embed my flash player on other websites and have it tie into my architecture.
Note I have looked into: http://highscalability.com/youtube-architecture
But I still don't get the overall picture of how this stuff ties together.
If someone can explain in high level terms how all of this stuff works?
Are there dedicated client servers running internally to shuffle around all of this stuff between the application servers, file storage, etc. Is it all via HTTP using JSON, what is going on here!
Thanks
Two books I'd recommend are:
Scalable Internet Architectures
Building Scalable Web Sites
The latter is by the director of engineering at flickr. Not youtube, but I think you'll find it enlightening.
Beyond that, the High Scalability blog is a good source of case studies and collected wisdom, all of which provide a good starting point for further exploration.
Start by hiring the right people; if you hire smart people, they'll be able to come up with answers to these questions, and more which will crop up.
Also, start at the scale that you plan to initially operate at. Don't plan for scalability you don't need. You aren't going to be making another Youtube - even if you're very successful within your field.
Scalability is expensive - very expensive - to develop and maintain. If you don't need it, it will drain your resources and restrict your developers needlessly. Just building a credible test environment for high performance systems tends to be a big job, and such a system would require several such environments.
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 7 years ago.
Improve this question
I am a programmer who writes a lot of code for desktop applications, now started considering cross-platform apps as an issue but at work I write C# apps and I come from C++ and CS background and of course, I wrote several things in QT/C++. But now I am kinda confused about web applications, I have done some work on PHP and I know how things go there, I was a gmail and google docs user for a lot of time and I have seen how much web applications were improved with new web 2.0 technology including Ajax, XML so on. And my confusion is that should I start looking forward for web application development? and continue exploring the power of web 2.0 or I have to just stick with my old world where I feel very comfortable on parallelism and other stuff? Because believe me I had too many offers to work as a web application developer but I didn't realize this opportunity and now I am kinda confused whether I must start writing web apps. Have you been writing desktop applications and switched to web? or have somebody experience in this scenario?
Thank you.
The boundaries between desktop and web applications have really blurred. Whilst once upon a time the nature of developing for the web was totally different to developing for the desktop, nowadays you find the same concepts (such as parallelism which you referred to) cropping up in both. Don't think of developing web applications as taking a huge step away from traditional software development as you'll employ just as many skills and concepts as you already use. You wouldn't need to learn a whole lot more to get involved in web development if you have C# experience, as you could code backends to web applications in a very similar way to how you currently work. If you wanted/needed to get involved in the UI side of things, there are new technologies you'd need to pick up, but they're not essential to get a job in web development (as long as you weren't looking for a frontend role obviously).
To follow up Dustman's comments about companies wanting to keep tight control of their data etc; bear in mind that not all "web applications" involve the use of the internet. Really all the term means is "applications developed on web-based technologies" and as well as being deployed publically on the web, they're commonly deployed on intranets and other closed-access environments. I work for a software company which develops "web applications" but a large number of systems are hosted by clients for use on their internal networks for the very reasons Dustman refers to - they want to keep tight control of their data. The beauty of web based technologies is that you can achieve this whilst still reaping the benefits of a centralised system, meaning there is no need to manage deployment across 100s of workstations, no need to worry too much about the specifications of client devices, the ability to access the system across different types of device (mobile etc), regular and easily deployed updates, and so I could continue.
It's all about what kind of programs you want to be writing. End-user apps already have already started a significant move to being web-oriented, because of the advantages that some companies find in outsourcing their data handling and IT infrastructure. Because this area of development is a new and growing sector, I have no doubt that you will be getting all kinds of offers, and hearing all about new startups and so forth that are centered on developing these kinds of applications.
That doesn't mean that desktop apps are going to go away. Some companies, and lots of private individuals like to have a sense of being in physical possession of their data, and see no monetary benefit in "renting" an online app or in outsourcing their data handling. These people are going to keep the desktop app market open in the foreseeable future, although perhaps not to the extent that we have seen previously.
So at this point, you needn't feel forced to make a move into the web game, but there are certainly opportunities there if you want them.
In the near future, the boundary between the web development and the desktop application development will go on erasing. For a professional programmer, learning new things is the real growth. learning web development for an experienced programmer is not a difficult task. you can obviously go ahead and learn the web development. You should recognize web well as it will certainly come to meet the desktop apps in the near future.