I've been trying to wrap my head around the best solution for hosting development sites for our company lately.
To be completely frank I'm new to AWS and it's architecture, so more then anything I just want to know if I should keep learning about it, or find another more suitable solution.
Right now we have a dedicated server which hosts our own website, our intranet, and a lot of websites we've developed for clients.
Our own web and the intranet isn't an issue, however I'm not quite sure about the websites we produced for our clients.
There are about 100 of them right now, these sites are only used pre-launch so our clients can populate the sites with content. As soon as the content is done we host the website somewhere else. And the site that is still on our developer server is no longer used at all, but we keep them there if the client wants a new template/function so we can show it there before sending it to production.
This means the development sites have almost zero traffic, with perhaps at most 5 or so people adding content to them at any given time (5 people for all 100 sites, not 5 per site).
These sites needs to be available at all times, and should always feel snappy.
These are not static sites, they all require a database connection.
Is AWS (ES2, or any other kind of instance, lightsail?) a valid solution for hosting these sites. Or should I just downgrade our current dedicated server to a VPS, and just worry about hosting our main site on AWS?
I'll put this in an answer because it's too long, but it's just advice.
If you move those sites to AWS you're likely to end up paying (significantly) more than you do now. You can use the Simple Monthly Calculator to get an idea.
To clarify, AWS is cost-effective for certain workloads. It is cost effective because it can scale automatically when needed so you don't have to provision for peak traffic all the time. And because it's easy to work with, so it takes fewer people and you don't have to pay a big ops team. It is cost effective for small teams that want to run production workloads with little operational overhead, up to big teams that are not yet big enough to build their own cloud.
Your sites are development sites that just sit there and see very little activity. Which means those sites are probably under the threshold of cost effective.
You should clarify why you want to move. If the reason is that you want as close to 100% uptime as possible, then AWS is a good choice. But it will cost you, both in terms of bill paid to Amazon and price of learning to set up such infrastructure. If cost is a primary concern, you might want to think it over.
That said, if your requirements for the next year or more are predictable enough and you have someone who knows what they are doing in AWS, there are ways to lower the cost, so it might be worth it. But without further detail it's hard for anyone to give you a definitive answer.
However. You also asked if you should keep learning AWS. Yes. Yes, you should. If not AWS, one of the other major clouds. Cloud and serverless[1] are the future of much of this industry. For some that is very much the present. Up to you if you start with those dev sites or something else.
[1] "Serverless" is as misleading a name as NoSQL. It doesn't mean no servers.
Edit:
You can find a list of EC2 (Elastic Cloud Compute) instance types here. That's CPU and RAM. Realistically, the cheapest instance is about $8 per month. You also need storage, which is called EBS (Elastic Block Store). There are multiple types of that too, you probably want GP2 (General Purpose SSD).
I assume you also have one or more databases behind those sites. You can either set up the database(s) on EC2 instance(s), or use RDS (Relational Database Service). Again, multiple choices there. You probably don't want Multi-AZ there for dev. In short, Multi-AZ means two RDS instances so that if one crashes the other one takes over, but it's also double the price. You also pay for storage there, too.
And, depending on how you set things up you might pay for traffic. You pay for traffic between zones, but if you put everything in the same zone traffic is free.
Storage and traffic are pretty cheap though.
This is only the most basic of the basics. As I said, it can get complicated. It's probably worth it, but if you don't know AWS you might end up paying more than you should. Take it slow and keep reading.
Related
You'll have to excuse my ignorance on this one...but honestly, I've had a hard time finding clarity on this. That being said, I'm looking for a non technical answer...something in layman's terms!
Anyways, I've been playing around building a web app (first time obviously) and I'm getting to the point where I've started looking into hosting services. A quick google search and a few blogs later, I thought AWS would be a good place to start, since they give a free-year trial. I don't care about speedy upstarts or other hosting serves, so save your key strokes on offering other services.
My question is based on the fact that AWS charges "Linux Usage per hour" and they also use this term "instance". Yeah...an "instance" is an "object", which is also above my head (probably the real source of the problem), but that was the extent I was able to learn via a google search. That being said, I don't know how to translate the cost into a ball park cost. Yes, I can probably use the trial to help monitor predictable costs, but I don't want to go through the effort of learning one hosting companies system just to find out it's not going to work in the end.
OK...so hopefully by now you see where I'm coming from. What is an "instance" and how do I use the "Linux Usage per hour" to estimate cost? Is an instance a server? For example if I start NGINX is that in instance? Is it just one instance running NGINX or does every VPN represent an instance? If I have 100 people calling the server at once, can they fit on one instance? If I start another server say, Apache or Node, does that become another instance? If I connect to a database, is that an instance? Do instances start as needed? Yes, I know, that's more than one question...I'm just trying to express my confusion.
If I'm suppose to choose a pricing model from this list, "Linux Usage per hour", I need to know what them mean by "Linux Usage". If it's based on an "instance", I need to know what that is. So please, in layman's terms, help clear this up. Maybe some examples or analogies, but no deep technical stuff.
This is more a side note, but I was reading this article and it said
For a client needing to run 800 virtual instances, the annual cost of
a private cloud came to below $400,000 vs. somewhere between $800,000
and $1.2 million for public cloud services.
Considering I don't know what an instance is, that kinda made me a bit nervous...WAAAAAAyyyyyy outta my price range! Yes, it's obviously a big company, but can you imagine "hitting the lottery" with an app everyone loves then before you know it, AWS hits you with a bill of $1,000,000. Or even worse, your security sucks and someone spawns millions of these "instances"...help alive my paranoia!!
Basically, an instance is a virtual machine, which looks very much like a server. As such it's running an operating system - e.g. linux - which is capable of running many programs (aka 'processes' or sometimes, 'services') at the same time.
To go through your questions (some of the explanations below are not technically accurate, but are hopefully more explanatory for it - if anything is obvious or already known, apologies - trying not to assume any knowledge)
An instance is an object
This definition is coming up in your searches because 'instance' has many definitions in different situations. If you see the definition of 'instance' as an object, it's from the topic of object oriented programming languages - you define a class in your code (kind of like a 'template'), and then create instances of the class - kind of like real copies of the template.
Amazon borrowed the term to be analogous - because in the 'cloud' world, you can create an AMI (Amazon Machine Image - the template) and then create lots of instances that are copies or clones of that template.
Is an instance a server?
In terms of what you can do with it, yes, it's a server.
(Technically it's a virtual server - Amazon runs multiple virtual servers on each physical server.)
how do I use the "Linux Usage per hour" to estimate cost?
Estimate how long you will have your instance running for in hours per month, multiply it by cost per hour and you will have your estimated cost per instance per month.
e.g. - one instance always turned on would be - 24 hrs * 31 days = 744 hours. At $0.013/hr (for a t2.micro) that would be 744 * $0.013 = $9.672/mth.
(And that's the reason the free tier gives you 750 hours of instance time per month.)
Instances come in different types and sizes and each size costs a different amount. If you are not sure what size you need, I'd start with the smallest until you discover you need more - which would be when your program starts running too slowly.
For example if I start NGINX is that in instance?
Nginx is a program that runs as a daemon in linux terms - a program that runs in the background so it's always on. It will be one of the many programs running on the server (aka the instance)
If I have 100 people calling the server at once, can they fit on one instance?
It depends - on how big your instance is, and how efficient the program is that is responding to their requests. If you are just getting started learning to program websites, I wouldn't worry about handling 100 people issuing requests to the server all at once just yet - walk before you run :) (also, even when there are 100 people visiting your website, the odds that all of them issue a request at exactly the same time is low - usually they load a page and read it - while they're reading it, some of the other people are loading other pages, and it all spreads out so you might only have ~10 page requests actively being processed by your server at the same time.)
However, if you have 2,000 people on your site at the same time, you might be processing 200 page requests at once, so by then you do need to have put some thought into performance and scalability.
(Note: these numbers are arbitrary and depend entirely on the type of site and it's traffic patterns.)
Generally, most websites pick a mid-level instance size, and then to handle more requests they 'scale out' - create lots of copies of that instance, and allow each instance to handle a portion of the traffic.
If I start another server say, Apache or Node, does that become another instance
The language to use here would be 'start another service say, Apache or Node' - they are other programs, and your instance will be perfectly fine running nginx, apache and node all at the same time. Although each will consume some of the resources (e.g. memory and cpu) and the more activity they are doing, the faster you will run out of resources and need to get a bigger instance size
So - no, they don't automatically become another instance. The language is confusing because sometimes people don't distinguish between the 'server' (aka the instance) and the service (aka the program) and will say the 'apache server' and the 'apache service' interchangably.
If I connect to a database, is that an instance?
Your instance, as a fully capable server, could run a database service on it at the same time as the other services - e.g. you could install and run mysql on your instance.
There is another option, though - if you use the AWS RDS product, then you will be starting an RDS instance. An RDS instance is different from an EC2 instance (what we've been talking about so far) in that RDS instances are specialised to just run the database service and nothing else, but EC2 instances are general servers that you can do pretty much anything on.
It's usually recommended to use RDS, but if you are trying to save money and aren't serving many users, there's nothing particularly wrong with installing mysql on your instance yourself (especially while you're learning how it works) and then moving your data to an RDS instance when you want to support more load or traffic.
Do instances start as needed?
Not by default, no - you have to manually start and stop them.
However, there are options other than manually starting and stopping. Amazon provides a lot of APIs, so you could write a program that would connect to the API and automatically start and stop your instance(s) based on rules you build into your program..
Also, Amazon offers a product called "AutoScalingGroups" which allows you to have a related group of instances and for Amazon to automatically start and stop them according to rules that you configure into that product. These rules can be 'scheduled actions' - start/stop at certain times of day - or they can be reactive - e.g. when the average CPU usage is > 50% for more than 5 minutes, start another instance.
This is more a side note, but I was reading this article and it said
For a client needing to run 800 virtual instances, the annual cost of
a private cloud came to below $400,000 vs. somewhere between $800,000
and $1.2 million for public cloud services.
The 'free tier' gives you a t2.micro sized instance (1 vCPU, 1 GiB RAM) which you could leave turned on permanently for free during that free year.
Even after your free tier expires, that same instance would cost you $9.67/mth, and you have the option to go downgrade to a t2.nano (0.5 GiB RAM) which would only cost ~$4/mth - but 0.5GiB RAM isn't much these days, so may not be enough for you.
A t2.micro should be more than enough to learn how to build websites on. If you are fortunate enough to build a site that is popular enough that you are getting more requests than that server can handle, then you will have to decide if you can generate revenue from that popularity sufficient to cover the cost, but by then you'll have more of a sense of how efficient your program is, and what instance size (and/or how many instances) you'll need.
Yes, it's obviously a big company, but can you imagine "hitting the
lottery" with an app everyone loves then before you know it, AWS hits
you with a bill of $1,000,000
AWS protects you from yourself here a bit - they have limits which generally restrict you from running more than 20 instances at a time - unless you ask for permission. So, by default, your instance won't go multiplying like rabbits on it's own - unless you set it up to. And even if you have set it up to, it won't be able to grow beyond 20 instances unless you have asked amazon to let you. So, worst case is 20 x $9.67/mth - $197/mth.
But - that's just the instance cost. Amazon charges you for lots of things including data traffic in and out, RDS instance costs, and if you start using other service such as S3 buckets and/or elastic load balancers, they all attract their own costs.
But hopefully, if you hit the lottery with an app everyone loves, you've worked out how to convert that love into dollars and cents so you can pay for all those instances you're going to need :)
I have several web-based projects or apps and their server side programs are currently running on a VPS. I found out the AWS EC2 service recently and I'm quite curious about whether it's better for my situation. I've searched up on Google for some nice and simple explanations of the pros and cons of EC2, but some of the articles are a little complicated and confusing and I didn't really understand. I did, however, understand they're saying two big things of EC2 are flexibility and scalability. But these two don't seem to apply to my situation. All my projects or apps are nonprofit highly customized private services. Normally they're invite only and I don't really expect a lot of people to use them. So I don't really worry about what if they grow so fast and crash my server. As for the cost, EC2 is not necessarily better either. I pay a little below $200 each year for the VPS service and it runs great. I had little trouble with it. EC2 service actually costs a little more than that and I didn't find my apps have better performance running on it. Some people say with EC2 you pay only for what you use but the thing is I can't shutdown my apps just because I think nobody would be using it at the moment.
IN your case, sounds like no. If you are paying only $200/year, and everything works to your satisfaction, why mess with it.
I have a web app running on php, mysql, apache on a virtual windows server. I want to redesign it so it is scalable (for fun so I can learn new things) on AWS.
I can see how to setup an EC2 and dump it all in there but I want to make it scalable and take advantage of all the cool features on AWS.
I've tried googling but just can't find a simple guide (note - I have no command line experience of Linux)
Can anyone direct me to detailed resources that can lead me through the steps and teach me? Or alternatively, summarise the steps in an answer so I can research based on what you say.
Thanks
AWS is growing and changing all the time, so there aren't a lot of books to help. Amazon offers training that's excellent. I took their three day class on Architecting with AWS that seems to be just what you're looking for.
Of course, not everyone can afford to spend the travel time and money to attend a class. The AWS re:Invent conference in November 2012 had a lot of sessions related to what you want, and most (maybe all) of the sessions have videos available online for free. Building Web Scale Applications With AWS is probably relevant (slides and video available), as is Dissecting an Internet-Scale Application (slides and video available).
A great way to understand these options better is by fiddling with your existing application on AWS. It will be easy to just move it to an EC2 instance in AWS, then start taking more advantage of what's available. The first thing I'd do is get rid of the MySql server on your own machine and use one offered with RDS. Once that's stable, create one or more read replicas in RDS, and change your application to read from them for most operations, reading from the main (writable) database only when you need completely current results.
Does your application keep any data on the web server, other than in the database? If so, get rid of all local storage by moving that data off the EC2 instance. Some of it might go to the database, some (like big files) might be suitable for S3. DynamoDB is a good place for things like session data.
All of the above reduces the load on the web server to just your application code, which helps with scalability. And now that you keep no state on the web server, you can use ELB and Auto-scaling to automatically run multiple web servers (and even automatically launch more as needed) to handle greater load.
Does the application have any long running, intensive operations that you now perform on demand from a web request? Consider not performing the operation when asked, but instead queueing the request using SQS, and just telling the user you'll get to it. Now have long running processes (or cron jobs or scheduled tasks) check the queue regularly, run the requested operation, and email the result (using SES) back to the user. To really scale up, you can move those jobs off your web server to dedicated machines, and again use auto-scaling if needed.
Do you need bigger machines, or perhaps can live with smaller ones? CloudWatch metrics can show you how much IO, memory, and CPU are used over time. You can use provisioned IOPS with EC2 or RDS instances to improve performance (at a cost) as needed, and use difference size instances for more memory or CPU.
All this AWS setup and configuration can be done with the AWS web console, or command-line tools, or SDKs available in many languages (Python's boto library is great). After learning the basics, look into CloudFormation to automate it better (I've written a couple of posts about that so far).
That's a bit of the 10,000 foot high view of one approach. You'll need to discover the details of each AWS service when you try to use them. AWS has good documentation about all of them.
Depending on how you look at it, this is more of a comment than it is an answer, but it was too long to write as a comment.
What you're asking for really can't be answered on SO--it's a huge, complex question. You're basically asking is "How to I design a highly-scalable, durable application that can be deployed on a cloud-based platform?" The answer depends largely on:
The specifics of your application--what does it do and how does it work?
Your tolerance for downtime balanced against your budget
Your present development and deployment workflow
The resources/skill sets you have on-staff to support the application
What your launch time frame looks like.
I run a software consulting company that specializes in consulting on Amazon Web Services architecture. About 80% of our business is investigating and answering these questions for our clients. It's a multi-week long project each time.
However, to get you pointed in the right direction, I'd recommend that you look at Elastic Beanstalk. It's a PaaS-like service that abstracts away the underlying AWS resources, making AWS easier to use for developers who don't have a lot of sysadmin experience. Think of it as "training wheels" for designing an autoscaling application on AWS.
I was contracted to make a groupon-clone website for my client. It was done in PHP with MYSQL and I plan to host it on an Amazon EC2 server. My client warned me that he will be email blasting to about 10k customers so my site needs to be able to handle that surge of clicks from those emails. I have two questions:
1) Which Amazon server instance should I choose? Right now I am on a Small instance, I wonder if I should upgrade it to a Large instance for the week of the email blast?
2) What are the configurations that need to be set for a LAMP server. For example, does Amazon server, Apache, PHP, or MySQL have a maximum-connections limit that I should adjust?
Thanks
Technically, putting the static pages, the PHP and the DB on the same instance isn't the best route to take if you want a highly scalable system. That said, if the budget is low and high availablity isn't a problem then you may get away with it in practise.
One option, as you say, is to re-launch the server on a larger instance size for the period you expect heavy traffic. Often this works well enough. You problem is that you don't know the exact model of the traffic that will come. You will get a certain percentage who are at their computers when it arrives and they go straight to the site. The rest will trickle in over time. Having your client send the email whilst the majority of the users are in bed, would help you somewhat, if that's possible, by avoiding the surge.
If we take the case of, say, 2,000 users hitting your site in 10 minutes, I doubt a site that hasn't been optimised would cope, there's very likely to be a silly bottleneck in there. The DB is often the problem, a good sized in-memory cache often helps.
This all said, there are a number of architectural design and features provided by the likes of Amazon and GAE, that enable you, with a correctly designed back-end, to have to worry very little about scalability, it is handled for you on the most part.
If you split the database away from the web server, you would be able to put the web server instances behind an elastic load balancer and have that scale instances by demand. There also exist standard patterns for scaling databases, though there isn't any particular feature to help you with that, apart from database instances.
You might want to try Amazon mechanical turk, which basically lots of people who'll perform often trivial tasks (like navigate to a web page click on this, etc) for a usually very small fee. It's not a bad way to simulate real traffic.
That said, you'd probably have to repeat this several times, so you're better off with a load testing tool. And remember, you can't load testing a time-slicing instance with another time-slicing instance...
Is there an easy way to migrate a hosted LAMP site to Amazon Web Services? I have hobby sites and sites for family members where we're spending far too much per month compared to what we would be paying on AWS.
Typical el cheapo example of what I'd like to move over to AWS:
GoDaddy domain
site hosted at 1&1 or MochaHost
a handful of PHP files within a certain directory structure
a small MySQL database
.htaccess file for URL rewriting and the like
The tutorials I've found online necessitate PuTTY, Linux commands, etc. While these aren't the most cumbersome hurdles imaginable, it seems overly complicated. What's the easiest way to do this?
The ideal solution would be something like what you do to set up a web host: point GoDaddy to it, upload files, import database, done. (Bonus points for phpMyAdmin being already installed but certainly not necessary.)
It would seem the amazon AWS marketplace has now got a solution for your problem :
https://aws.amazon.com/marketplace/pp/B0078UIFF2/ref=gtw_msl_title/182-2227858-3810327?ie=UTF8&pf_rd_r=1RMV12H8SJEKSDPC569Y&pf_rd_m=A33KC2ESLMUT5Y&pf_rd_t=101&pf_rd_i=awsmp-gateway-1&pf_rd_p=1362852262&pf_rd_s=right-3
Or from their own site
http://www.turnkeylinux.org/lampstack
A full LAMP stack including PHPMyAdmin with no setup required.
As for your site and database migration itself (which should require no more than file copies and a database backup/restore) the only way to make this less cumbersome is to have someone else do it for you...
Dinah,
As a Web Development company I've experienced an unreal number of hosting companies. I've also been very closely involved with investigating cloud hosting solutions for sites in the LAMP and Windows stacks.
You've quoted GoDaddy, 1And1 and Mochahost for micro-sized Linux sites so I'm guessing you're using a benchmark of $2 - $4 per month, per site. It sounds like you have a "few" sites (5ish?) and need at least one database.
I've yet to see any tool that will move more than the most basic (i.e. file only, no db) websites into Cloud hosting. As most people are suggesting, there isn't much you can do to avoid the initial environment setup. (You should factor your time in too. If you spend 10 hours doing this, you could bill clients 10 x $hourly-rate and have just bought the hosting for your friends and family.)
When you look at AWS (or anyone) remember these things:
Compute cycles is only where it starts. When you buy hosting from traditional ISPs they are selling you cycles, disk space AND database hosting. Their default levels for allowed cycles, database size and traffic is also typically much higher before you are stopped or charged for "overage", or over-usage.
Factor in the cost of your 1 database, and consider how likely it will be that you need more. The database hosting charges can increase Cloud costs very quickly.
While you are likely going to need few CCs (compute cycles) for your basic sites, the free tier hosting maximums are still pretty low. Anticipate breaking past the free hosting and being charged monthly.
Disk space it also billed. Factor in your costs of CCs, DB and HDD by using their pricing estimator: http://calculator.s3.amazonaws.com/calc5.html
If your friends and family want to have access to the system they won't get it unless you use a hosting company that allows "white labeling" and provides a way to split your main account into smaller mini-hosting accounts. They can even be setup to give self-admin and direct billing options if you went with a host like www.rackspace.com. The problem is you don't sound like you want to bill anyone and their minimum account is likely way too big for your needs.
Remember that GoDaddy (and others) frequently give away a year of hosting with even simple domain registrations. Before I got my own servers I used to take HUGE advantage of these. I've probably been given like 40+ free hosting accounts, etc. in my lifetime as a client. (I still register a ton of domain through them. I also resell their hosting.)
If you aren't already, consider the use of CMS systems that support portaling (one instance, many websites under different domains). While I personally prefer DotNetNuke I'm sure that one of its LAMP stack competitors can do the same for you. This will keep you using only one database and simplify your needs further.
I hope this helps you make a well educated choice. I think it'll be a fine-line between benefits and costs. Only knowing the exact size of every site, every database and the typical traffic would allow this to be determined in advance. Database count and traffic will be your main "enemies". Optimize files to reduce disk-space needs AND your traffic levels in terms of data transferred.
Best of luck.
Actually it depends upon your server architecture, whether you want to migrate whole of your LAMP stack to Amazon EC2.
Or use different Amazon web services for different server components like Amazon S3 for storage and Amazon RDS for mysql database and so.
In case if you are going with LAMP on EC2: This tutorial will atleast give you a head up.
Anyways you still have to go with essential steps of setting up the AMI and installing LAMP through SSH.