From an AWS newbie. Before I throw money at Amazon for higher-performing servers for my ETL tool, am I getting the most out of my existing setup?
I have set up a SQL Server RDS instance (db.m1.small) to which I'm connecting from a RHEL-6 EC2 instance (m1.small). On the EC2 I have a Python ETL job to read source flat-file data and run inserts to the RDS over the Microsoft ODBC driver. This process it an order of magnitude slower than when everything is hosted on my 64-bit Core-i7 Windows laptop with a large SSD drive.
My two servers are not in a VPC but are talking to each other via the same Internet IPs that I connect to them with from my laptop; could this be at the root of my performance problem? Could performance be improved if they were talking over local IPs instead? I've looked at Amazon's documentation of VPCs but they tout security improvements only, never performance improvements, over EC2 security groups.
Thanks for any help, BH
Related
I'm working on an eCommerce website (developed and set up by other develpers) that have the web server on a m4.2xlarge instance and the database on a t2.micro (EC2 and EBS instances).
The usual traffic is about 5,000 unique visitors/day but we are expecting a huge increment in the nex future because of some advertising on big newspapers and I would like to understand if I need to scale up the web server instance or the DB instance or both.
It is not clear to me if the two instances are sharing the same RAM and CPU or if they are completely separated.
Thank you !
Both the systems will have different CPU and RAM.
You should choose RDS instead of a normal EC2 instance for Database.
You can take a backup from older instance and load into your RDS instances. This will also help you maintaining high availability as well.
I need to set up a public-facing web server and a private database server in AWS. So that I can allow higher availability - is it okay that I have my public subnet in one AZ hosting my web server and another (so duplicate) WebServer in a different AZ? The same principle would apply to database server configuration. Essentially, Zone-A would host WebServer, Zone-B would host WebServer-copy. Zone-B would host DatabaseServer, Zone-A would host Database-copy. Is this architecture a good practice?
If yes, does this configuration mean site files and database files are duplicated on each AZ?
Yes, that's basically how you setup high availability on AWS. I would recommend using RDS for the database which will manage multi-az deployments for you automatically. Managing the data replication manually can be a real challenge.
I would also recommend looking into Elastic Beanstalk which will manage distributing the traffic across multiple zones, deploying and updating your application across multiple zones, and all the details that go along with that. I would not recommend diving straight in and trying to do all this manually in EC2 if you are new to AWS.
Let me preface this by saying that I am primarily a programmer, though I have a pretty good working knowledge of Linux and "standard" LAMP installations. I have been tasked with setting up a persistent LAMP environment in Amazon Web Services (AWS), which is a good deal more involved than what I'm used to in this regard.
Although AWS is very well documented, I have yet to find a clear, definitive "Best Practices" overview for setting up a persistent LAMP environment. I followed the official Amazon tutorial ( http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/install-LAMP.html ) to set up a LAMP server on our EC2 instance, but found out later that these instances are "temporary" and that I need an EBS to make anything persist. Interestingly, EBS (Elastic Block Storage) does not appear in my Management Console , though they offer pricing out on the public side ( https://aws.amazon.com/ebs/pricing/ ). Is it still called EBS?
Of course, that begs the question - what happens to the programs I installed (Apache, MySQL) along with their respective config files? Surely Amazon doesn't expect us to reconfigure our server from scratch every time it boots up?
What I have now
1x EC2 instance running Amazon Linux. I installed and configured Apache and MySQL following the "Install LAMP" tutorial posted by Amazon.
1x Route 53 Hosted Zones (for DNS routing)
1x Elastic IP attached to the EC2 server
Additionally, there appears to be one unencrypted 8GB volume attached to /dev/xvda, although I didn't set it up and nobody has access but myself - it seems to have been generated when I requisitioned the EC2 - no idea if it is persistent or not.
What I think I need
So, here is what I'm thinking I need to do. Please tell me if I'm way off - is there a more sane alternative?
1x EC2 instance running Amazon Linux and Apache
1x RDS for MySQL
1x Route 53 Hosted Zone
1x Elastic IP attached to the EC2 server
1x (EBS? S3? EFS?) for storing htdocs
1x Snapshot of the EC2 to save server configuration
Does that sound right? Is there a better way to do this? Thanks so much for any advice. Amazon docs seem to be very good at giving granular information, but not as great at addressing overall strategy concerns.
Web Application
It is recommended to have 2 EC2 instances under an Elastic Load Balancer with these 2 instances being in separate availability zones for high availability. Going further is better to monitor these instances for CPU and bandwidth - CloudWatch - and once you see they are above some threshold you could automatically add more instances to the ELB, this is the auto scaling. Of course like you said you will need the AMI (snapshot) with the server software to be ready to be launched. You also need to take down servers when the load is small - again automatically with the metrics, but you should never go down under 2 machines. And don't forget to update these images when you upgrade the software.
Route 53
Because you would use an ELB you don't need Elastic IPs anymore, your web servers could only have private IPs. And on the Route 53 you need to point your website to the name of the ELB - here are more details about it
Database
For the database part go for the RDS for MySQL including Multi-AZ deployment, so you will have the master and one stand by replica in different availability zones.
EBS (disks)
For the EBS part you will ned to use that and they come in 3 flavours: magnetic(slowest), General Purpose SSD (faster) and Provisioned IOPS (fastest). These are the disks you mount on your machines, web servers and databases. For the database you should go with Provisioned, it is much harder to change them later, while for the web server we use General Purpose. In the AWS Console you find them under EC2, section Elastic Block Store.
The 8G disk that appeared is the default when you create a Linux machine and it is a General Purpose SSD which is good enough for a web server, but I think you should go with a bigger one, 50G at least.
I don't understand the use for AWS EC2 and RDS as separate services.
Is an EC2 instance a necessary prerequisite of launching an RDS instance? Can you have a database on your EC2 instance?
RDS is a database-as-a-service; you don't know or care what it is running on, because you don't install it, manage it or perform any of the maintenance on the box it runs on (i.e. software updates). AWS takes care of all of that for you and simply charges you by the hour for using the service.
You can install SQL Server or MySQL (or any other database) on your own EC2 instance, but now its your responsibility to administer. Both are valid uses cases, but in general, for small databases, and especially for test and dev databases, I install SQL Server on my own EC2 instances and manage it myself to have flexibility and save money.
For larger, more critical and production databases, RDS is often a better choice - at a higher cost. The best scenario for you will depend on a lot of things (your budget, your skills, the need for high availability etc).
You can install any databases on ec2 instance (whatever in windows or Linux). ec2 is a standalone virtual server with your fully control
But RDS service, you needn't care of the detail on how to install the database, set the environment, upgrade the version, backup/restore, multi-AZ setup, and so on, AWS RDS Service will take care all of them. Get it ready and you can immediately connect it and start your work. The way to provide RDS services are very similar as service elasticache (memcached & redis)
There is no require for you to launch a new ec2 instance for RDS service setup.
Don't mix ec2 and RDS. They are separate services.
Another way to make you understand this question is,
Amazon RDS is a Software as a Service (SaaS) that provides a full featured relational database service
Is it possible to have most of our server hardware outside of EC2, but with some kind of load balancer to divert traffic to EC2 when there's load that our servers can't handle, or as a backup incase these servers go down?
For example, have a physical server serving out our service (let's ignore database consistency for the moment), but there's a huge spike due to some coolness - can we spin up some EC2 instances and divert traffic off to it? This is much like Amazon's own auto scaling.
And also, if our server hardware dies for some reason (gremlins eat the power cables for example) - can we route all our traffic over to EC2 instances?
Thanks
Yes you can but you will have to code. AWS has Command Line Tools for doing EC2/Autoscaling/S3 stuff with simple commands in bash or other interfaces and SDKs, like Boto for Python etc.
You can find it here: http://aws.amazon.com/code/
Each Ec2 instance has a public network interface associated with it. Use a DNS CNAME record to "switch" your site traffic to the Ec2 instance. If you need to load-balance across multiple machines you can use round-robin DNS, or start a ELB and put any number of Ec2 instances behind it.
Ec2 infrastructure is extremely easy to scale. Deploying your application on top of an Ec2 is a whole other matter. It could be trivial -- or insanely complicated.