I'm looking for a way to pick the best AWS region to host a Proof of Concept installation for a potential customer in India.
For this, I'd like to try to ping the customer's web site (I verified that it's hosted in India, I assume by the customer itself since that's part of their business) from multiple AWS regions and see which one gives best results.
I found multiple tools which would allow me to run ping from my own browser to multiple AWS locations (e.g. https://cloudharmony.com/speedtest, http://www.cloudping.info/) but none which will allow me to ping between all AWS regions and a specific third party.
Does such a tool exist, or is my only option to run up an EC2 instance in each region and try to ping from it?
You might want to check the answers to this very similar question.
Keep in mind that not all regions have all AWS services available at this time, so make sure the region you pick has all the services that you plan to use. Also, Amazon has said that an India region is in the works.
Related
Im developing global mobile service communicating with back end server ( S3 - file server , EC2 - application server)
But i don't know how many s3 and ec2 are needed and where i should launch these.
So i'd like to know about below
Im planning to mount S3 in Oregon. As you know, CloudFront is the good solution for getting image quickly but the problem i wanna solve is uploading. I thought 2 solutions. The first solution it that using Put method to CloudFront, upload file to S3 through CloudFront. The second solution is mounting several S3 in different regions. Which is the better solution?
Now i am developing application server in only one EC2. I might have to mount several EC2s for global service. but i don't know how to make end users to connect to specific ec2 of several EC2s. Can you explain me?
thanks
I think your understanding of S3, is slightly off.
You wouldn't and shouldn't need to create "Geo"-specific S3 buckets for the purposes you are describing.
If you are using the service for image delivery over pure HTTP, then you can create the bucket anywhere, and then use a Amazon Cloudfront Distribution as the "frontend" whihc will give you approximately 40 edge locations around the world for your Geo-optimizations.
The more relevant edge location will be used for each user around the world, and they will then request the image from your S3 bucket and store it based on your meta settings. (Typically, from my experience, it's about every 24 hours for low serving traffic websites even when you set an Expire age of months/years.
You also don't "mount" S3. You just create a bucket, and you shouldn't ever want to create multiple buckets which store the same data.
.........
For your sercond question, regarding creating a "global service" for EC2, what are you hopeing to actually achieve.
The web is naturally global. Are your users going to be frett over an additional 200ms latency?
You haven't really descrived what your service will do, but one approach, would be to do all of your computing in Oregon, and then just create cache servers, such as Varnish in different regions. You can use Route53 for the routing, and you can also take advantage of ELB.
My recommendation would be to stop what you are doing, and launch everything from Oregon. The "web" is already "global" and I don't think you are going to need to worry about issues such as this until you hit scale. At which point, I'm going to assume you can just hire someone to solve this problem for you. It sounds like you have the budget for it...
If I turn on a machine in EC2, what expectation of privacy do I have for my running processes, command line history, data stored on ephemeral disk, etc?
Can people at Amazon decide to take a look at what I'm running?
Could Amazon decide to do some profiling for the purposes of upselling?
Hi there! Looks like you're running Cassandra! Here's the optimal
tuning requirements for Cassandra on your m1.xlarge machine!
I can't seem to find anything in the docs...
This is the most applicable thing I found:
AWS only uses each a customer's content to provide the AWS services
selected by that customer and does not use customer content for any
other purposes. AWS treats all customer content the same and has no
insight into what type of content the customer chooses to store in
AWS. AWS simply makes available the compute, storage, database,
mobile, and network services selected by the customer. AWS does not
require access to customer content to provide its services.
http://aws.amazon.com/compliance/data-privacy-faq/
What you are asking about should be addressed in their "Data Privacy" policy (http://aws.amazon.com/agreement/) in their Customer Agreement page:
3.2 Data Privacy. We participate in the safe harbor programs described in the Privacy Policy. You may specify the AWS regions in which Your
Content will be stored and accessible by End Users. We will not move
Your Content from your selected AWS regions without notifying you,
unless required to comply with the law or requests of governmental
entities. You consent to our collection, use and disclosure of
information associated with the Service Offerings in accordance with
our Privacy Policy, and to the processing of Your Content in, and the
transfer of Your Content into, the AWS regions you select.
Here's a link to their "Privacy Policy":
http://aws.amazon.com/privacy/
So in essence, it's saying that you need to consent for them to gather information stored in your server. Now that's different from poking at the TCP ports on your machines from the outside. Amazon constantly runs port checking and traffic checking from the outside (it could be in their intranet too) to make sure you are complying with their customer agreement. For example, they can monitor that you are not hosting something illegal (through public content) or that you are not sending spam or robot traffic to hack into other servers.
Having said that, it's quite possible that they use some of these monitoring tools to check: ok this person has port so and so open. So he/she must be running this application and we can suggest something better for them.
Hope it helps.
I was dorking around with AWS (and related services), hoping that I could stay in the Free Tier, like I do when I'm exploring Google App Engine.
A few days ago, I get a letter from Amazon that they've charged me $33 or so for my 2 days of exploration.
This has got to end, but I forget what services I've enabled. Ideally, I'd just disable the AWS account entirely, as without a free sandbox there's no way I'm going to be using their service. Is there a global off button, or do I have to stumble around to turn all their services off individually? Or do I have to delete my CC information and just create a new Amazon account altogether?
You can close your entire account in AWS Billing: https://console.aws.amazon.com/billing/home?#/account
Or if you just want to disable your "Free-Tier" services that has charges, view them here:
https://console.aws.amazon.com/billing/home#/freetier
Then open your EC2 dashboard - and cancel those services:
https://us-west-2.console.aws.amazon.com/ec2
For example:
Stop running instances, delete volumes, remove elastic IPs, etc.
Otherwise, I recommend sending an email to webservices#amazon.com from the email you used to signup with their service.
I had an RDS running and I couldn't figure out how to cancel just that service
Here's how to do it:
Go to billing services
https://console.aws.amazon.com/billing/home?region=us-west-2#/
Click "Bill Details"
Inspect it
You'll find NAME OF SERVICE + ITS LOCATION. This is the information you need.
https://console.aws.amazon.com/rds/home?region=us-east-1
Go to topright of page. Select the correct server location
The rest is straightforward from here
I was also frustrated (by being charged on the free tier without any info/warning in prior) and found a simple and elegant solution to turn off all AWS services. You delete your account and forget about these fraudulent (to be honest) AWS services.
Here is the link:
https://console.aws.amazon.com/billing/home?#/account
Here is the section:
I know this is a somehow an old question, but I would like to add a new answer because I think AWS has changed a lot since this was asked. I have stumbled on a similar situation as the OP and I found out that there are 3 possible ways to achieve this:
To have a single turn-off-everything button, but I'm not sure if this exists.
Overkill, go through the services and check them one by one and shutdown/delete any instances or running services.
To find out the actual source of leaking (cost occurring services) by viewing what is posting charges on your account and then turn off these services one by one. This can be done by visiting:
your AWS account >> My Billing Dashboard
Find your account username and open the drop down menu:
You can check what services are incurring fees.
Percentage table:
I followed the services by searching for their name on AWS console, if I couldn't find it I'd Google how to do so and then turned them off one by one.
In my case, there was no charge towards my bank even thought billing showed I have some balance, I think it's because I was using the free tier, maybe?
I just hit my free tier limit. I terminated my ec2 instance, deleted my storage volume and even removed my security group and key pair so I have nothing now. Hopefully no charge :P
Always make sure you select the right region. I once had 2 instances running and didnt realize it.
Today I finally discovered a global view to detect all the active services, you still have to disable every service manually but at least you don't have to switch all the regions to understand where you have active services.
I am working on a project and am at a point where the POC is done and now want to move towards a real product. I am trying to understand the Amazon cloud offerings just to see if I need to be aware of them at development time. I have a bunch of questions that I cannot get answered from the Amazon site. Its probably because I am new to the whole web services thing and have never hosted a site before. I am hoping someone out here will explain this to me like I am a C programmer :)
I see amazon has a bunch of offerings -
EC2
Elastic Block Store
Simple DB
AuotScaling
Elastic Load Balancing
I understand EC2 is virtual server instances that I can use and these could come pre-loaded with what I want (say Apache + python). I have the following questions -
If I want a custom instance of something (like say a custom apache module I wrote for my project). Can I create a server instance using the exact modules and make it the default the next time I create a new instance or in Autoscaling?
Do I get an IP Address to access this? Can I set my own hostname to it? I mean do I get a DNS record? Or is it what Elastic IP is?
How do I access it from the outside? SSH? Remote Desktop? Or is it entirely up to how I configure the instance?
What do they mean by Inter-Region or Intra-Region data transfer? What is data transfer to begin with? Is it just people using my instance? So if I go live with it that will be the cost I have to pay for people using it?
What is the difference between AutoScaling and Elastic Load Balancing?
What is Elastic Block Store? Is it storage? If so do I have to worry about backups or do they take care of it?
About the Simple DB -
It looks like the interface to use this is different to my regular SQL calls. Am I correct?
If so the whole development needs to be tailored specifically for Amazon. Which kind of sucks. Is there a better alternative?
Do I get data backups or do I have to worry about it myself?
Will I be able to connect to the DB using regular tools to inspect the DB (during or afte development). Or do I get other tools made by Amazon for it?
What about security? The DB is obviously somewhere in the cloud farm away from the EC2 instance. My DB password is going over the wire and so is all my data totally unencrypted. Don't I have to worry about that? The question comes up only because I don't own any of the hardware.
I really hope some one points me in the right direction here.
Thanks for taking the time to read.
P
I just went through the question and here I tried to answer few of them,
1) AWS EC2 instances doesnt publish pre-configured instances, in fact its configured by the developers and made it publicly available to the users so that they can use it. One can any one of those instances or you can just opt for what ever OS you want which is raw and provision it accordingly and create a snap shot of it so that you can use it for autos caling.The snap shot becomes the base AMI in your case.
2) Every instance you boot will have a public DNS attach to it, you can use the public DNS to connect to that instance using ssh if your are a linux user or using putty if you are a windows users. Apart from that, you can also attach a elastic IP which comes with a cost will is like peanuts and attach it to the instance and access your instance through the elastic IP and you can either map the public DNS or elastic ip to map to a website by adding a A record or Cname respectively.
3)AWS owns databases in the different parts of the world. For example you deploy your application depending upon your customer base, if you target customers are based out of India, the nearest region available is Singapore which is called as ap-southeast-1 by AWS. Each region will have multiple availability zones, example ap-southeast-1a and ap-southeast-1b, which are two different databases and geographically part. Intre region means from ap-southeast-1a to ap-southeast-1b. Inter Region means, from ap-southeast-1 to us-east-1 which is Northern Virginia Data centre. AWS charges from in coming and out going bandwidth, trust me its nothing.
They chargge 1/8th of a cent per GB. Its a thing to even think about it.
4)Elastic Load balancer is cluster which divides the load equally to all your regions across availability zones (if you are running in multi AZ) ELB sits on top the AWS EC2 instances and monitors the instance health periodically and enables auto scaling
5) To help you understand what is autoscaling please go through this document http://aws.amazon.com/autoscaling/
6)Elastic Block store or EBS are like hard disk which is a persistent data storage which can be attached to your instance.Regarding back up yes dependents upon your use case. I do backups of EBS periodically.
7)Simple Db now renamed as dynamo DB is nosql DB, I hope you understand what is nosql db, its a non RDMS db systems. Please read some documentation to understand what is nosql db is.
8)If you have mysql or oracle db you can opt for RDS, please read the documents.
9)I personally feel you are newbie to the entire cloud eco system, you need to understand what exactly cloud does first.
10)You dont have to make large number of changes to development as such, just make sure it works fine in your local box, it can be deployed to cloud with out much ado.
11) You dont have to use any extra tool for that, change the database end point to RDS(if your use it) or else install mysql in your ec2 instance and connect to the local db which resides in the ec2 instance and connect to it,which is as simple as your development mode.
12)You dont have to worry about any security issues aws, it is secured. Dont follow the myths, I am have been using aws since 3 years running I dont even know remember how many applications, like(e-commerce,m-commerce,social media apps) I never faced any kind of security issues and also aws allows to set your security how ever you want.
Go ahead, happy coding. Contact me if you have any problem.
The answer above is a good summary on AWS. Just wanted to add
AWS offers full data center, so it depends what you are trying to achieve. For starters you will need,
EC2 - This is your server, it comes with instance storage, which will be lost on restart
EBS - Your mounted storage, the data is persisted across reboots
S3 - Provides storage (RESTful API's on top, the cost is usage based rather than "provisioned" as in EBS)
Databases - can start with Amazon RDS, which provides managed database services, you can chose between various available databases. You can also install your own database using EC2 + EBS, you will have to take care of managing the database yourself.
Elastic IP: Public facing IP address, you can point your DNS server to this.
One great tool to calculate the pricing,
http://calculator.s3.amazonaws.com/calc5.html
Some other services to take in account are:
VPC (Virtual Private Cloud). This is your own private network. You can define subnets, route tables and internet gateways there. I would strongly recommend to use VPC for any serious deployment of more than one instance.
Glacier - this will replace your tape library to storing backups.
Cloud Formation - great tool for deployment and automation of instances.
When writing a web app with Django or such, what's the best way to connect to dynamic EC2 instances, such as a cluster of Redis or memcache instances? IP addresses change between reboots, etc. Elastic IPs are limited to 5 by default - what are some other options for auto-discovering/auto-updating which machines are available?
Late answer, but use Boto: http://boto.cloudhackers.com/en/latest/index.html
You can use security groups, tags, and other means to hit the EC2 API and pick the instances/IPs for each thing (DB Server, caching server, etc.) at load-time. We do this with great success in deployment, and are moving that way with our Django settings.py, as well.
One method that I heard mentioned recently in an AWS webinar was to store this sort of information in SimpleDB. Essentially, you would use SimpleDB as the central configuration location, and each instance that you launch would register its IP etc. with this configuration, so you would always have a complete description of all of your instances in one place. I haven't seen this in practice so I don't know what the best practices would be exactly, but the idea sounds reasonable. I suppose you could use SNS or something to signal all the other instances whenever the configuration changes, so everyone could refresh their in-memory cache of the configuration.
I don't know the AWS administrative APIs yet really, but there's probably an API call to list your EC2 instances, at which point you could use some sort of custom protocol to ping each of them and ask it what it is -- part of the memcache cluster, Redis, etc.
I'm having a similar problem and didn't found a solution yet because we also need to map Load Balancers addresses.
For your problem, there are two good alternatives:
If you are not using EC2 micro instances or load balancers, you should definitely use Amazon Virtual Private Cloud, because it lets you control instances IPs and routing tables (check all limitations before using this service).
If you are only using EC2 instances, you could write a script that uses the EC2 API tools to run the command ec2-describe-instances to find all instances and their public/private IPs. Then, the script could parameterize instances names to hosts and update /etc/hosts. Finally, you should put the script in the crontab of every computer/instance that need to access the EC2 instances (see ec2-describe-instances).
If you want to stay with EC2 instances (I'm in the same boat, I've read that you can do such things with their VPC or use an S3 bucket or something like that.) but with EC2, I'm in the middle of writing stuff like this...it's all really simple up till the part where you need to contact the server with a server from your data center or something. The way I'm doing it currently is using the API to create the instance and start it...then once its ready, I contact the server to execute a powershell script that I have on the server....the powershell renames the computer and reboots it...that takes care of needing the hostname and MAC for our data center firewalls. I haven't found a way yet to remotely rename a computer.
As far as knowing the IP, the elastic IPs are the way to go. They say you're only allowed 5 and gotta apply for more but we've been regularly requesting more and they give em to us..we're up to like 15 now and they haven't complained yet.
Another option if you dont' want to do all the computer renaming and such...you could use DHCP and set your computer up so when it boots it gets the computer name and everything from DHCP....I'm not sure how to do this exactly, I've come across very smart people telling me that's the way to do it during my research for Amazon.
I would definitely recommend that you get into the Amazon API...I've been working with it for less than a month and I can do all kinds of crazy things. My code can detect areas of our system that are getting stressed, spin up 10 amazon servers all configured to act as whatever needs stress relief, and be ready to send jobs to all in less than 7 minutes. Brings a tear to my eye.
The documentation is very complete...the API itself is a work of art and a joy to program against...I've very much enjoyed working with it. (and no, i dont' work for them lol)
Do it the traditional way: with DNS. This is what it was built for, so use it! When a machine boots, have it ask for the domain name(s) related to its function, and use that for your configuration. If it stops responding, re-resolve the DNS (or just do that periodically anyway).
I think route53 and the elastic load balancing stuff can be used to do this, if you want to stick to Amazon solutions.