This is from the amazon ec2 FAQ :
Q: How quickly can I scale my capacity both up and down?
Amazon EC2 provides a truly elastic computing environment. Amazon EC2 enables you to increase or decrease capacity within minutes, not hours or days. You can commission one, hundreds or even thousands of server instances simultaneously. When you need more instances, you simply call RunInstances, and Amazon EC2 will typically set up your new instances in a matter of minutes. Of course, because this is all controlled with web service APIs, your application can automatically scale itself up and down depending on its needs.
Now again as per the same FAQ, I am only allowed to launch 20 instances per region. They said, I have to fill in a request form if I need more than 20 instances. So, in effect, I cant spin up more than 20 programmatically ?
What am I missing here ? how can we launch 100 instances let alone thousands. Sorry if this is the wrong place for such a question.
You cannot launch instances beyond the instance limit. You need to make a request to increase the instance limit. This is a safety feature so that:
A wild loop in your SDK/API script does not launch instances continuously
A malicious user does not launch a large number of instances
A hacker gets access to your account and launches a large number of instances
An incorrectly configured autoscaling group launches huge number of instances
If you require more than your instance limit, you need to submit a request to AWS. See: Amazon EC2 Service Limits. AWS will review your request and approve it.
You are missing the fact that limit increase requests are very easy to make and are almost always granted with no questions asked within a day or two.
To request a limit increase:
Open the AWS Support Center page, sign in if necessary, and choose Create Case.
For Regarding, choose Service Limit Increase.
Complete Limit Type, Use Case Description, and Contact method. If this request is urgent, choose Phone as the method of contact instead of Web.
Choose Submit.
AWS faqs provides a clear answer
You are limited to running up to a total of 20 On-Demand instances across the instance family, purchasing 20 Reserved Instances, and requesting Spot Instances per your dynamic Spot limit per region. New AWS accounts may start with limits that are lower than the limits described here. Certain instance types are further limited per region as follows
For Spot instance limits AWS states
The usual Amazon EC2 limits apply to instances launched by a Spot Fleet, such as Spot request price limits, instance limits, and volume limits. In addition, the following limits apply:
The number of active Spot Fleets per region: 1,000
The number of launch specifications per fleet: 50
The size of the user data in a launch specification: 16 KB
The target capacity per Spot Fleet: 3,000
The target capacity across all Spot Fleets in a region: 5,000
A Spot Fleet request can't span regions.
A Spot Fleet request can't span different subnets from the same Availability Zone.
These limits protect you from a hacker attack, stolen API keys, ETC. If you want to increase these limits, you need to send a form to AWS support team: AWS Support Center
Related
While launching 2nd instance getting below error
you have requested more vcpu capacity than your current vcpu limit of 1 allows for the instance bucket that the specified instance type belongs to. please visit http://aws.amazon.com/contact-us/ec2-request to request an adjustment to this limit.
First, please note that the AWS Free Tier is a billing discount. It does not impact your ability to use any services. If the resources you use are within the limits imposed by the AWS Free Tier, you simply won't be charged for those resources. Thus, the AWS Free tier is not the cause of this error.
From On-Demand Instances - Amazon Elastic Compute Cloud:
There is a limit on the number of running On-Demand Instances per AWS account per Region. On-Demand Instance limits are managed in terms of the number of virtual central processing units (vCPUs) that your running On-Demand Instances are using, regardless of the instance type.
You did not tell us what Instance Type you are running, or attempted to run, but it sounds like you have exceeded this vCPU limit. This is more likely to happen with large or expensive instance types, or instance types that use GPU resources. You can view the limits in the Amazon EC2 management console by clicking Limits and then searching for vCPU.
If this is impacting your use of AWS, you can request an increase to these default limits.
See: Calculate a vCPU limit increase request for an Amazon EC2 On-Demand Instance
I am trying to use aws spot instances (m5.large in eu-west-2 region) with a maximum bid equal to the price of on demand instances. According to https://aws.amazon.com/ec2/spot/instance-advisor/ these instances should have a < 5% frequency of interruption, however, after launching 40 such instances this morning, I have found that within the hour 34 of them were evicted by aws ("instance-terminated-no-capacity" according to the spot requests page on the ec2 dashboard).
This eviction rate looks much too high compared to both amazon's own advisor and other users experiences. Does anybody know what could be causing this behaviour, if there is any better way to debug it or predict it, or if this is just what I should expect from spot instances?
Thank you!
Actually, for m5.large instance in eu-west-2 region(Oregon) it's 5%-10% frequency of interruption, so you can expect a max of 10%. I'm not saying that issue you are facing is because of this.
AWS terminates your spot instances because of any of these reasons,
The Spot price is above the maximum price.
There isn't enough capacity.
Amazon EC2 can't meet the constraints you placed on your Spot request.
In your case, since you are seeing instance-terminated-no-capacity message it is definitely because of the second reason. Since you've asked for 40 such instances, the amazon spot instance pool might not have enough capacity at that time.
The capacity of available spot instances pool depends on the demand for regular instances, and when users ask for regular on-demand instances, AWS will start terminating spot instances to fulfil those requests if there is not enough capacity
In my experience spot interruption is highly variable, and for some instances more or less likely at different times of the day.
If you need 40 instances and they do not need to be in the same availability zone (AZ) to each other you might reduce the chance of a mass interruption of all/most instances if you spread the machines across different AZs within the region as each availability zone has its own pool of machines. Although you will likely increase the chance that some machines will be interrupted.
Note this is not an option if you are using EMR, then they have to be in the same AZ.
I'm getting ready to launch a mobile app that I have hosted on AWS with an EC2 instance. ($0.0464 per On Demand Linux t2.medium Instance Hour).
This past month I was charged $112 for the EC2 usage, but only had a handful of internal users testing the private version of the app. It's a fairly simple app, not anything that should require a lot of computing power.
So what I'm wondering is if 10 users and dev team costs $112/mo, what happens if I get 1,000 users, or 10k users? Would the cost increase 100x, 1000x? I can't imagine getting auto-billed for $112,000 for a month of service with a small user base like 10k users.
Thanks for any help and guidance, I don't know much about AWS.
Here are the details of my billing for last month:
The billing page shows 2219 hours of t2.medium during this billing month.
That is the equivalent of 92 days. So, it might be 3 instances running for a full month.
Amazon EC2 is charged when the instance is in the Running state. If you are not using an instance, you can Stop the instance. The attached disks (EBS) will still be charged, but there will be no charge for the instance itself.
The charge is not based on the number of users, nor how 'busy' the instance is. It is simply charged when the instance is 'running'. This is because computer resources are exclusively assigned to instances (CPU, RAM) that nobody else can use.
Bottom line: Stop instances that you don't need. Use the smallest instance type for your use-case to reduce costs.
If you were not aware of the charges involved, you can contact AWS Customer Service and request a refund.
FYI, the T2 and T3 family are great for workloads that occasionally 'burst' but then have low-usage periods, but they are not great for sustained workloads. See: Burstable performance instances - Amazon Elastic Compute Cloud
I am getting this error Master Instance Group: Exceeded EC2 Instance Quota, when I create a new cluster on Amazon EMR with 1 Master node only or 1 Master and 2 Core nodes. However, there are no EC2 instances running on my account.
What should I do? I raised a ticket, asking if I can get a quicker solution here.
In your case I think you are trying to access a new region or new instance type, AWS sometimes does that when you are in free tier, they allow access to 2-3 regions or free instance types only. Then you have to request access from AWS by raising a case.
But in Normal scenario, this is what happens:
You may face an error like Exceeded EC2 Instance Quota while you are trying to spin up new instances either standalone or in cluster.
This error is caused because you have hit the limit on number of instances allowed in your AWS account.
This limit is region and instance size specific. To get rid of this error you will have to request Amazon to increase the EC2 instance limit.
Requesting a limit increase is simple. Below are the steps:
Most service limit increases can be requested through the AWS Support Center by choosing Create Case and then choosing Service Limit Increase.
Most service limits are specific to a particular AWS Region, so be sure to submit a request for each Region you plan to use. Many services support requesting multiple limit increases for the same service through one support case. After creating your first request, choose Add another request and then choose a new limit type or Region.
This is probably too simple a question, but how does one create many instances (in the low hundreds).
Our process can require up to ten instance/task and we can need to run up to a dozen instances.
However, the upper limit of instances we can run/account is 10-20.
Up until now, we have gone around this with multiple AWS accounts, which is creaky to say the least. We would prefer something more like a large cluster.
Is there a way of upping the limit programmatically, or does one have to make a special AWS request?
Thanks.
All accounts have a default limit of around 20 instances. This limit can easily be increased through the AWS console.
You can do this by following these steps:
-Go to the AWS Console to this link
https://console.aws.amazon.com/ec2/v2/home?region=us-east-1#Limits:
-Click Request limit increase for each instance type you want increased
-You'll need to do this for each region
Provided you don't submit a crazy high limit request (1000) instances, your request should be approved automatically within a few minutes.
See the Docs for more info EC2 limits:
http://docs.aws.amazon.com/general/latest/gr/aws_service_limits.html#limits_ec2