This is probably too simple a question, but how does one create many instances (in the low hundreds).
Our process can require up to ten instance/task and we can need to run up to a dozen instances.
However, the upper limit of instances we can run/account is 10-20.
Up until now, we have gone around this with multiple AWS accounts, which is creaky to say the least. We would prefer something more like a large cluster.
Is there a way of upping the limit programmatically, or does one have to make a special AWS request?
Thanks.
All accounts have a default limit of around 20 instances. This limit can easily be increased through the AWS console.
You can do this by following these steps:
-Go to the AWS Console to this link
https://console.aws.amazon.com/ec2/v2/home?region=us-east-1#Limits:
-Click Request limit increase for each instance type you want increased
-You'll need to do this for each region
Provided you don't submit a crazy high limit request (1000) instances, your request should be approved automatically within a few minutes.
See the Docs for more info EC2 limits:
http://docs.aws.amazon.com/general/latest/gr/aws_service_limits.html#limits_ec2
Related
I am trying to host Unreal Engine Pixel Streaming Build to AWS and while doing the setup I am getting stuck at the launch of Instance. I followed this tutorial here.
Please look at the error below:
Any help would be appreciated since I am a beginner in AWS.
There are default limits for various Amazon EC2 instance types. These are based upon the total number of vCPUs simultaneously running. You can access this information by selecting Limits in the sidebar of the EC2 management console.
You can click the Request limit increase button to submit a request for the limit to be increased.
These limits are sometimes to prevent fraud (eg people consuming lots of resources and then not paying their bill), and sometimes to protect people from accidentally running the more-expensive instances (eg the X family).
The g4dn.4xlarge shown in that video tutorial costs $1.204/hour (depending upon region used).
Check out this aws doc:
https://aws.amazon.com/premiumsupport/knowledge-center/ec2-on-demand-instance-vcpu-increase/ It will let you see your current limit, and if you need to request an increase (or just choose a different instance type/class)
Also double-check you are in the right region. An increase is only for the region you request it in. You can check what your limit is via the Quotas page: https://console.aws.amazon.com/servicequotas/home?region=us-east-1#!/dashboard
This is from the amazon ec2 FAQ :
Q: How quickly can I scale my capacity both up and down?
Amazon EC2 provides a truly elastic computing environment. Amazon EC2 enables you to increase or decrease capacity within minutes, not hours or days. You can commission one, hundreds or even thousands of server instances simultaneously. When you need more instances, you simply call RunInstances, and Amazon EC2 will typically set up your new instances in a matter of minutes. Of course, because this is all controlled with web service APIs, your application can automatically scale itself up and down depending on its needs.
Now again as per the same FAQ, I am only allowed to launch 20 instances per region. They said, I have to fill in a request form if I need more than 20 instances. So, in effect, I cant spin up more than 20 programmatically ?
What am I missing here ? how can we launch 100 instances let alone thousands. Sorry if this is the wrong place for such a question.
You cannot launch instances beyond the instance limit. You need to make a request to increase the instance limit. This is a safety feature so that:
A wild loop in your SDK/API script does not launch instances continuously
A malicious user does not launch a large number of instances
A hacker gets access to your account and launches a large number of instances
An incorrectly configured autoscaling group launches huge number of instances
If you require more than your instance limit, you need to submit a request to AWS. See: Amazon EC2 Service Limits. AWS will review your request and approve it.
You are missing the fact that limit increase requests are very easy to make and are almost always granted with no questions asked within a day or two.
To request a limit increase:
Open the AWS Support Center page, sign in if necessary, and choose Create Case.
For Regarding, choose Service Limit Increase.
Complete Limit Type, Use Case Description, and Contact method. If this request is urgent, choose Phone as the method of contact instead of Web.
Choose Submit.
AWS faqs provides a clear answer
You are limited to running up to a total of 20 On-Demand instances across the instance family, purchasing 20 Reserved Instances, and requesting Spot Instances per your dynamic Spot limit per region. New AWS accounts may start with limits that are lower than the limits described here. Certain instance types are further limited per region as follows
For Spot instance limits AWS states
The usual Amazon EC2 limits apply to instances launched by a Spot Fleet, such as Spot request price limits, instance limits, and volume limits. In addition, the following limits apply:
The number of active Spot Fleets per region: 1,000
The number of launch specifications per fleet: 50
The size of the user data in a launch specification: 16 KB
The target capacity per Spot Fleet: 3,000
The target capacity across all Spot Fleets in a region: 5,000
A Spot Fleet request can't span regions.
A Spot Fleet request can't span different subnets from the same Availability Zone.
These limits protect you from a hacker attack, stolen API keys, ETC. If you want to increase these limits, you need to send a form to AWS support team: AWS Support Center
My scenario is mentioned below, please provide the solution.
I need to run 17 HTTP Rest API's for 30K users.
I will create 6 AWS instances (Slaves) for running 30K (6 Instances*5000 Users) users.
Each AWS instance (Slave) needs to handle 5K Users.
I will create 1 AWS instance (Master) for controlling 6 AWS slaves.
1) For Master AWS instance, what instance type and storage I need to use?
2) For Slave AWS instance, what instance type and storage I need to use?
3) The main objective is a Single AWS instance need to handle 5000Users (5k) users, for this what instance type and storage I need to use? This objective needs to solve for low cost (pricing)?
Full ELB DNS Name:
The answer is I don't know, this is something you need to find out how many users you will be able to simulate on this or that AWS instance as it depends on the nature of your test, what it is doing, response size, number of postprocessors/assertions, etc.
So I would recommend the following approach:
First of all make sure you are following recommendations from the 9 Easy Solutions for a JMeter Load Test “Out of Memory” Failure
Start with single AWS server, i.e. t2.large and single virtual user. Gradually increase the load at the same time monitor the AWS health (CPU,RAM, Disk, etc) using either Amazon CloudWatch or JMeter PerfMon Plugin. Once there will be a lack of the monitored metrics (i.e. CPU usage exceeds 90%) stop your test and mention the number of virtual users at this stage (you can use i.e. Active Threads Over Time listener for this)
Depending on the outcome either switch to other instance type (i.e. Compute Optimized if there is a lack of CPU or Memory Optimized if there is a lack of RAM) or go for higher spec instance of the same tier (i.e. t2.xlarge)
Once you get the number of users you can simulate on a single host you should be able to extrapolate it to other hosts.
JMeter master host doesn't need to be as powerful as slave machines, just make sure it has enough memory to handle incoming results.
In amazon aws how many m1.small spot request can i create in one region by default
when i see in Instances limit it shows me some thing like
Spot Instance requests default Request limit increase
but currently i am running 36 m1.small
and it gives me an error while creating new spot request
MaxSpotInstanceCountExceeded
so my question is how many spot request can i make of one particular type (eg.m1.small or m1.large)
or how does amazon aws calculate MaxSpotInstanceCountExceeded
MaxSpotInstanceCount is the hard limit of on number of spot instances that you can launch irrespective of their type. the default is 5. However, in your case it seems to be 36.
You can request AWS to increase the limit by filling out this form: https://aws.amazon.com/support/createCase?type=service_limit_increase&serviceLimitIncreaseType=ec2-instances
For any other doubts, please reach out to their support: https://aws.amazon.com/support
I want to store around 10 GB of data on AWS services and use map reduce to process the data.
Is using EC2 the best option ? I want to use free tier service, it says maximum of 613 MB for free services on EC2 and that does not satisfy my requirement. I am doing a hobby project and my expenses are limited.
The free tier FAQ also talks about using AWS EBS with free 30 GB of data. Can I use Map Reduce services on EBS too, since AFAIK EMR is only available on EC2 ?
Does anyone know of any other alternatives that I can use for the same ?
Try the AWS Simply Monthly Calculator, located at http://calculator.s3.amazonaws.com/calc5.html#s=EMR, to get a feel for how much your project will cost using AWS.
The recommended workflow for EMR is to store data in an S3 bucket. So in the calculator, click S3 on the left. In the form enter 10G. The price for S3 storage is about $0.10 per gb/mo, so 10G costs about $1.00/mo.
Then, click on Amazon Elastic Map Reduce on the left. The form allows you to select predicted number of instances, hours/week or hours/mo expected usage, and expected instance type needed for your project. For example, for a project that requires 20 hrs/week using 1 Small EC2 is estimated to cost around $6.00. Micro instances do not seem to be offered with EMR.
Therefore, if you think you can get by with a Small Instance, and you plan to use it infrequently, your expenses might be under $10 per mo.
To reduce expenses even further, you could use spot instances rather than standard instances, as explained here: http://aws.amazon.com/ec2/spot-instances/#7.