So I am following this link: Autoscale based on SQS queue size to create an autoscaling group for my instances. I have read many articles about this problem that I am getting and many people are getting the same problem, but theirs occurs when they try to use "t1.micro". Whereas, I am using "c4.xlarge" instance type and I already have a VPC defined for my Image. Why am I still getting this error:
Launching a new EC2 instance. Status Reason: The specified instance
type can only be used in a VPC. A subnet ID or network interface ID is
required to carry out the request. Launching EC2 instance failed.
Does anybody have a solution for this?
you need to include VPC information in your scripts or init:
http://docs.aws.amazon.com/autoscaling/latest/userguide/asg-in-vpc.html
Not sure what sdk you are using, but with any sdk you have choosen, you need to specify the VPC subnets where you generate the instances.
When using aws cli to create a ASG, you specify the same using --vpc-zone-identifier
Please check the link to documentation below:
http://docs.aws.amazon.com/cli/latest/reference/autoscaling/create-auto-scaling-group.html
Make sure you are defining the subnet id in the cli command.
Although the service is different, aws cli generally follows the same syntax so adjust this to any resource.
aws emr create-cluster \
--name "Test cluster" \
--release-label emr-4.2.0 \
--applications Name=Hadoop Name=Hive Name=Pig \
--use-default-roles \
--ec2-attributes KeyName=myKey,SubnetId=subnet-77XXXX03 \
--instance-type m4.large \
--instance-count 3
Related
I want to launch an EC2 machine using aws cli. I want several things to take place before I connect, including setting my configuration.
I successfully launch the machine using:
aws ec2 run-instances --image-id ami-062f7200baf2fa504 --count 1 \
--instance-type t2.micro --key-name MyFirstKey --security-group-ids \
launch-wizard-3 --user-data file://aws_setup_script.txt
my aws_setup_script.txt is
sudo yum update -y
aws configure set aws_access_key_id AAAAABBBBBCCCCCDDDDD
aws configure set aws_secret_access_key AAAAABBBBBCCCCCDDDDDEEEEEFFFFFGGGGGHHHHH
aws configure set default.region us-east-1
sudo yum update -y successfully runs, but the aws configure steps do not.
It is insecure passing secrets in user-data.
Your script is failing because it isn't running as ec2-user so it doesn't have aws in the path. Even if it worked, it wouldn't be configuring the CLI tool for the ec2-user account so it isn't going to work the way you want.
Most importantly, there is a much better way to accomplish this. You should be assigning an IAM instance profile to the instance. When you run the aws cli tool on an instance with an IAM role assigned it will automatically use those credentials.
As per best practice, It's always better to use the IAM instance role attached to your Ec2 instead of setting the AWS credentials within Ec2.
Create an IAM instance role (refer AWS Doc) with the required permission want to give to Ec2.
Use --iam-instance-profile in aws cli command to attache the Ec2 with specific Iam role.
aws ec2 run-instances --image-id ami-062f7200baf2fa504 --count 1 \
--instance-type t2.micro --key-name MyFirstKey --security-group-ids \
launch-wizard-3 --iam-instance-profile
How do I enable the AWS CLI on an EC2 instance? After I create the EC2 instance, I can SSH into the machine, but when I try to do something like aws s3 ls, it prompts me to do aws configure first, which I then have to enter my keys. I want to be able to automate this so that I can grab additional artifacts from S3 buckets to install. Note that I am using the AWS CLI on my computer to create the EC2 instance, but I need to use the AWS CLI on the EC2 instance itself.
My AWS command to create a simple EC2 instance looks like the following (this is done on my computer).
aws ec2 run-instances \
--image-id ami-14c5486b \
--count 1 \
--instance-type t2.micro \
--key-name testkey \
--subnet-id subnet-xxxxxxxx \
--security-group-ids sg-xxxxxxxx \
--tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=test}]'
--user-data file://install-software.sh
The install-software.sh looks something like the following (this is submitted to the EC2 instance).
#!/bin/bash
aws s3 cp s3://mybucket/some-archive.tar.gz some-archive.tar.gz
tar xf some-archive.tar.gz
sudo some-archive/bin/install.sh
You need to use an instance profile when launching your EC2 instance – if it has an instance profile attached then the AWS CLI will automatically use the permissions set in it to grant access to resources, rather than relying on your providing credentials.
You need to assign an instance role to your instance. Give it rights to get objects from your bucket. Then the aws cli will get the credentials from instance metadata automatically so you won't need to configure aws first.
Is it possible to create and Auto Scaling Group Launch config with the CLI and define the instance tags in one command?
Maybe I am missing something but right now it looks like have to do it in two steps.
i.e.
aws autoscaling create-launch-configuration ...
and then
aws autoscaling create-or-update-tags --tags ...
Since you need to have asg LC created first to tag it, it is two step process as you mentioned.
https://docs.aws.amazon.com/cli/latest/reference/autoscaling/create-launch-configuration.html
This example creates a launch configuration based on an existing instance. In addition, it also specifies launch configuration attributes such as a security group, tenancy, Amazon EBS optimization, and a bootstrapping script:
https://docs.aws.amazon.com/autoscaling/ec2/userguide/autoscaling-tagging.html
aws autoscaling create-launch-configuration --launch-configuration-name my-launch-config --key-name my-key-pair --instance-id i-7e13c876 --security-groups sg-eb2af88e --instance-type m1.small --user-data file://myuserdata.txt --instance-monitoring Enabled=true --no-ebs-optimized --no-associate-public-ip-address --placement-tenancy dedicated --iam-instance-profile my-autoscaling-role
aws autoscaling create-or-update-tags --tags "ResourceId=my-asg,ResourceType=auto-scaling-group,Key=environment,Value=test,PropagateAtLaunch=true"
It's pretty easy via the console but I need to do the same from CLI.
First I created a db snapshot:
aws rds create-db-cluster-snapshot \
--db-cluster-snapshot-identifier $SNAPSHOT_ID \
--db-cluster-identifier $CLUSTER \
CLUSTER contains only one writer instance
I did not use create-db-snapshot method because it throwned an error
A client error (InvalidParameterValue) occurred when calling the CreateDBSnapshot operation: The specified instance is a member of a cluster and a snapshot cannot be created directly. Please use the CreateDBClusterSnapshot API instead.
It works:
aws rds create-db-cluster-snapshot \
--db-cluster-snapshot-identifier $SNAPSHOT_ID \
--db-cluster-identifier $CLUSTER \
{
"DBClusterSnapshot": {
"Engine": "aurora",
"SnapshotCreateTime": "2016-12-08T11:48:07.534Z",
....
}
So, I wanted to restore a new Aurora cluster from the snapshot, then I tried:
aws rds restore-db-instance-from-db-snapshot \
--db-instance-identifier from-snap2 \
--db-snapshot-identifier snap2 \
A client error (DBSnapshotNotFound) occurred when calling the RestoreDBInstanceFromDBSnapshot operation: DBSnapshot not found: snap2
So I tried to restore with:
aws rds restore-db-cluster-from-snapshot \
--db-cluster-identifier from-snap2 \
--snapshot-identifier snap2 \
--engine aurora \
--vpc-security-group-ids $PREPROD_SG \
--db-subnet-group-name my-db-subnet-group \
It works...
{
"DBCluster": {
...
"EngineVersion": "5.6.10a",
"DBClusterIdentifier": "from-snap2",
...
"DBClusterMembers": [],
...
}
But why the cluster does not contain any Aurora instance?
Where is the mistake?
This is very counterintuitive. If you restore a cluster from a snapshot, but there are no member instances in the cluster, what operation has actually succeeded? It seems as if all this does is create some kind of logical entity, maybe the backing store, but no instances.
Strange. But, the API documentation does show the cluster members as an empty set in the example response.
<DBClusterMembers/>
So it seems you create a cluster, as you did, then you apparently create instances in the cluster, as explained in an AWS Forum post:
aws rds create-db-instance --db-instance-identifier my-instance --db-instance-class db.r3.large --engine aurora --db-subnet-group-name default-vpc-xxxxxx --db-cluster-identifier my-instance-cluster
https://forums.aws.amazon.com/thread.jspa?messageID=688727
Apparently the console encapsulates multiple API requests behind the same action.
Response from AWS Support:
This is a known issue when using the API calls and our engineers are working on it. Even if the cluster is visible on AWS Console after the creation via CLI it will not create any instance automatically in your Aurora Cluster. In this case, you will need to create a db-instance and associate it to your newly restored cluster. When performing this Action on the AWS Console a new instance is automatically created for the cluster, but the action from the CLI uses separated API calls.
The following documentation provides detailed information on how to create a DB instance:
http://docs.aws.amazon.com/cli/latest/reference/rds/create-db-instance.html
You can describe your clusters using the AWS Console or using the CLI:
http://docs.aws.amazon.com/cli/latest/reference/rds/describe-db-clusters.html
Here is a command line example that creates the instance and associate it to a fictional cluster:
aws rds create-db-instance --engine aurora --db-cluster-identifier yourauroraclusteridentifier --db-instance-class db.t2.medium --db-instance-identifier yourinstanceidentifier
In my case, --db-cluster-identifier is the cluster created from the cluster snapshot.
If you create with aws rds create-db-cluster-snapshot then you can't restore with aws rds restore-db-instance-from-db-snapshot. The first creates a DB snapshot and the second restores a Cluster snapshot, different types.
From your question it looks like your restore is correct, maybe you need --database-name specified. Also you could try the restore with only the required parameters, i.e no vpc sg or DB subnet.
I am using this link to install Spark Cluster on EMR(Elastic Map Reduce on Amazon) https://aws.amazon.com/articles/Elastic-MapReduce/4926593393724923
For creating a Spark cluster I run the following command and my cluster is running into bootstrap failure every single time. I am not able to resolve this issue, and it will be great if any could help me here.
aws emr create-cluster --name SparkCluster --ami-version 3.2 \
--instance-type m3.xlarge --instance-count 3 --ec2-attributes \
KeyName=MYKEY --applications Name=Hive --bootstrap-actions \
Path=s3://support.elasticmapreduce/spark/install-spark
SOLVED : Use this:
aws emr create-cluster --name SparkCluster --ami-version 3.7 \
--instance-type m3.xlarge --instance-count 3 --service-role \
EMR_DefaultRole --ec2-attributes \
KeyName=emr,InstanceProfile=EMR_EC2_DefaultRole \
--applications Name=Hive --bootstrap-actions \
Path=s3://support.elasticmapreduce/spark/install-spark
Summary of the answer (it took a bit of back and forth in comments) that worked for this user given the user's SSH key and IAM roles:
aws emr create-cluster --name SparkCluster --ami-version 3.7 --instance-type m3.xlarge --instance-count 3 --service-role EMR_DefaultRole --ec2-attributes KeyName=emr,InstanceProfile=EMR_EC2_DefaultRole --applications Name=Hive --bootstrap-actions Path=s3://support.elasticmapreduce/spark/install-spark
Explanations of EMR IAM roles can be found at http://docs.aws.amazon.com/ElasticMapReduce/latest/DeveloperGuide/emr-iam-roles-creatingroles.html and http://docs.aws.amazon.com/ElasticMapReduce/latest/DeveloperGuide/emr-iam-roles-launch-jobflow.html
The 4th point under the section Spark with YARN on an Amazon EMR cluster at the link you provide says the following:
Substitute "MYKEY" value for the KeyName parameter with the name of the EC2 key pair you want to use to SSH into the master node of your EMR cluster.
As far as I can see, you have not changed the value of MYKEY for your own EC2 key name. You should try changing its value to an existing EC2 key name you have already created.
In case you still do not have a keypair, you can created following several methods, one of which is described in this link.
Update (from the comments below)
From your pictures, it seems there is a problem downloading the bootstrap action file from S3. I am not sure what the cause of the problem could be, but you might want to change the AMI and launch EMR with a different AMI version, 3.0, for example.
There is another way to directly start spark cluster in EMR.
Step 1 - Go to the EMR section in aws and click on create cluster.
Step 2 - Go to bootstrap actions in the configuration and add this line
s3://support.elasticmapreduce/spark/install-spark
https://www.pinterest.com/pin/429953095652701745/
Step 3 - Click on create cluster
Your cluster will start in minutes :)