I am currently migrating the automated EBS snapshot from a Bash script to Python Boto3. In the original Bash shell, the script was just one line below:
ec2-describe-instances --filter "instance-state-code=16" | grep "vol-" | awk '{print $3}' | xargs -n 1 -t ec2-create-snapshot -d "automated daily backup"
instance state code 16 refer to the running EC2 instances. I am new to Boto3, I have searched up everywhere the closest I can find is to taking snapshots of attached volumes, but that is not good enough as the stopped instances will still be snapshot every night despite nothing is changed on its EBS volumes.
With boto3, you can create a filter for the ec2 resource, where you get only the running instances. From the resulting list of instances, iterate over each of them, and check their block_device_mappings.
You can get the volume-id from the above dictionary. Now, all you need to do is create a snapshot.
A rough code would be:
ec2 = boto3.resource('ec2')
for instance in ec2.instances.filter(
Filters=[{
'Name': "instance-state-name",
'Values': ["running"]
}]
):
for device in instance.block_device_mappings:
ec2.create_snapshot(VolumeId=device.get('Ebs').get('VolumeId'))
This doesn't answer your boto question, but I notice you are using the old-style command-line interface. These days, it is recommended to use the AWS Command-Line Interface (CLI) that has some great capabilities.
For example, this command will list the Volume ID for all EBS volumes attached to instances:
aws ec2 describe-instances --query Reservations[*].Instances[*].BlockDeviceMappings[*].Ebs.VolumeId --output text
You could then add a filter to only show running instances:
aws ec2 describe-instances --query Reservations[*].Instances[*].BlockDeviceMappings[*].Ebs.VolumeId --filter Name=instance-state-name,Values=running --output text
Then you could put it within another command to snapshot volumes of running instances:
aws ec2 create-snapshot --volume-id `aws ec2 describe-instances --query Reservations[*].Instances[*].BlockDeviceMappings[*].Ebs.VolumeId --filter Name=instance-state-name,Values=running --output text`
No strange awk/grep commands required!
Related
I have DB instances in my AWS account. Many volumes are attached to one instance. I want to create an AMI of an Amazon EC2 instance with a root volume on weekly basis. At any point of time I should have the latest AMIs for an instance.
I have tried with systems manager. It’s creating snapshots of all volumes attached with the instance.
I have written a Bash script to create an AMI of an instance with a root volume. I need an approach to delete older images.
Note: The instance should not reboot the AMI creation
How can I update the script or is there is another way
to achieve it?
#!/bin/bash
root_device=$(aws ec2 describe-instances --instance-ids i-12345 --query 'Reservations[*].Instances[*].RootDeviceName' --output text)
echo root device is $root_device
devices=$(for i in $(aws ec2 describe-instances --instance-ids i-12345 --query 'Reservations[*].Instances[*].BlockDeviceMappings[*].DeviceName' --output text );
do if [ $i != $root_device ];
then echo DeviceName=$i,NoDevice=;
fi;
done)
aws ec2 create-image --instance-id i-12345 --block-device-mappings $devices --name "test-ami" --no-reboot
I have created a lambda function to create an AMI on a weekly basis. That solved my problem.
Another advantage is irrespective of the OS, I can use the function take AMI. :)
I have to query my AWS account to find latest created volume with specific tags and should have it attached to running EC2 instance. How do I achieve this using aws cli and powershell?
I was using below powershell script and aws cli to achieve this, But I was not able to find out exact query to get latest volume id using aws cli command to replace variable $volumeid.
Any help would be appreciated.
$volumeid = "aws ec2 describe-volumes --region us-east-1 --filters Name=tag:Application_Name,Values=APPone Name=tag:Name,Values=APP_test --query "Volumes[*].{ID:VolumeId}"
$instanceId = (New-Object System.Net.WebClient).DownloadString("http://169.254.169.254/latest/meta-data/instance-id")
Add-EC2Volume -VolumeId $volumeid -InstanceId $instanceId -Device xvdf -Region us-east-1
You can use the following query to fetch the latest volume id using the specific tag :
aws ec2 describe-volumes --query Volumes[*].[VolumeId] --filters Name=tag-value,Values=testvolume --output text
The above query first describes the volume descriptions of all the volumes.
Then the "--query" returns only the volume ids of all the volumes present.
Later it filters out the volume id based on the Tag you specify.
And the "--output text" converts the output in text format which you can store in the variable.
For more details you can refer describe-volumes
This should help.
I want to attach my newly created IAM Role to all my existence 250 instances, is there any "one-shot" way, because i don't want to go and attach one-by-one for all 250 instances.
Thank You
There's not an API to assign an instance role to more than one instance at a time, but it should be pretty easy to write a few commands that would do it at the command line. There's also SDKs for most languages if you'd prefer a programmatic approach.
To add a little color to your ask, you'll need to create an instance profile associated with your role, and then attach that instance profile to each of your instances, not the role directly.
https://aws.amazon.com/blogs/security/new-attach-an-aws-iam-role-to-an-existing-amazon-ec2-instance-by-using-the-aws-cli/ has all the commands you need to do it, so you just need to iterate over the list and do it for each instance that doesn't already have the correct profile set.
If you need some help programming it, you should come up with some code yourself, paste it here, and we can assist further.
Update: since you don't seem to be ready to jump into the code end of this, I'll get you started in my favorite ad hoc execution environment, the posix shell:
iprofile="your-instance-profile-name"
aws ec2 describe-instances --query 'Reservations[*].Instances[*].InstanceId' \
| xargs -n 1 -P 25 $aws ec2 associate-iam-instance-profile \
YourInstanceId --iam-instance-profile Name=${iprofile} --instance-id
that should list all your instances, and then, in parallel groups of 25, assiciate $iprofile with each instance.
The Python equivalent would be something like:
import boto3
ec2 = boto3.client('ec2', region_name='ap-southeast-2')
instances = ec2.describe_instances()
for reservation in instances['Reservations']:
for instance in reservation['Instances']:
response = client.associate_iam_instance_profile(
IamInstanceProfile={Name='XYZ'},
InstanceId=instance['InstanceId']
)
(I didn't test it!)
Off the back of Daniel's solution, I created something more readable that worked for me
#!/bin/bash
#### Variables
region=eu-west-2
profile_id=<insert_profile_from_/.aws/credentials>
iam_instance_profile=<insert_IAM_instance_profile>
instances=$(aws ec2 describe-instances --query 'Reservations[*].Instances[*].InstanceId' --profile $profile_id --region $region | awk '{print $1}' | sed 's/[]","[]//g')
assign_instances()
{
for instance in $instances; do aws ec2 associate-iam-instance-profile --instance-id $instance --iam-instance-profile Name=$iam_instance_profile --region $region --profile $profile_id; done
}
assign_instances
Faily new to AWS however I am looking to terminate a set of ec2 instances using the AWS CLI by filtering by a Tag name.
If I use describe-instances, I can filter by tag:key=value . For terminate-instances I don't see a way of filtering. I assume this is possible since I can filter and terminate using the AWS console but I am looking to do this via CLI.
Latest AWS CLI allows you to avoid the need for any scripts or jq:
aws ec2 terminate-instances --instance-ids $(aws ec2 describe-instances --query 'Reservations[].Instances[].InstanceId' --filters "Name=tag:tagkey,Values=tagvalue" --output text)
as long as the number of expected instances is not huge, the above can be used.
The terminate-instances command only takes a list of instance IDs. You would need to write a script to run the describe-instances command first and capture the instance IDs, then pass those IDs to the terminate-instances command.
I created the following script(.sh) and it worked for me:
aws ec2 describe-instances --query 'Reservations[*].Instances[*].[InstanceId]' --filters 'Name=tag-value,Values=MYTAG' --output text |
grep stopped |
awk '{print $2}' |
while read line;
do aws ec2 terminate-instances --instance-ids $line
done
I have found a script for starting/stopping a dynamically created ec2 instance, but how do I start any instances in my inventory?
Seems you are talking about scripting, not SDK. So there are two tools to do the job.
1 AWS CLI tools
download aws cli tool and set the API Key in $HOME/.aws/credentials
list all instances on region us-east-1
Confirm which instances you are targeting.
aws ec2 describe-instances --query 'Reservations[].Instances[].InstanceId' --region us-east-1 --output text
2 Amazon EC2 Command Line Interface Tools
download and setup instruction
list all instances on region us-east-1
You should get same output as WAY #1.
ec2-describe-instances --region us-west-2 |awk '/INSTANCE/{print $2}'
With the instance ID list, you can use your command to start them one by one.
for example, the instance name are saved in file instance.list
while read instance
do
echo "Starting instance $instance ..."
ec2-start-instances "$linstance"
done < instance.list
BMW, give you an excellent startup, but you can even summarise the thing like this:
1) First get the id of all the instances and save them into a file
aws ec2 describe-instances --query 'Reservations[].Instances[].InstanceId' --region us-east-1 --output text >> id.txt
2) Then simply run this command to start all the instances
for id in $(awk '{print $1}' id.txt); do echo "starting the following instance $id"; aws ec2 start-instances --instance-ids --region us-east-1 $id; done
Please change the region, I am considering that you have installed and setup the AWS CLI tools properly. Thanks