Find and remove outdated AMIs in AWS - amazon-web-services

I need to remove a list of outdated Ami's in AWS. But first I require to verifying are there any place still using those Ami's. Is there an efficient way to find it out.

First, you need to define "Outdated AMIs", it can be the creation date or something else. But here is the script that will list Instance ID along with AMI details that used by particular instance in particular region. By default it will look into default region.
#Get list of EC2 instance
echo "Getting EC2"
EC2_LIST=$(aws ec2 describe-instances --query 'Reservations[].Instances[].{InstanceId:InstanceId}' --output text | tr '\n' ' ')
#Get list of AMI used by ec2
echo "Getting AMI"
LIST_AMI_ID=$(aws ec2 describe-instances --query 'Reservations[].Instances[].{ImageId:ImageId,InstanceId:InstanceId}' --output text | tr '\n' ' ')
EC2_LIST_ARRAY=($EC2_LIST)
LIST_AMI_ID_ARRAY=($LIST_AMI_ID)
for index in ${!LIST_AMI_ID_ARRAY[*]}; do
echo "Get details for AMI ${LIST_AMI_ID_ARRAY[$index]}"
#Get details of AMI
AMI_DETAILS=$(aws ec2 describe-images --image-ids ${LIST_AMI_ID_ARRAY[$index]} --query 'Images[].{CreationDate:CreationDate,Tags:Tags[]}')
echo "Instance having ID ${EC2_LIST_ARRAY[$index]} using AMI ID ${LIST_AMI_ID_ARRAY[$index]} Details: $AMI_DETAILS"
done
output
Get details for AMI ami-0219162cf838b3455
Instance having ID i-0ceb0dfa197fd7455 using AMI ID ami-0219162cf838b3455 Details: [
{
"CreationDate": "2019-10-22T05:17:46.000Z",
"Tags": [
{
"Key": "Base_AMI_Name",
"Value": "ubuntu/images/hvm-ssd/ubuntu-bionic-18.04-amd64-server-20191021"
},
{
"Key": "OS_Version",
"Value": "Ubuntu"
},
{
"Key": "Release",
"Value": "Latest"
},
{
"Key": "Name",
"Value": "postgres"
},
{
"Key": "Scope",
"Value": "database"
}
]
}
]
aws-cli-cheatsheet

For what it's worth, I built a small tool for this purpose as well -- a bit of JavaScript wrapped around AWS API calls.
This basically tries to find all the AMIs in use in your account and looks at both AMIs by their age and also the newly introduced deprecated flag:
https://github.com/geoffreywiseman/oldamis
https://www.codiform.com/blog/oldamis/
It's pretty simplistic currently, but I'm willing to entertain feature requests. ;)

Related

Find all aws load balancers which has a particular instance

I would like to get names of all the aws load balancers which has a particular instance.
I can list the instances in the ELB's using the following command
aws elb describe-load-balancers --query "LoadBalancerDescriptions[*].{ID:LoadBalancerName,InstanceId:Instances[*].InstanceId}[*]. {ELB:ID,InstanceId:InstanceId[*]}" --output=json
Sample Output:
[
{
"ELB": "my_name",
"InstanceId": [
"instance-id-A",
"instance-id-B",
]
},
{
"ELB": "my_name2",
"InstanceId": [
"instance-id-B",
"instance-id-C"
]
},
{
"ELB": "my_name3",
"InstanceId": [
"instance-id-A",
"instance-id-C"
]
}
]
How do I filter this output to only return the ELB names which has instance id A?
The contains command will give you what you're looking for
aws elb describe-load-balancers --query "LoadBalancerDescriptions[*].{ID:LoadBalancerName,InstanceId:Instances[*].InstanceId}[?contains(to_string(#),'instance-id-A')]"
Provides output:
[
{
"ID": "lb_name",
"InstanceId": [
"i-1234567890"
]
}
]
References
http://jmespath.org/specification.html#contains
Another approach, which I like better as I think is more easy to use, is to use jq for filtering:
aws elb describe-load-balancers | jq -r '.LoadBalancerDescriptions[] | select (.Instances[].InstanceId == "instance-id-A") | .LoadBalancerName'
Output:
SampleLBName1
SampleLBName2

Get a count of active AWS network interfaces

I need to get a count of the number of 'active' network interfaces in my region.
I can use the query below to list out the active interfaces, however I am uncertain on how to get a 'count' of the active interfaces and not just a json payload as the output.
I can run this cli command to get the list of all attached interfaces.
aws ec2 describe-network-interfaces --filters "Name=group-name,Values=Redis" "Name=attachment.status,Values=attached"
However I'm not sure how to get a count of interfaces that are attached, i tried the query below but I'm not getting the desired output to just get a count.
aws ec2 describe-network-interfaces --filters "Name=group-name,Values=Redis" "Name=attachment.status,Values=attached" --query 'NetworkInterfaces[*][Attachment.Status,Attachment.Status.Count]'
The output comes out like below.
[
[
"attached",
null
],
[
"attached",
null
],
[
"attached",
null
],
[
"attached",
null
]
]
What I'd like to see if something like below.
[
[
type: "attached",
Count: "x"
]
]
It seems that you want to count the number of Elastic Network Interfaces (ENIs) in the region to know when you are approaching the limit.
The limits page in the EC2 management console doesn't seem to state that the ENI count is only for attached images, so you should probably count the total number of ENIs.
This could be done with:
aws ec2 describe-network-interfaces --query 'length(NetworkInterfaces)'
If you only wish to count count ENIs that are attached, use:
aws ec2 describe-network-interfaces --filters Name=attachment.status,Values=attached --query 'length(NetworkInterfaces)'

AWS CLI EMR get Master node Instance ID and tag it

I want to automate the running of a cluster and can use tags to get attributes of an EC2 instance like its instance-id.
The documentation on https://docs.aws.amazon.com/cli/latest/reference/emr/create-cluster.html states that
--tags (list)
A list of tags to associate with a cluster, which apply to each Amazon
EC2 instance in the cluster. Tags are key-value pairs that consist of
a required key string with a maximum of 128 characters, and an
optional value string with a maximum of 256 characters.
You can specify tags in key=value format or you can add a tag without
a value using only the key name, for example key . Use a space to
separate multiple tags.
So this applies tags to every EC2 instance including the master and slaves. How do I discern which instance is the master node?
Additional Info:
I am using the following command to get attributes from aws cli based on tags where you can replace the "Name" and "Prod" with your tags key-value pairs respectively.
aws ec2 describe-instances | jq '.Reservations[].Instances | select(.[].Tags[].Value | startswith("Prod") ) | select(.[].Tags[].Key == "Name") | {InstanceId: .[].InstanceId, PublicDnsName: .[].PublicDnsName, State: .[].State, LaunchTime: .[].LaunchTime, Tags: .[].Tags} | [.]' | jq .[].InstanceId
As you noted when you create an EMR cluster, the tags are the same for all nodes (Master, Slave, Task).
You will find that this process using the AWS CLI to be complicated. My recomendation is to review the examples below and then write a Python program to do this.
Process to add your own tags to the EC2 instances.
STEP 1: List your EMR Clusters:
aws emr list-clusters
This will output JSON:
{
"Clusters": [
{
"Id": "j-ABCDEFGHIJKLM",
"Name": "'MyCluster'",
"Status": {
"State": "WAITING",
"StateChangeReason": {
"Message": "Cluster ready after last step completed."
},
"Timeline": {
"CreationDateTime": 1536626095.303,
"ReadyDateTime": 1536626568.482
}
},
"NormalizedInstanceHours": 0
}
]
}
STEP 2: Make a note of the Cluster ID from the JSON:
"Id": "j-ABCDEFGHIJKLM",
STEP 3: Describe your EMR Cluster:
aws emr describe-cluster --cluster-id j-ABCDEFGHIJKLM
This will output JSON (I have truncated this output to just the MASTER section):
{
"Cluster": {
"Id": "j-ABCDEFGHIJKLM",
"Name": "'Test01'",
....
"InstanceGroups": [
{
"Id": "ig-2EHOYXFABCDEF",
"Name": "Master Instance Group",
"Market": "ON_DEMAND",
"InstanceGroupType": "MASTER",
"InstanceType": "m3.xlarge",
"RequestedInstanceCount": 1,
"RunningInstanceCount": 1,
"Status": {
"State": "RUNNING",
"StateChangeReason": {
"Message": ""
},
"Timeline": {
"CreationDateTime": 1536626095.316,
"ReadyDateTime": 1536626533.886
}
},
"Configurations": [],
"EbsBlockDevices": [],
"ShrinkPolicy": {}
},
....
]
}
}
STEP 4: InstanceGroups is an array. Find the entry where InstanceGroupType is MASTER. Make note of the Id.
"Id": "ig-2EHOYXFABCDEF",
STEP 5: List your cluster instances:
aws emr list-instances --cluster-id j-ABCDEFGHIJKLM
This will output JSON (I have truncated the output):
{
"Instances": [
....
{
"Id": "ci-31LGK4KIECHNY",
"Ec2InstanceId": "i-0524ec45912345678",
"PublicDnsName": "ec2-52-123-201-221.us-west-2.compute.amazonaws.com",
"PublicIpAddress": "52.123.201.221",
"PrivateDnsName": "ip-172-31-41-111.us-west-2.compute.internal",
"PrivateIpAddress": "172.31.41.111",
"Status": {
"State": "RUNNING",
"StateChangeReason": {},
"Timeline": {
"CreationDateTime": 1536626164.073,
"ReadyDateTime": 1536626533.886
}
},
"InstanceGroupId": "ig-2EHOYXFABCDEF",
"Market": "ON_DEMAND",
"InstanceType": "m3.xlarge",
"EbsVolumes": []
}
]
}
STEP 6: Find the matching InstanceGroupId ig-2EHOYXFABCDEF. This will give you the EC2 Instance ID for the MASTER: "Ec2InstanceId": "i-0524ec45912345678"
Step 7: Tag your EC2 instance:
aws ec2 create-tags --resources i-0524ec45912345678 --tags Key=EMR,Value=MASTER
The above steps might be simpler with CLI Filters and / or jq, but this should be enough information so that you know how to find and tag the EMR Master Instance.
Below can be used to directly get instance Id
aws emr list-instances --cluster-id ${aws_emr_cluster.cluster.id} --instance-
group-id ${aws_emr_cluster.cluster.master_instance_group.0.id} --query
'Instances[*].Ec2InstanceId' --output text
In an enviroinment where you does not have the aws cli, you can cat the following file:
cat /mnt/var/lib/info/job-flow.json
An example of the content is the following one:
{
"jobFlowId": "j-0000X0X0X00XX",
"jobFlowCreationInstant": 1579512208006,
"instanceCount": 2,
"masterInstanceId": "i-00x0xx0000xxx0x00",
"masterPrivateDnsName": "localhost",
"masterInstanceType": "m5.xlarge",
"slaveInstanceType": "m5.xlarge",
"hadoopVersion": "2.8.5",
"instanceGroups": [
{
"instanceGroupId": "ig-0XX00XX0X0XXX",
"instanceGroupName": "Master - 1",
"instanceRole": "Master",
"marketType": "OnDemand",
"instanceType": "m5.xlarge",
"requestedInstanceCount": 1
},
{
"instanceGroupId": "ig-000X0XXXXXXX",
"instanceGroupName": "Core - 2",
"instanceRole": "Core",
"marketType": "OnDemand",
"instanceType": "m5.xlarge",
"requestedInstanceCount": 1
}
]
NOTE: i've omitted the ID of the jobs using 0 where a number is expected and X where a ltter is expected.
You can do this programmatically in 3 lines of code, without having to copy-paste any of the specific information:
# get cluster id
CLUSTER_ID=$(aws emr list-clusters --active --query "Clusters[0].Id" --output text)
# get instance id
INSTANCE_ID=$(aws emr list-instances --cluster-id $CLUSTER_ID --instance-group-types MASTER --query "Instances[0].Ec2InstanceId" --output text)
# tag
aws ec2 create-tags --resources $INSTANCE_ID --tags Key=EMR,Value=MASTER
Below example is for Instance Fleet, it saves Cluster ID, Instance Fleet ID and Master IP as environment variables.
Replace cluster name "My-Cluster" to the actual cluster name.
export CLUSTER_ID=$(aws emr list-clusters --active --query 'Clusters[?Name==`My-Cluster`].Id' --output text)
export INSTANCE_FLEET=$(aws emr describe-cluster --cluster-id $CLUSTER_ID | jq -r '.[].InstanceFleets | .[] | select(.InstanceFleetType=="MASTER") | .Id')
export PRIVATE_IP=aws emr list-instances --cluster-id $CLUSTER_ID --instance-fleet-id $INSTANCE_FLEET --query 'Instances[*].PrivateIpAddress' --output text
"Cleanest" way:
aws emr list-clusters --active
Search for Master cluster ID (j-xxxxxxxxxxx), then
aws emr list-instances --region {your_region} --instance-group-types MASTER --cluster-id j-xxxxxxxxxxxxx
Immediately filters out the master instance(s) with it's information using --instance-group-types MASTER flag.
For tagging refers to the other answers with aws {resource} create-tags and --tag flag.

How to filter results from AWS CLI command

How can I filter the following so just results with IP starting with 10.* are returned?
aws ec2 describe-instances --filters "Name=tag-value,Values=mytagavalue" --query 'Reservations[*].Instances[*].{InstanceId:InstanceId,PrivateDnsName:PrivateDnsName,State:State.Name, IP:NetworkInterfaces[0].PrivateIpAddress}'
[
[
{
"InstanceId": "i-12345bnmsdfod",
"PrivateDnsName": "ip-10-34-24-4.my.there.com",
"State": "running",
"IP": "10.10.10.4"
}
],
[
{
"InstanceId": "i-12345bnmsdfop",
"PrivateDnsName": "",
"State": "terminated",
"IP": null
}
],
Option 1) Via Filters
Use the network-interface.addresses.private-ip-address filter to select values matching only "10.*", which will match addresses starting with "10.".
--filters "Name=network-interface.addresses.private-ip-address,Values=10.*"
Simply include a space between different filters to delimit them.
Full Example
aws ec2 describe-instances --filters "Name=tag-value,Values=mytagavalue" "Name=network-interface.addresses.private-ip-address,Values=10.*" --query 'Reservations[*].Instances[*].{InstanceId:InstanceId,PrivateDnsName:PrivateDnsName,State:State.Name, IP:NetworkInterfaces[0].PrivateIpAddress}'
Option 2) Via Query
Use the JMESPath starts_with() function to perform a partial string comparison of "10." against each network interface's private IP address.
Step By Step
First, select all instances:
Reservations[].Instances[]
Then pipe to filter for only instances containing network interfaces that have a private ip address starting with "10.":
| [? NetworkInterfaces [? starts_with(PrivateIpAddress, '10.')]]
Then select the fields just like you did before. This has not changed. (Note that you may want to select for all network interfaces instead of just the first.)
.{InstanceId:InstanceId,PrivateDnsName:PrivateDnsName,State:State.Name, IP:NetworkInterfaces[0].PrivateIpAddress}"
Full Example
aws ec2 describe-instances --filters "Name=tag-value,Values=mytagavalue" --query "Reservations[].Instances[] | [? NetworkInterfaces [? starts_with(PrivateIpAddress, '10.')]].{InstanceId:InstanceId,PrivateDnsName:PrivateDnsName,State:State.Name, IP:NetworkInterfaces[0].PrivateIpAddress}"
Further Reading
JMESPath Documentation - starts_with()
AWS Documentation - Controlling Command Output from the AWS Command Line Interface
AWS Documentation - Listing and Filtering Using the CLI and API

How to extract a particular Key-Value Tag from ec2 describe-instances

I've got the following from describe-instances:
{
"Reservations": [
{
"Instances": [
{
"PublicDnsName": "ec2-xxxxx.amazonaws.com",
"Tags": [
{
"Key": "Name",
"Value": "yyyyy"
},
{
"Key": "budget_cluster",
"Value": "zzzzz"
},
{
"Key": "poc",
"Value": "aaaaaaa"
}
]
}
]
}
]
}
For each instance, I would like to extract the PublicDnsName and the value of the "budget_cluster" tag key. How to do this either with ec2 describe-instances or with jq ?
Modifying Frédéric's answer:
aws ec2 describe-instances --output text --query \
'Reservations[].Instances[].[PublicDnsName, Tags[?Key==`budget_cluster`].Value | [0]]'
Would produce:
ec2-xxxxx.amazonaws.com zzzzz
ec2-bbbbb.amazonaws.com yyyyy
I've changed the output to text, which removes as much formatting as possible and selected the individual tag value with | [0] since there will only ever be one per instance anyway. Finally, I removed the [] at the end so that the resulting list isn't flattened. That way in text output each entry will be on its own line.
You can also make this more robust by only selecting instances that actually have that tag. You could do so with further modifications to the --query parameter, but it is better in this case to use the --filters parameter since it does service-side filtering. Specifically you want the tag-key filter: --filters "Name=tag-key,Values=budget_cluster"
aws ec2 describe-instances --output text \
--filters "Name=tag-key,Values=budget_cluster" --query \
'Reservations[].Instances[?Tags[?Key==`budget_cluster`]].[PublicDnsName, Tags[?Key==`budget_cluster`].Value | [0]]'
Would still produce:
ec2-xxxxx.amazonaws.com zzzzz
ec2-bbbbb.amazonaws.com yyyyy
But over the wire you would only be getting the instances you care about, thus saving money on bandwidth.
Using jq 1.5 or later, the simplest approach is to use from_entries.
After a minimal fix of the illustrative input, the following invocation:
$ jq '.Reservations[]
| .Instances[]
| [.PublicDnsName, (.Tags|from_entries|.budget_cluster)]' input.json
produces:
[
"ec2-xxxxx.amazonaws.com",
"zzzzz"
]
If you do not have jq 1.5 or later, the following should work:
$ jq1.4 '.Reservations[]
| .Instances[]
| [.PublicDnsName, (.Tags[]|select(.Key=="budget_cluster") | .Value)]' input.json
The answer of #peak is great and I keep learning from him on the jq part but again you can achieve quite a lot from the AWS CLI
aws ec2 describe-instances --query \
'Reservations[].Instances[].{PublicDnsName:PublicDnsName, Budget:Tags[?Key==`budget_cluster`].Value}'
would produce
[
{
"PublicDnsName": "ec2-xxxxx.amazonaws.com",
"Budget": [
"zzzz",
]
}
]
If you do not want to make a new strict JSon out of it, just take the values
aws ec2 describe-instances --query \
'Reservations[].Instances[].[PublicDnsName, Tags[?Key==`budget_cluster`].Value][]'
will produce
[
"ec2-xxxxx.amazonaws.com",
[
"zzzz"
]
]