How to see all running Amazon EC2 instances across all regions? - amazon-web-services

I switch instances between different regions frequently and sometimes I forget to turn off my running instance from a different region. I couldn't find any way to see all the running instances on Amazon console.
Is there any way to display all the running instances regardless of region?

Nov 2021 Edit: AWS has recently launched the Amazon EC2 Global View with initial support for Instances, VPCs, Subnets, Security Groups and Volumes.
See the announcement or documentation for more details
A non-obvious GUI option is the Tag Editor in the Resource Groups console. Here you can find all instances across all regions, even if the instances were not tagged.

I don't think you can currently do this in the AWS GUI. But here is a way to list all your instances across all regions with the AWS CLI:
for region in `aws ec2 describe-regions --region us-east-1 --output text | cut -f4`
do
echo -e "\nListing Instances in region:'$region'..."
aws ec2 describe-instances --region $region
done
Taken from here (If you want to see full discussion)
Also, if you're getting a
You must specify a region. You can also configure your region by running "aws configure"
You can do so with aws configure set region us-east-1, thanks #Sabuncu for the comment.
Update
Now (in 2019) the cut command should be applied on the 4th field: cut -f4

In Console
Go to VPC dashboard https://console.aws.amazon.com/vpc/home and click on Running instances -> See all regions.
In CLI
Add this for example to .bashrc. Reload it source ~/.bashrc, and run it
Note: Except for aws CLI you need to have jq installed
function aws.print-all-instances() {
REGIONS=`aws ec2 describe-regions --region us-east-1 --output text --query Regions[*].[RegionName]`
for REGION in $REGIONS
do
echo -e "\nInstances in '$REGION'..";
aws ec2 describe-instances --region $REGION | \
jq '.Reservations[].Instances[] | "EC2: \(.InstanceId): \(.State.Name)"'
done
}
Example output:
$ aws.print-all-instances
Listing Instances in region: 'eu-north-1'..
"EC2: i-0548d1de00c39f923: terminated"
"EC2: i-0fadd093234a1c21d: running"
Listing Instances in region: 'ap-south-1'..
Listing Instances in region: 'eu-west-3'..
Listing Instances in region: 'eu-west-2'..
Listing Instances in region: 'eu-west-1'..
Listing Instances in region: 'ap-northeast-2'..
Listing Instances in region: 'ap-northeast-1'..
Listing Instances in region: 'sa-east-1'..
Listing Instances in region: 'ca-central-1'..
Listing Instances in region: 'ap-southeast-1'..
Listing Instances in region: 'ap-southeast-2'..
Listing Instances in region: 'eu-central-1'..
Listing Instances in region: 'us-east-1'..
Listing Instances in region: 'us-east-2'..
Listing Instances in region: 'us-west-1'..
Listing Instances in region: 'us-west-2'..

From VPC Dashboard:
First go to VPC Dashboard
Then find Running instances and expand see all regions. Here you can find all the running instances of all region:
From EC2 Global view:
Also you can use AWS EC2 Global View to watch Resource summary
and Resource counts per Region.

#imTachu solution works well. To do this via the AWS console...
AWS console
Services
Networking & Content Delivery
VPC
Look for a block named "Running Instances", this will show you the current region
Click the "See all regions" link underneath

Every time you create a resource, tag it with a name and now you can use Resource Groups to find all types of resources with a name tag across all regions.

After reading through all the solutions and trying bunch of stuff, the one that worked for me was-
List item
Go to Resource Group
Tag Editor
Select All Regions
Select EC2 Instance in resource type
Click Search Resources

Based on imTachus answer but less verbose, plus faster. You need to have jq and aws-cli installed.
set +m
for region in $(aws ec2 describe-regions --query "Regions[*].[RegionName]" --output text); do
aws ec2 describe-instances --region "$region" | jq ".Reservations[].Instances[] | {type: .InstanceType, state: .State.Name, tags: .Tags, zone: .Placement.AvailabilityZone}" &
done; wait; set -m
The script runs the aws ec2 describe-instances in parallel for each region (now 15!) and extracts only the relevant bits (state, tags, availability zone) from the json output. The set +m is needed so the background processes don't report when starting/ending.
Example output:
{
"type": "t2.micro",
"state": "stopped",
"tags": [
{
"Key": "Name",
"Value": "MyEc2WebServer"
},
],
"zone": "eu-central-1b"
}

You can run DescribeInstances() across all regions.
Additionally, you can:
Automate it through Lambda and Cloud watch.
Create api endpoint using Lambda and api gateway and use it in your code
A sample in NodeJS:
Create and array of regions (endpoints). [can also use AWS describeRegions() ]
var regionNames = ['us-west-1', 'us-west-2', 'us-east-1', 'eu-west-1', 'eu-central-1', 'sa-east-1', 'ap-southeast-1', 'ap-southeast-2', 'ap-northeast-1', 'ap-northeast-2'];
regionNames.forEach(function(region) {
getInstances(region);
});
Then, in getInstances function, DescribeInstances() can be
called.
function getInstances(region) {
EC2.describeInstances(params, function(err, data) {
if (err) return console.log("Error connecting to AWS, No Such Instance Found!");
data.Reservations.forEach(function(reservation) {
//do any operation intended
});
}
And Off Course, feel free to use ES6 and above.
I wrote a lambda function to get you all the instances in any state [running, stopped] and from any regions, will also give details about instance type and various other parameters.
The Script runs across all AWS regions and calls DescribeInstances(), to get the instances.
You just need to create a lambda function with run-time nodejs.
You can even create API out of it and use it as and when required.
Additionally, You can see AWS official Docs For DescribeInstances to explore many more options.

A quick bash oneliner command to print all the instance IDs in all regions:
$ aws ec2 describe-regions --query "Regions[].{Name:RegionName}" --output text |xargs -I {} aws ec2 describe-instances --query Reservations[*].Instances[*].[InstanceId] --output text --region {}
# Example output
i-012344b918d75abcd
i-0156780dad25fefgh
i-0490122cfee84ijkl
...

My script below, based on various tips from this post and elsewhere. The script is easier to follow (for me at least) than the long command lines.
The script assumes credential profile(s) are stored in file ~/.aws/credentials looking something like:
[default]
aws_access_key_id = foobar
aws_secret_access_key = foobar
[work]
aws_access_key_id = foobar
aws_secret_access_key = foobar
Script:
#!/usr/bin/env bash
#------------------------------------#
# Script to display AWS EC2 machines #
#------------------------------------#
# NOTES:
# o Requires 'awscli' tools (for ex. on MacOS: $ brew install awscli)
# o AWS output is tabbed - we convert to spaces via 'column' command
#~~~~~~~~~~~~~~~~~~~~#
# Assemble variables #
#~~~~~~~~~~~~~~~~~~~~#
regions=$(aws ec2 describe-regions --output text | cut -f4 | sort)
query_mach='Reservations[].Instances[]'
query_flds='PrivateIpAddress,InstanceId,InstanceType'
query_tags='Tags[?Key==`Name`].Value[]'
query_full="$query_mach.[$query_flds,$query_tags]"
#~~~~~~~~~~~~~~~~~~~~~~~~#
# Output AWS information #
#~~~~~~~~~~~~~~~~~~~~~~~~#
# Iterate through credentials profiles
for profile in 'default' 'work'; do
# Print profile header
echo -e "\n"
echo -e "~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~"
echo -e "Credentials profile:'$profile'..."
echo -e "~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~"
# Iterate through all regions
for region in $regions; do
# Print region header
echo -e "\n"
echo -e "Region: $region..."
echo -e "--------------------------------------------------------------"
# Output items for the region
aws ec2 describe-instances \
--profile $profile \
--region $region \
--query $query_full \
--output text \
| sed 's/None$/None\n/' \
| sed '$!N;s/\n/ /' \
| column -t -s $'\t'
done
done

AWS has recently launched the Amazon EC2 Global View with initial support for Instances, VPCs, Subnets, Security Groups, and Volumes.
To see all running instances go to EC2 or VPC console and click EC2 Global View in the top left corner.
Then click on Global Search tab and filter by Resource type and select Instance. Unfortunately, this will show instances in all states:
pending
running
stopping
stopped
shutting-down
terminated

I created an open-source script that helps you to list all AWS instances. https://github.com/Appnroll/aws-ec2-instances
That's a part of the script that lists the instances for one profile recording them into an postgreSQL database with using jq for json parsing:
DATABASE="aws_instances"
TABLE_NAME="aws_ec2"
SAVED_FIELDS="state, name, type, instance_id, public_ip, launch_time, region, profile, publicdnsname"
# collects the regions to display them in the end of script
REGIONS_WITH_INSTANCES=""
for region in `aws ec2 describe-regions --output text | cut -f3`
do
# this mappping depends on describe-instances command output
INSTANCE_ATTRIBUTES="{
state: .State.Name,
name: .KeyName, type: .InstanceType,
instance_id: .InstanceId,
public_ip: .NetworkInterfaces[0].Association.PublicIp,
launch_time: .LaunchTime,
\"region\": \"$region\",
\"profile\": \"$AWS_PROFILE\",
publicdnsname: .PublicDnsName
}"
echo -e "\nListing AWS EC2 Instances in region:'$region'..."
JSON=".Reservations[] | ( .Instances[] | $INSTANCE_ATTRIBUTES)"
INSTANCE_JSON=$(aws ec2 describe-instances --region $region)
if echo $INSTANCE_JSON | jq empty; then
# "Parsed JSON successfully and got something other than false/null"
OUT="$(echo $INSTANCE_JSON | jq $JSON)"
# check if empty
if [[ ! -z "$OUT" ]] then
for row in $(echo "${OUT}" | jq -c "." ); do
psql -c "INSERT INTO $TABLE_NAME($SAVED_FIELDS) SELECT $SAVED_FIELDS from json_populate_record(NULL::$TABLE_NAME, '${row}') ON CONFLICT (instance_id)
DO UPDATE
SET state = EXCLUDED.state,
name = EXCLUDED.name,
type = EXCLUDED.type,
launch_time = EXCLUDED.launch_time,
public_ip = EXCLUDED.public_ip,
profile = EXCLUDED.profile,
region = EXCLUDED.region,
publicdnsname = EXCLUDED.publicdnsname
" -d $DATABASE
done
REGIONS_WITH_INSTANCES+="\n$region"
else
echo "No instances"
fi
else
echo "Failed to parse JSON, or got false/null"
fi
done

To run jobs in parallel and use multiple profiles use this script.
#!/bin/bash
for i in profile1 profile2
do
OWNER_ID=`aws iam get-user --profile $i --output text | awk -F ':' '{print $5}'`
tput setaf 2;echo "Profile : $i";tput sgr0
tput setaf 2;echo "OwnerID : $OWNER_ID";tput sgr0
for region in `aws --profile $i ec2 describe-regions --output text | cut -f4`
do
tput setaf 1;echo "Listing Instances in region $region";tput sgr0
aws ec2 describe-instances --query 'Reservations[*].Instances[*].[Tags[?Key==`Name`].Value , InstanceId]' --profile $i --region $region --output text
done &
done
wait
Screenshot:

Not sure how long this option's been here, but you can see a global view of everything by searching for EC2 Global View
https://console.aws.amazon.com/ec2globalview/home#

Using bash-my-aws:
region-each instances

Based on #hansaplast code I created Windows friendly version that supports multiple profiles as an argument. Just save that file as cmd or bat file. You also need to have jq command.
#echo off
setlocal enableDelayedExpansion
set PROFILE=%1
IF "%1"=="" (SET PROFILE=default)
echo checkin instances in all regions for %PROFILE% account
FOR /F "tokens=* USEBACKQ" %%F IN (`aws ec2 describe-regions --query Regions[*].[RegionName] --output text --profile %PROFILE%`) DO (
echo === region: %%F
aws ec2 describe-instances --region %%F --profile %PROFILE%| jq ".Reservations[].Instances[] | {type: .InstanceType, state: .State.Name, tags: .Tags, zone: .Placement.AvailabilityZone}"
)

You may use cli tool designed for enumerating cloud resources (cross-region and cross-accounts scan) - https://github.com/scopely-devops/skew
After short configuration you may use the following code for list all instances in all US AWS regions (assuming 123456789012 is your AWS account number).
from skew import scan
arn = scan('arn:aws:ec2:us-*:123456789012:instance/*')
for resource in arn:
print(resource.data)

Good tool to CRUD AWS resources. Find [EC2|RDS|IAM..] in all regions. There can do operations (stop|run|terminate) on filters results.
python3 awsconsole.py ec2 all // return list of all instances
python3 awsconsole.py ec2 all -r eu-west-1
python3 awsconsole.py ec2 find -i i-0552e09b7a54fa2cf --[terminate|start|stop]

Related

Terraform throwing this error Getting Extra characters after interpolation expression;

I'm trying to associate the Elastic IP address with Auto scaling group, so whenever the autoscaling triggers it will automatically associate with the EIP.
For this I'm trying to add the script in user data.
My intention is to we have 2 servers so its associated with 2 EIP's, whenever the autoscaling triggers it has to check whether the EIP is free or not if its free it has to associate with that instance using the instance id.
Below is my script where I'm getting the error
In EIP_LIST im getting this error Extra characters after interpolation expression; Expected a closing brace to end the interpolation expression, but found extra characters.
INSTANCE_ID=$(ec2-metadata --instance-id | cut -d " " -f 2);
MAXWAIT=10
# Get list of EIPs
EIP_LIST=${"eipalloc-09e7274dd3c641ae6" "eipalloc-05e8e926926f9de55"}
# Iterate over EIP list
for EIP in ${EIP_LIST}; do
echo "Checking if EIP with ALLOC_ID[$EIP] is free...."
ISFREE=$(aws ec2 describe-addresses --allocation-ids $EIP --query Addresses[].InstanceId --output text --region ap-south-1)
STARTWAIT=$(date +%s)
while [ ! -z "$ISFREE" ]; do
if [ "$(($(date +%s) - $STARTWAIT))" -gt $MAXWAIT ]; then
echo "WARNING: We waited 30 seconds, we're forcing it now."
ISFREE=""
else
echo "Waiting for EIP with ALLOC_ID[$EIP] to become free...."
sleep 3
ISFREE=$(aws ec2 describe-addresses --allocation-ids $EIP --query Addresses[].InstanceId --output text --region ap-south-1)
fi
done
echo Running: aws ec2 associate-address --instance-id $INSTANCE_ID --allocation-id $EIP --allow-reassociation --region ap-south-1
aws ec2 associate-address --instance-id $INSTANCE_ID --allocation-id $EIP --allow-reassociation --region ap-south-1
$ is TF symbol used for interpolation of variables. It clashes with the same symbol used in bash. You have to escape it using $$ if you want to us $ in bash, not in TF, e.g.
$${EIP_LIST}
will result in ${EIP_LIST} in your script.

Is there any way to figure out public ip belongs to which aws account?

I have multiple aws accounts and i don't remember in which aws account this EC2 instance was created, is there any optimal way to figure out in very less time?
Note: i need to know account DNS name or Alias name.(Not account number)
If you have access to the instance you could use Instance metadata API:
[ec2-user ~]$ curl http://169.254.169.254/latest/dynamic/instance-identity/document
It returns json with accountId field.
If you configure AWS CLI for all account, then you can get the Account ID, ARN and user ID.
The script does the following.
Get the list of AWS configuration profile
Loop over all profile
Get a list of All Ec2 public IP address
print account info if IP matched and exit
RUN
./script.sh 52.x.x.x
script.sh
#!/bin/bash
INSTANCE_IP="${1}"
if [ -z "${INSTANCE_IP}" ]; then
echo "pls provide instance IP"
echo "./scipt.sh 54.x.x.x"
exit 1
fi
PROFILE_LIST=$(grep -o "\\[[^]]*]" < ~/.aws/credentials | tr -d "[]")
for PROFILE in $PROFILE_LIST; do
ALL_IPS=$(aws ec2 describe-instances --profile "${PROFILE}" --query "Reservations[].Instances[][PublicIpAddress]" --output text | tr '\r\n' ' ')
echo "looking against profile ${PROFILE}"
for IP in $ALL_IPS; do
if [ "${INSTANCE_IP}" == "${IP}" ]; then
echo "Instance IP matched in below account"
aws sts get-caller-identity
exit 0
fi
done
done
echo "seems like instance not belong to these profile"
echo "${PROFILE_LIST}"
exit 1
loop over accounts
loop over regions
also be aware of lightsail!
I came up with the following and helped me. I didn't exclude the regions that did not have lightsail
for region in `aws ec2 describe-regions --output text --query 'Regions[*].[RegionName]' --region eu-west-1` ; do \
echo $region; \
aws ec2 describe-network-interfaces --output text --filters Name=addresses.private-ip-address,Values="IPv4 address" --region $region ; \
aws lightsail get-instances --region eu-west-1 --output text --query 'instances[*].[name,publicIpAddress]' --region $region; \
done

How can I call file using AWS CLI

I am looking forward to call \File.txt using below AWS CLI , if you have exp please share it.
aws ec2 describe-instances --filters Name=network-interface.group-name,Values='\file.txt' --query 'Reservations[].Instances[].[Placement.AvailabilityZone, State.Name, InstanceId,Tags[?Key==Name].Value|[0]]'
File.txt includes Security Group Names.
Here is a way you can do it, I'm assuming Security Group Name is on a separate line.
Command:
$(cat file.txt|sed 'N;s/\n/,/')
Input: file.txt:
cat file.txt
security-group-name-1
security-group-name-2
security-group-name-3
Output:
security-group-name-1,security-group-name-2
Full command:
aws ec2 describe-instances --filters Name=network-interface.group-name,Values=$(cat file.txt|sed 'N;s/\n/,/') --query 'Reservations[].Instances[].[Placement.AvailabilityZone, State.Name, InstanceId,Tags[?Key==Name].Value|[0]]'

How to delete untagged images from AWS ECR Container Registry

When pushing images to Amazon ECR, if the tag already exists within the repo the old image remains within the registry but goes in an untagged state.
So if i docker push image/haha:1.0.0 the second time i do this (provided that something changes) the first image gets untagged from AWS ECR.
Is there a way to safely clean up all the registries from untagged images?
You can delete all images in a single request, without loops:
IMAGES_TO_DELETE=$( aws ecr list-images --region $ECR_REGION --repository-name $ECR_REPO --filter "tagStatus=UNTAGGED" --query 'imageIds[*]' --output json )
aws ecr batch-delete-image --region $ECR_REGION --repository-name $ECR_REPO --image-ids "$IMAGES_TO_DELETE" || true
First it gets a list of images that are untagged, in json format:
[ {"imageDigest": "sha256:..."}, {"imageDigest": "sha256:..."}, ... ]
Then it sends that list to batch-image-delete.
The last || true is required to avoid an error code when there are no untagged images.
Now, that ECR support lifecycle policies (https://docs.aws.amazon.com/AmazonECR/latest/userguide/LifecyclePolicies.html) you can use it to delete the untagged images automatically.
Setting up a lifecycle policy preview using the console
Open the Amazon ECS console at https://console.aws.amazon.com/ecs/.
From the navigation bar, choose the region that contains the
repository on which to perform a lifecycle policy preview.
In the navigation pane, choose Repositories and select a repository.
On the All repositories: repository_name page, choose Dry-Run
Lifecycle Rules, Add.
Enter the following details for your lifecycle policy rule:
For Rule Priority, type a number for the rule priority.
For Rule Description, type a description for the lifecycle policy
rule.
For Image Status, choose either Tagged or Untagged.
If you specified Tagged for Image Status, then for Tag Prefix List,
you can optionally specify a list of image tags on which to take
action with your lifecycle policy. If you specified Untagged, this
field must be empty.
For Match criteria, choose values for Count Type, Count Number, and
Count Unit (if applicable).
Choose Save
Create additional lifecycle policy rules by repeating steps 5–7.
To run the lifecycle policy preview, choose Save and preview results.
Under Preview Image Results, review the impact of your lifecycle
policy preview.
If you are satisfied with the preview results, choose Apply as
lifecycle policy to create a lifecycle policy with the specified
rules.
From here:
https://docs.aws.amazon.com/AmazonECR/latest/userguide/lpp_creation.html
I actually forged a one line solution using aws cli
aws ecr describe-repositories --output text | awk '{print $5}' | egrep -v '^$' | while read line; do repo=$(echo $line | sed -e "s/arn:aws:ecr.*\///g") ; aws ecr list-images --repository-name $repo --filter tagStatus=UNTAGGED --query 'imageIds[*]' --output text | while read imageId; do aws ecr batch-delete-image --repository-name $repo --image-ids imageDigest=$imageId; done; done
What it's doing is:
get all repositories
for each repository give me all images with tagStatus=UNTAGGED
for each image+repo issue a batch-delete-image
If you have JQ, you can use this version that is more robust by not relying on the changing text format and also more efficient as it batch deletes once per repository:
aws ecr describe-repositories \
| jq --raw-output .repositories[].repositoryName \
| while read repo; do
imageIds=$(aws ecr list-images --repository-name $repo --filter tagStatus=UNTAGGED --query 'imageIds[*]' --output json | jq -r '[.[].imageDigest] | map("imageDigest="+.) | join (" ")');
if [[ "$imageIds" == "" ]]; then continue; fi
aws ecr batch-delete-image --repository-name $repo --image-ids $imageIds;
done
This has been broken up into more lines for readability, so better put it into a function in your .bashrc, but you could of course stuff it into a single line:
aws ecr describe-repositories | jq --raw-output .repositories[].repositoryName | while read repo; do imageIds=$(aws ecr list-images --repository-name $repo --filter tagStatus=UNTAGGED --query 'imageIds[*]' --output json | jq -r '[.[].imageDigest] | map("imageDigest="+.) | join (" ")'); if [[ "$imageIds" == "" ]]; then continue; fi; aws ecr batch-delete-image --repository-name $repo --image-ids $imageIds; done
Setting a Lifecycle policy is definitely the best way of managing this. That being said - if you do have a bunch of images that you want to delete keep in mind that the max for batch-delete-images is 100. So you need to do this is for the number of untagged images is greater than 100:
IMAGES_TO_DELETE=$( aws ecr list-images --repository-name $ECR_REPO --filter "tagStatus=UNTAGGED" --query 'imageIds[0:100]' --output json )
echo $IMAGES_TO_DELETE | jq length # Gets the number of results
aws ecr batch-delete-image --repository-name $ECR_REPO --image-ids "$IMAGES_TO_DELETE" --profile qa || true
If you want to remove an untagged image from a repository you can simply create a JSON lifecycle policy and then use python to apply the JSON policy to the repo
In my case, I am applying the policy to all the ECR repositories that are there in ECR and I have created a "lifecyclepolicy.json" file in my current directory where I have added the lifecycle policy of ECR
Here is my python code:-
import os
import json
import boto3
def ecr_lifecycle(lifecycle_policy):
ecr_client = boto3.client('ecr')
repositories = []
describe_repo_paginator = ecr_client.get_paginator('describe_repositories')
for response_list_repopaginator in describe_repo_paginator.paginate():
for repo in response_list_repopaginator['repositories']:
repositories.append(repo['repositoryName'])
for repository in repositories:
response=ecr_client.put_lifecycle_policy(repositoryName=repository,
lifecyclePolicyText=json.dumps(lifecycle_policy))
return response
if __name__ == '__main__':
path = os.path.dirname(__file__)
json_file = open(os.path.join(path, 'lifecyclepolicy.json'))
data = json.load(json_file)
ecr_lifecycle(data)
If you want to see the JSON file:-
{
"rules": [
{
{
"rulePriority": 10,
"description": "Only keep untagged images for 7 days",
"selection": {
"tagStatus": "untagged",
"countType": "sinceImagePushed",
"countUnit": "days",
"countNumber": 7
}
"action": {
"type": "expire"
}
}
]
}
Base on #Ken J's anwer,
Here is a python script that will clean ALL your ECR:
#!/usr/bin/python3
import subprocess
import json
import os
# Based on: https://stackoverflow.com/questions/40949342/how-to-delete-untagged-images-from-aws-ecr-container-registry
region="us-east-1"
debug = False
def _runCommand(command):
if debug:
print(" ".join(command))
p = subprocess.Popen(command, shell = False, stdout = subprocess.PIPE, stderr = subprocess.PIPE)
return [p.stdout.read().decode("utf-8"), p.stderr.read().decode("utf-8")]
command = "aws ecr describe-repositories --region " + region + " --output json".split(" ")
data = _runCommand(command)[0]
for i in json.loads(data)["repositories"]:
name = i["repositoryName"]
print(name)
command = ["aws", "ecr", "list-images", "--region", region, "--repository-name", name, "--filter", "tagStatus=UNTAGGED", "--query", 'imageIds[*]', "--output" , "json"]
data = _runCommand(command)[0]
command = ["aws", "ecr", "batch-delete-image", "--region", region, "--repository-name", name, "--image-ids",data]
data = _runCommand(command)[0]
print(data)
First Step -->
untaggedImages = aws ecr list-images --repository-name <your_repo_name> --filter "tagStatus=UNTAGGED" --query 'to_string(imageIds[*])' --output json""")
Second step -->
aws ecr batch-delete-image --repository-name <your_repo_name> --image-ids "$untaggedImages" || true """)
to_string function is required because the returned JSON won't be in string format, instead it will be as an Object.

Query EC2 tags from within instance

Amazon recently added the wonderful feature of tagging EC2 instances with key-value pairs to make management of large numbers of VMs a bit easier.
Is there some way to query these tags in the same way as some of the other user-set data? For example:
$ curl http://169.254.169.254/latest/meta-data/placement/availability-zone
us-east-1d
Is there some similar way to query the tags?
The following bash script returns the Name of your current ec2 instance (the value of the "Name" tag). Modify TAG_NAME to your specific case.
TAG_NAME="Name"
INSTANCE_ID="`wget -qO- http://instance-data/latest/meta-data/instance-id`"
REGION="`wget -qO- http://instance-data/latest/meta-data/placement/availability-zone | sed -e 's:\([0-9][0-9]*\)[a-z]*\$:\\1:'`"
TAG_VALUE="`aws ec2 describe-tags --filters "Name=resource-id,Values=$INSTANCE_ID" "Name=key,Values=$TAG_NAME" --region $REGION --output=text | cut -f5`"
To install the aws cli
sudo apt-get install python-pip -y
sudo pip install awscli
In case you use IAM instead of explicit credentials, use these IAM permissions:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [ "ec2:DescribeTags"],
"Resource": ["*"]
}
]
}
Once you've got ec2-metadata and ec2-describe-tags installed (as mentioned in Ranieri's answer above), here's an example shell command to get the "name" of the current instance, assuming you have a "Name=Foo" tag on it.
Assumes EC2_PRIVATE_KEY and EC2_CERT environment variables are set.
ec2-describe-tags \
--filter "resource-type=instance" \
--filter "resource-id=$(ec2-metadata -i | cut -d ' ' -f2)" \
--filter "key=Name" | cut -f5
This returns Foo.
You can use a combination of the AWS metadata tool (to retrieve your instance ID) and the new Tag API to retrieve the tags for the current instance.
You can add this script to your cloud-init user data to download EC2 tags to a local file:
#!/bin/sh
INSTANCE_ID=`wget -qO- http://instance-data/latest/meta-data/instance-id`
REGION=`wget -qO- http://instance-data/latest/meta-data/placement/availability-zone | sed 's/.$//'`
aws ec2 describe-tags --region $REGION --filter "Name=resource-id,Values=$INSTANCE_ID" --output=text | sed -r 's/TAGS\t(.*)\t.*\t.*\t(.*)/\1="\2"/' > /etc/ec2-tags
You need the AWS CLI tools installed on your system: you can either install them with a packages section in a cloud-config file before the script, use an AMI that already includes them, or add an apt or yum command at the beginning of the script.
In order to access EC2 tags you need a policy like this one in your instance's IAM role:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "Stmt1409309287000",
"Effect": "Allow",
"Action": [
"ec2:DescribeTags"
],
"Resource": [
"*"
]
}
]
}
The instance's EC2 tags will available in /etc/ec2-tags in this format:
FOO="Bar"
Name="EC2 tags with cloud-init"
You can include the file as-is in a shell script using . /etc/ec2-tags, for example:
#!/bin/sh
. /etc/ec2-tags
echo $Name
The tags are downloaded during instance initialization, so they will not reflect subsequent changes.
The script and IAM policy are based on itaifrenkel's answer.
If you are not in the default availability zone the results from overthink would return empty.
ec2-describe-tags \
--region \
$(curl -s http://169.254.169.254/latest/meta-data/placement/availability-zone | sed -e "s/.$//") \
--filter \
resource-id=$(curl --silent http://169.254.169.254/latest/meta-data/instance-id)
If you want to add a filter to get a specific tag (elasticbeanstalk:environment-name in my case) then you can do this.
ec2-describe-tags \
--region \
$(curl -s http://169.254.169.254/latest/meta-data/placement/availability-zone | sed -e "s/.$//") \
--filter \
resource-id=$(curl --silent http://169.254.169.254/latest/meta-data/instance-id) \
--filter \
key=elasticbeanstalk:environment-name | cut -f5
And to get only the value for the tag that I filtered on, we pipe to cut and get the fifth field.
ec2-describe-tags \
--region \
$(curl -s http://169.254.169.254/latest/meta-data/placement/availability-zone | sed -e "s/.$//") \
--filter \
resource-id=$(curl --silent http://169.254.169.254/latest/meta-data/instance-id) \
--filter \
key=elasticbeanstalk:environment-name | cut -f5
You can alternatively use the describe-instances cli call rather than describe-tags:
This example shows how to get the value of tag 'my-tag-name' for the instance:
aws ec2 describe-instances \
--instance-id $(curl -s http://169.254.169.254/latest/meta-data/instance-id) \
--query "Reservations[*].Instances[*].Tags[?Key=='my-tag-name'].Value" \
--region ap-southeast-2 --output text
Change the region to suit your local circumstances. This may be useful where your instance has the describe-instances privilege but not describe-tags in the instance profile policy
I have pieced together the following that is hopefully simpler and cleaner than some of the existing answers and uses only the AWS CLI and no additional tools.
This code example shows how to get the value of tag 'myTag' for the current EC2 instance:
Using describe-tags:
export AWS_DEFAULT_REGION=us-east-1
instance_id=$(curl -s http://169.254.169.254/latest/meta-data/instance-id)
aws ec2 describe-tags \
--filters "Name=resource-id,Values=$instance_id" 'Name=key,Values=myTag' \
--query 'Tags[].Value' --output text
Or, alternatively, using describe-instances:
aws ec2 describe-instances --instance-id $instance_id \
--query 'Reservations[].Instances[].Tags[?Key==`myTag`].Value' --output text
For Python:
from boto import utils, ec2
from os import environ
# import keys from os.env or use default (not secure)
aws_access_key_id = environ.get('AWS_ACCESS_KEY_ID', failobj='XXXXXXXXXXX')
aws_secret_access_key = environ.get('AWS_SECRET_ACCESS_KEY', failobj='XXXXXXXXXXXXXXXXXXXXX')
#load metadata , if = {} we are on localhost
# http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AESDG-chapter-instancedata.html
instance_metadata = utils.get_instance_metadata(timeout=0.5, num_retries=1)
region = instance_metadata['placement']['availability-zone'][:-1]
instance_id = instance_metadata['instance-id']
conn = ec2.connect_to_region(region, aws_access_key_id=aws_access_key_id, aws_secret_access_key=aws_secret_access_key)
# get tag status for our instance_id using filters
# http://docs.aws.amazon.com/AWSEC2/latest/CommandLineReference/ApiReference-cmd-DescribeTags.html
tags = conn.get_all_tags(filters={'resource-id': instance_id, 'key': 'status'})
if tags:
instance_status = tags[0].value
else:
instance_status = None
logging.error('no status tag for '+region+' '+instance_id)
A variation on some of the answers above but this is how I got the value of a specific tag from the user-data script on an instance
REGION=$(curl http://instance-data/latest/meta-data/placement/availability-zone | sed 's/.$//')
INSTANCE_ID=$(curl -s http://instance-data/latest/meta-data/instance-id)
TAG_VALUE=$(aws ec2 describe-tags --region $REGION --filters "Name=resource-id,Values=$INSTANCE_ID" "Name=key,Values='<TAG_NAME_HERE>'" | jq -r '.Tags[].Value')
Starting January 2022, this should be also available directly via ec2 metadata api (if enabled).
curl http://169.254.169.254/latest/meta-data/tags/instance
https://aws.amazon.com/about-aws/whats-new/2022/01/instance-tags-amazon-ec2-instance-metadata-service/
Using the AWS 'user data' and 'meta data' APIs its possible to write a script which wraps puppet to start a puppet run with a custom cert name.
First start an aws instance with custom user data: 'role:webserver'
#!/bin/bash
# Find the name from the user data passed in on instance creation
USER=$(curl -s "http://169.254.169.254/latest/user-data")
IFS=':' read -ra UDATA <<< "$USER"
# Find the instance ID from the meta data api
ID=$(curl -s "http://169.254.169.254/latest/meta-data/instance-id")
CERTNAME=${UDATA[1]}.$ID.aws
echo "Running Puppet for certname: " $CERTNAME
puppet agent -t --certname=$CERTNAME
This calls puppet with a certname like 'webserver.i-hfg453.aws' you can then create a node manifest called 'webserver' and puppets 'fuzzy node matching' will mean it is used to provision all webservers.
This example assumes you build on a base image with puppet installed etc.
Benefits:
1) You don't have to pass round your credentials
2) You can be as granular as you like with the role configs.
Jq + ec2metadata makes it a little nicer. I'm using cf and have access to the region. Otherwise you can grab it in bash.
aws ec2 describe-tags --region $REGION \
--filters "Name=resource-id,Values=`ec2metadata --instance-id`" | jq --raw-output \
'.Tags[] | select(.Key=="TAG_NAME") | .Value'
No jq.
aws ec2 describe-tags --region us-west-2 \
--filters "Name=resource-id,Values=`ec2-metadata --instance-id | cut -d " " -f 2`" \
--query 'Tags[?Key==`Name`].Value' \
--output text
Download and run a standalone executable to do that.
Sometimes one cannot install awscli that depends on python. docker might be out of the picture too.
Here is my implementation in golang:
https://github.com/hmalphettes/go-ec2-describe-tags
The Metadata tool seems to no longer be available, but that was an unnecessary dependency anyway.
Follow the AWS documentation to have the instance's profile grant it the "ec2:DescribeTags" action in a policy, restricting the target resources as much as you wish. (If you need a profile for another reason then you'll need to merge policies into a new profile-linked role).
Then:
aws --region $(curl -s http://169.254.169.254/latest/meta-data/placement/availability-zone | sed -e 's/.$//') ec2 describe-tags --filters Name=resource-type,Values=instance Name=resource-id,Values=$(curl http://169.254.169.254/latest/meta-data/instance-id) Name=key,Values=Name |
perl -nwe 'print "$1\n" if /"Value": "([^"]+)/;'
Well there are lots of good answers here but none quite worked for me exactly out of the box, I think the CLI has been updated since some of them and I do like using the CLI. The following single command works out of the box for me in 2021 (as long as the instance's IAM role is allowed to describe-tags).
aws ec2 describe-tags \
--region "$(ec2-metadata -z | cut -d' ' -f2 | sed 's/.$//')" \
--filters "Name=resource-id,Values=$(ec2-metadata --instance-id | cut -d " " -f 2)" \
--query 'Tags[?Key==`Name`].Value' \
--output text
AWS has recently announced support for instance tags in Instance Metadata Service: https://aws.amazon.com/about-aws/whats-new/2022/01/instance-tags-amazon-ec2-instance-metadata-service/
If you have have the tag metadata option enabled for an instance, you can simply do
$ TOKEN=`curl -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 900"`
$ curl -H "X-aws-ec2-metadata-token: $TOKEN" -v http://169.254.169.254/latest/meta-data/tags/instance
It is possible to get Instance tags from within the instance via metadata.
First, allow access to tags in instance metadata as explained here
Then, run this command for IMDSv1, Refer
curl http://169.254.169.254/latest/meta-data/tags/instance/Name
or this command for IMDSv2
TOKEN=`curl -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600"` \
&& curl -H "X-aws-ec2-metadata-token: $TOKEN" -v http://169.254.169.254/latest/meta-data/tags/instance
Install AWS CLI:
curl "https://s3.amazonaws.com/aws-cli/awscli-bundle.zip" -o "awscli-bundle.zip"
sudo apt-get install unzip
unzip awscli-bundle.zip
sudo ./awscli-bundle/install -i /usr/local/aws -b /usr/local/bin/aws
Get the tags for the current instance:
aws ec2 describe-tags --filters "Name=resource-id,Values=`ec2metadata --instance-id`"
Outputs:
{
"Tags": [
{
"ResourceType": "instance",
"ResourceId": "i-6a7e559d",
"Value": "Webserver",
"Key": "Name"
}
]
}
Use a bit of perl to extract the tags:
aws ec2 describe-tags --filters \
"Name=resource-id,Values=`ec2metadata --instance-id`" | \
perl -ne 'print "$1\n" if /\"Value\": \"(.*?)\"/'
Returns:
Webserver
For those crazy enough to use Fish shell on EC2, here's a handy snippet for your /home/ec2-user/.config/fish/config.fish. The hostdata command now will list all your tags as well as the public IP and hostname.
set -x INSTANCE_ID (wget -qO- http://instance-data/latest/meta-data/instance-id)
set -x REGION (wget -qO- http://instance-data/latest/meta-data/placement/availability-zone | sed 's/.$//')
function hostdata
aws ec2 describe-tags --region $REGION --filter "Name=resource-id,Values=$INSTANCE_ID" --output=text | sed -r 's/TAGS\t(.*)\t.*\t.*\t(.*)/\1="\2"/'
ec2-metadata | grep public-hostname
ec2-metadata | grep public-ipv4
end