Description
Looking at this AWS EC2 Doc It should be possible to add exec > >(tee /var/log/user-data.log|logger -t user-data -s 2>/dev/console) 2>&1 to a user-data script that is run on EC2 initialization.
When running aws ec2 --region eu-west-1 get-console-output --instance-id i-<id> | grep "user-data" (or searching for other patterns that should be present) none are found after the ec2 initialization.
Goal
To read the results and debug information from this initialization script without needing to SSH into the ec2 instance and poll the logs for the shutdown statement. Using the Instance shutdown as the "finished" state has a significant simplification on the deployment process for this repository.
Question
What about this particular setup is not correct such that we are not getting logs out of the aws ec2 get-console-output command.
Alternative answer: What's a better method of retrieving the logs from an EC2 instance.
User-Data Script
#!/bin/bash -xe
# redirect output to log file and console
exec > >(tee /var/log/user-data.log|logger -t user-data -s 2>/dev/console) 2>&1
if [ -n "postgresql,jq" ]
then
yum -y -q update
echo "complete: yum update"
IFS=', ' read -r -a REQS <<< "postgresql,jq"
for REQ in "${REQS[#]}"
do
yum -y -q install $REQ
echo "complete: yum install $REQ"
done
fi
echo "EC2P: ENTERING BOOTSTRAP SCRIPT....."
export PGPASSWORD=$( aws secretsmanager get-secret-value --secret-id <secret_arn> --query SecretString --region eu-west-1 | jq -r . | jq -r .password)
aws s3 cp s3://<query_object> /tmp/postgres-query.sql
echo "EC2P: Starting PSQL Execution"
psql -h <host> \
-p 5432 \
-U <user> \
-o /tmp/postgres-query-result.txt \
<db_name>_db \
< /tmp/postgres-query.sql \
> /tmp/postgres-query-output.txt 2>&1
echo "psql exit code is $?"
echo "EC2P: PSQL Execution Complete"
# since instance_initiated_shutdown_behavior = "terminate"
echo "shutdown"
shutdown
Terraform Declaration Of EC2 Instance
resource "aws_instance" "ec2_provisioner" {
count = var.enabled ? 1 : 0
ami = data.aws_ami.ec2_provisioner.id
iam_instance_profile = var.iam_instance_profile
instance_initiated_shutdown_behavior = "terminate"
instance_type = var.instance_type
root_block_device {
volume_type = "gp2"
volume_size = "16"
delete_on_termination = "true"
}
subnet_id = var.subnet_id
tags = merge(
{
"es:global:component-name" = "${var.component_name}-ec2-provisioner",
"Name" = var.name
},
jsondecode(var.additional_tags)
)
user_data = templatefile(
"${path.module}/user_data.tpl",
{
BASH_SCRIPT = var.bash_script,
PACKAGES = join(",", var.packages)
}
)
volume_tags = {
"Name" = var.name
}
vpc_security_group_ids = [var.security_group_id]
}
The core issue here is that our initialization of the instance is producing too much console output
74295 # > 64 KBs
Which is hitting a size limit built into the get-console-output command
By default, the console output returns buffered information that was posted shortly after an instance transition state (start, stop, reboot, or terminate). This information is available for at least one hour after the most recent post. Only the most recent 64 KB of console output is available
The solution we're going with to fix this issue to to enable SSM and to log into parse the log output.
I have some Terraform code with an aws_instance and a null_resource:
resource "aws_instance" "example" {
ami = data.aws_ami.server.id
instance_type = "t2.medium"
key_name = aws_key_pair.deployer.key_name
tags = {
name = "example"
}
vpc_security_group_ids = [aws_security_group.main.id]
}
resource "null_resource" "example" {
provisioner "local-exec" {
command = "ANSIBLE_HOST_KEY_CHECKING=False ansible-playbook -T 300 -i ${aws_instance.example.public_dns}, --user centos --private-key files/id_rsa playbook.yml"
}
}
It kinda works, but sometimes there is a bug (probably when the instance in a pending state). When I rerun Terraform - it works as expected.
Question: How can I run local-exec only when the instance is running and accepting an SSH connection?
The null_resource is currently only going to wait until the aws_instance resource has completed which in turn only waits until the AWS API returns that it is in the Running state. There's a long gap from there to the instance starting the OS and then being able to accept SSH connections before your local-exec provisioner can connect.
One way to handle this is to use the remote-exec provisioner on the instance first as that has the ability to wait for the instance to be ready. Changing your existing code to handle this would look like this:
resource "aws_instance" "example" {
ami = data.aws_ami.server.id
instance_type = "t2.medium"
key_name = aws_key_pair.deployer.key_name
tags = {
name = "example"
}
vpc_security_group_ids = [aws_security_group.main.id]
}
resource "null_resource" "example" {
provisioner "remote-exec" {
connection {
host = aws_instance.example.public_dns
user = "centos"
file = file("files/id_rsa")
}
inline = ["echo 'connected!'"]
}
provisioner "local-exec" {
command = "ANSIBLE_HOST_KEY_CHECKING=False ansible-playbook -T 300 -i ${aws_instance.example.public_dns}, --user centos --private-key files/id_rsa playbook.yml"
}
}
This will first attempt to connect to the instance's public DNS address as the centos user with the files/id_rsa private key. Once it is connected it will then run echo 'connected!' as a simple command before moving on to your existing local-exec provisioner that runs Ansible against the instance.
Note that just being able to connect over SSH may not actually be enough for you to then provision the instance. If your Ansible script tries to interact with your package manager then you may find that it is locked from the instance's user data script running. If this is the case you will need to remotely execute a script that waits for cloud-init to be complete first. An example script looks like this:
#!/bin/bash
while [ ! -f /var/lib/cloud/instance/boot-finished ]; do
echo -e "\033[1;36mWaiting for cloud-init..."
sleep 1
done
There is an ansible specific solution for this problem. Add this code to you playbook(there is all so pre_task clause if you use roles)
- name: will wait till reachable
hosts: all
gather_facts: no # important
tasks:
- name: Wait for system to become reachable
wait_for_connection:
- name: Gather facts for the first time
setup:
For cases where instances are not externally exposed (About 90% of the time in most of my projects), and SSM agent is installed on the target instance (newer AWS AMIs come pre-loaded with it), you can leverage SSM to probe the instance. Here's some sample code:
instanceId=$1
echo "Waiting for instance to bootstrap ..."
tries=0
responseCode=1
while [[ $responseCode != 0 && $tries -le 10 ]]
do
echo "Try # $tries"
cmdId=$(aws ssm send-command --document-name AWS-RunShellScript --instance-ids $instanceId --parameters commands="cat /tmp/job-done.txt # or some other validation logic" --query Command.CommandId --output text)
sleep 5
responseCode=$(aws ssm get-command-invocation --command-id $cmdId --instance-id $instanceId --query ResponseCode --output text)
echo "ResponseCode: $responseCode"
if [ $responseCode != 0 ]; then
echo "Sleeping ..."
sleep 60
fi
(( tries++ ))
done
echo "Wait time over. ResponseCode: $responseCode"
Assuming you have AWS CLI installed locally, you can have this null_resource required before you act on the instance. In my case, I was building an AMI.
resource "null_resource" "wait_for_instance" {
depends_on = [
aws_instance.my_instance
]
triggers = {
always_run = "${timestamp()}"
}
provisioner "local-exec" {
command = "${path.module}/scripts/check-instance-state.sh ${aws_instance.my_instance.id}"
}
}
I switch instances between different regions frequently and sometimes I forget to turn off my running instance from a different region. I couldn't find any way to see all the running instances on Amazon console.
Is there any way to display all the running instances regardless of region?
Nov 2021 Edit: AWS has recently launched the Amazon EC2 Global View with initial support for Instances, VPCs, Subnets, Security Groups and Volumes.
See the announcement or documentation for more details
A non-obvious GUI option is the Tag Editor in the Resource Groups console. Here you can find all instances across all regions, even if the instances were not tagged.
I don't think you can currently do this in the AWS GUI. But here is a way to list all your instances across all regions with the AWS CLI:
for region in `aws ec2 describe-regions --region us-east-1 --output text | cut -f4`
do
echo -e "\nListing Instances in region:'$region'..."
aws ec2 describe-instances --region $region
done
Taken from here (If you want to see full discussion)
Also, if you're getting a
You must specify a region. You can also configure your region by running "aws configure"
You can do so with aws configure set region us-east-1, thanks #Sabuncu for the comment.
Update
Now (in 2019) the cut command should be applied on the 4th field: cut -f4
In Console
Go to VPC dashboard https://console.aws.amazon.com/vpc/home and click on Running instances -> See all regions.
In CLI
Add this for example to .bashrc. Reload it source ~/.bashrc, and run it
Note: Except for aws CLI you need to have jq installed
function aws.print-all-instances() {
REGIONS=`aws ec2 describe-regions --region us-east-1 --output text --query Regions[*].[RegionName]`
for REGION in $REGIONS
do
echo -e "\nInstances in '$REGION'..";
aws ec2 describe-instances --region $REGION | \
jq '.Reservations[].Instances[] | "EC2: \(.InstanceId): \(.State.Name)"'
done
}
Example output:
$ aws.print-all-instances
Listing Instances in region: 'eu-north-1'..
"EC2: i-0548d1de00c39f923: terminated"
"EC2: i-0fadd093234a1c21d: running"
Listing Instances in region: 'ap-south-1'..
Listing Instances in region: 'eu-west-3'..
Listing Instances in region: 'eu-west-2'..
Listing Instances in region: 'eu-west-1'..
Listing Instances in region: 'ap-northeast-2'..
Listing Instances in region: 'ap-northeast-1'..
Listing Instances in region: 'sa-east-1'..
Listing Instances in region: 'ca-central-1'..
Listing Instances in region: 'ap-southeast-1'..
Listing Instances in region: 'ap-southeast-2'..
Listing Instances in region: 'eu-central-1'..
Listing Instances in region: 'us-east-1'..
Listing Instances in region: 'us-east-2'..
Listing Instances in region: 'us-west-1'..
Listing Instances in region: 'us-west-2'..
From VPC Dashboard:
First go to VPC Dashboard
Then find Running instances and expand see all regions. Here you can find all the running instances of all region:
From EC2 Global view:
Also you can use AWS EC2 Global View to watch Resource summary
and Resource counts per Region.
#imTachu solution works well. To do this via the AWS console...
AWS console
Services
Networking & Content Delivery
VPC
Look for a block named "Running Instances", this will show you the current region
Click the "See all regions" link underneath
Every time you create a resource, tag it with a name and now you can use Resource Groups to find all types of resources with a name tag across all regions.
After reading through all the solutions and trying bunch of stuff, the one that worked for me was-
List item
Go to Resource Group
Tag Editor
Select All Regions
Select EC2 Instance in resource type
Click Search Resources
Based on imTachus answer but less verbose, plus faster. You need to have jq and aws-cli installed.
set +m
for region in $(aws ec2 describe-regions --query "Regions[*].[RegionName]" --output text); do
aws ec2 describe-instances --region "$region" | jq ".Reservations[].Instances[] | {type: .InstanceType, state: .State.Name, tags: .Tags, zone: .Placement.AvailabilityZone}" &
done; wait; set -m
The script runs the aws ec2 describe-instances in parallel for each region (now 15!) and extracts only the relevant bits (state, tags, availability zone) from the json output. The set +m is needed so the background processes don't report when starting/ending.
Example output:
{
"type": "t2.micro",
"state": "stopped",
"tags": [
{
"Key": "Name",
"Value": "MyEc2WebServer"
},
],
"zone": "eu-central-1b"
}
You can run DescribeInstances() across all regions.
Additionally, you can:
Automate it through Lambda and Cloud watch.
Create api endpoint using Lambda and api gateway and use it in your code
A sample in NodeJS:
Create and array of regions (endpoints). [can also use AWS describeRegions() ]
var regionNames = ['us-west-1', 'us-west-2', 'us-east-1', 'eu-west-1', 'eu-central-1', 'sa-east-1', 'ap-southeast-1', 'ap-southeast-2', 'ap-northeast-1', 'ap-northeast-2'];
regionNames.forEach(function(region) {
getInstances(region);
});
Then, in getInstances function, DescribeInstances() can be
called.
function getInstances(region) {
EC2.describeInstances(params, function(err, data) {
if (err) return console.log("Error connecting to AWS, No Such Instance Found!");
data.Reservations.forEach(function(reservation) {
//do any operation intended
});
}
And Off Course, feel free to use ES6 and above.
I wrote a lambda function to get you all the instances in any state [running, stopped] and from any regions, will also give details about instance type and various other parameters.
The Script runs across all AWS regions and calls DescribeInstances(), to get the instances.
You just need to create a lambda function with run-time nodejs.
You can even create API out of it and use it as and when required.
Additionally, You can see AWS official Docs For DescribeInstances to explore many more options.
A quick bash oneliner command to print all the instance IDs in all regions:
$ aws ec2 describe-regions --query "Regions[].{Name:RegionName}" --output text |xargs -I {} aws ec2 describe-instances --query Reservations[*].Instances[*].[InstanceId] --output text --region {}
# Example output
i-012344b918d75abcd
i-0156780dad25fefgh
i-0490122cfee84ijkl
...
My script below, based on various tips from this post and elsewhere. The script is easier to follow (for me at least) than the long command lines.
The script assumes credential profile(s) are stored in file ~/.aws/credentials looking something like:
[default]
aws_access_key_id = foobar
aws_secret_access_key = foobar
[work]
aws_access_key_id = foobar
aws_secret_access_key = foobar
Script:
#!/usr/bin/env bash
#------------------------------------#
# Script to display AWS EC2 machines #
#------------------------------------#
# NOTES:
# o Requires 'awscli' tools (for ex. on MacOS: $ brew install awscli)
# o AWS output is tabbed - we convert to spaces via 'column' command
#~~~~~~~~~~~~~~~~~~~~#
# Assemble variables #
#~~~~~~~~~~~~~~~~~~~~#
regions=$(aws ec2 describe-regions --output text | cut -f4 | sort)
query_mach='Reservations[].Instances[]'
query_flds='PrivateIpAddress,InstanceId,InstanceType'
query_tags='Tags[?Key==`Name`].Value[]'
query_full="$query_mach.[$query_flds,$query_tags]"
#~~~~~~~~~~~~~~~~~~~~~~~~#
# Output AWS information #
#~~~~~~~~~~~~~~~~~~~~~~~~#
# Iterate through credentials profiles
for profile in 'default' 'work'; do
# Print profile header
echo -e "\n"
echo -e "~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~"
echo -e "Credentials profile:'$profile'..."
echo -e "~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~"
# Iterate through all regions
for region in $regions; do
# Print region header
echo -e "\n"
echo -e "Region: $region..."
echo -e "--------------------------------------------------------------"
# Output items for the region
aws ec2 describe-instances \
--profile $profile \
--region $region \
--query $query_full \
--output text \
| sed 's/None$/None\n/' \
| sed '$!N;s/\n/ /' \
| column -t -s $'\t'
done
done
AWS has recently launched the Amazon EC2 Global View with initial support for Instances, VPCs, Subnets, Security Groups, and Volumes.
To see all running instances go to EC2 or VPC console and click EC2 Global View in the top left corner.
Then click on Global Search tab and filter by Resource type and select Instance. Unfortunately, this will show instances in all states:
pending
running
stopping
stopped
shutting-down
terminated
I created an open-source script that helps you to list all AWS instances. https://github.com/Appnroll/aws-ec2-instances
That's a part of the script that lists the instances for one profile recording them into an postgreSQL database with using jq for json parsing:
DATABASE="aws_instances"
TABLE_NAME="aws_ec2"
SAVED_FIELDS="state, name, type, instance_id, public_ip, launch_time, region, profile, publicdnsname"
# collects the regions to display them in the end of script
REGIONS_WITH_INSTANCES=""
for region in `aws ec2 describe-regions --output text | cut -f3`
do
# this mappping depends on describe-instances command output
INSTANCE_ATTRIBUTES="{
state: .State.Name,
name: .KeyName, type: .InstanceType,
instance_id: .InstanceId,
public_ip: .NetworkInterfaces[0].Association.PublicIp,
launch_time: .LaunchTime,
\"region\": \"$region\",
\"profile\": \"$AWS_PROFILE\",
publicdnsname: .PublicDnsName
}"
echo -e "\nListing AWS EC2 Instances in region:'$region'..."
JSON=".Reservations[] | ( .Instances[] | $INSTANCE_ATTRIBUTES)"
INSTANCE_JSON=$(aws ec2 describe-instances --region $region)
if echo $INSTANCE_JSON | jq empty; then
# "Parsed JSON successfully and got something other than false/null"
OUT="$(echo $INSTANCE_JSON | jq $JSON)"
# check if empty
if [[ ! -z "$OUT" ]] then
for row in $(echo "${OUT}" | jq -c "." ); do
psql -c "INSERT INTO $TABLE_NAME($SAVED_FIELDS) SELECT $SAVED_FIELDS from json_populate_record(NULL::$TABLE_NAME, '${row}') ON CONFLICT (instance_id)
DO UPDATE
SET state = EXCLUDED.state,
name = EXCLUDED.name,
type = EXCLUDED.type,
launch_time = EXCLUDED.launch_time,
public_ip = EXCLUDED.public_ip,
profile = EXCLUDED.profile,
region = EXCLUDED.region,
publicdnsname = EXCLUDED.publicdnsname
" -d $DATABASE
done
REGIONS_WITH_INSTANCES+="\n$region"
else
echo "No instances"
fi
else
echo "Failed to parse JSON, or got false/null"
fi
done
To run jobs in parallel and use multiple profiles use this script.
#!/bin/bash
for i in profile1 profile2
do
OWNER_ID=`aws iam get-user --profile $i --output text | awk -F ':' '{print $5}'`
tput setaf 2;echo "Profile : $i";tput sgr0
tput setaf 2;echo "OwnerID : $OWNER_ID";tput sgr0
for region in `aws --profile $i ec2 describe-regions --output text | cut -f4`
do
tput setaf 1;echo "Listing Instances in region $region";tput sgr0
aws ec2 describe-instances --query 'Reservations[*].Instances[*].[Tags[?Key==`Name`].Value , InstanceId]' --profile $i --region $region --output text
done &
done
wait
Screenshot:
Not sure how long this option's been here, but you can see a global view of everything by searching for EC2 Global View
https://console.aws.amazon.com/ec2globalview/home#
Using bash-my-aws:
region-each instances
Based on #hansaplast code I created Windows friendly version that supports multiple profiles as an argument. Just save that file as cmd or bat file. You also need to have jq command.
#echo off
setlocal enableDelayedExpansion
set PROFILE=%1
IF "%1"=="" (SET PROFILE=default)
echo checkin instances in all regions for %PROFILE% account
FOR /F "tokens=* USEBACKQ" %%F IN (`aws ec2 describe-regions --query Regions[*].[RegionName] --output text --profile %PROFILE%`) DO (
echo === region: %%F
aws ec2 describe-instances --region %%F --profile %PROFILE%| jq ".Reservations[].Instances[] | {type: .InstanceType, state: .State.Name, tags: .Tags, zone: .Placement.AvailabilityZone}"
)
You may use cli tool designed for enumerating cloud resources (cross-region and cross-accounts scan) - https://github.com/scopely-devops/skew
After short configuration you may use the following code for list all instances in all US AWS regions (assuming 123456789012 is your AWS account number).
from skew import scan
arn = scan('arn:aws:ec2:us-*:123456789012:instance/*')
for resource in arn:
print(resource.data)
Good tool to CRUD AWS resources. Find [EC2|RDS|IAM..] in all regions. There can do operations (stop|run|terminate) on filters results.
python3 awsconsole.py ec2 all // return list of all instances
python3 awsconsole.py ec2 all -r eu-west-1
python3 awsconsole.py ec2 find -i i-0552e09b7a54fa2cf --[terminate|start|stop]