I have some EC2 instances that I don't really know who launched them.
Is there a way to know who launched a specific instance?
Unfortunately this information is not directly available via an API call - you currently have two options:
depending on your needs, you could approximate your goal by using the DescribeInstances API action to look at the key-name used for starting that instance (if any, it's optional, though usually in place) - assuming you have followed security best practices and are using a dedicated EC2 key pair per IAM user (rather than sharing keys), the key should usually denote the user who started the instance ;)
that's most easily tested via the AWS Command Line Interface, specifically describe-instances
nowadays you could activate AWS CloudTrail, which records AWS API calls for your account and delivers log files to you and provides exactly the information you are after:
The recorded information includes the identity of the API caller, the
time of the API call, the source IP address of the API caller, the
request parameters, and the response elements returned by the AWS
service.
AWS CloudTrail is a long overdue and invaluable tool and provides unprecedented insight into your AWS usage; it is easy to get started and only has a few remaining caveats currently:
most importantly, it isn't available in all regions yet, but AWS has just extended it to 3 more for a total of 5, see AWS CloudTrail Expands Again - More Locations and Services, thus quickly approaching coverage of their entire Global Infrastructure
not all services are covered yet, but AWS hast just extended it to 7 more for a total of 15, see AWS CloudTrail Update - Seven New Services
depending on your needs, you most likely want to have a Logging as a Service (LaaS) solution in place to ease digging through the vast amount of logs, and provide alerts etc. - several providers already offer dedicated CloudTrail integration (and usually a free tier sufficient for that as well)
events are delivered within 15 minutes of the API call and the resulting logs to your S3 bucket approximately every 5 minutes, resulting in a maximum delay of 20 minutes, which is enough for post hoc and batch analysis, but not sufficient for near real-time alerting of course (see the AWS CloudTrail FAQ)
Assuming that you have configured CloudTrail already in your console (to monitor the right events) here's a possible solution to retrieve the username of who launched a certain EC2 instance, given the instance ID.
This solution uses the AWS CLI to retrieve information about which instances were launched, and jq for parsing.
When calling the aws cloudtrail lookup-events function, we can specify:
The name of the event. In this case, RunInstances.
The date from which to start searching, start-date. This can be when the EC2 instance was launched.
The date up to which to search, end-date. This can be just one minute after the instance was started.
First, let's define the instance id and the region variables:
$ instance_id="i-08dcc12c018737061"
$ region="us-east-1"
then, let's find out when the instance was launched:
$ instance_launch_date=$(aws ec2 describe-instances --instance-ids $instance_id --region $region | jq '.Reservations[].Instances[].LaunchTime')
$ echo $instance_launch_date
"2020-07-08T15:21:02.000Z"
now, let's sanitize the date so that it's in the right format for the CloudTrail API.
$ launch_date=$(sed -E 's/\"([0-9]{4})-([0-9]{2})-([0-9]{2}).([0-9]{2}):([0-9]{2}):([0-9]{2}).+/\2\/\3\/\1 \4:\5:\6/' <<< $instance_launch_date)
$ start_date=$(date -d "${launch_date}" +"%s")
$ echo $start_date
1594221662
now, let's define the end date as the date the instance was launched plus 1 minute:
$ end_date=$(date -d "${launch_date} 1 min" +"%s")
echo $end_date
1594221722
and now let's call the CloudTrail API, and filter the output with jq:
aws cloudtrail lookup-events --lookup-attributes AttributeKey=EventName,AttributeValue=RunInstances --start-time $start_date --end-time $end_date --region $region | jq --arg INSTANCE_ID "$instance_id" '.Events[] | select(.CloudTrailEvent | fromjson | .responseElements.instancesSet.items[].instanceId | contains($INSTANCE_ID)) | .Username '
"paolo"
Related
I want to understand the idiomatic Bash (ie the Bash equivalent of 'Pythonic') way with the AWS CLI to create an AWS resource (say an AWS Cloud9 EC2 environment) and then 'poll'/'wait'/'sleep' for that resource to reach a certain state; when the AWS resources reaches the given state, the command completes and the user is able to enter commands back into the shell.
EG after the user hits Enter
[cloudshell-user#ip-10-10-10-10 ~]$ #an idiomatic Bash one-liner that combines 'aws cloud9 create-environment-ec2 --name this-resource-will-take-a-moment-to-create --instance-type t2.micro' and 'aws cloud9 describe-environments --environment-ids tHiSiDwAsThEoUtPuToFtHePrEvIoUsCoMmAnD'
Then after some time the user will see
[cloudshell-user#ip-10-10-10-10 ~]$ #an idiomatic Bash one-liner that combines 'aws cloud9 create-environment-ec2 --name this-resource-will-take-a-moment-to-create --instance-type t2.micro' and 'aws cloud9 describe-environments --environment-ids tHiSiDwAsThEoUtPuToFtHePrEvIoUsCoMmAnD'
[cloudshell-user#ip-10-10-10-10 ~]$
at which point the AWS Cloud9 EC2 environment has reached a given state.
As the create(create-,launch-, etc)/wait(get-/describe-) pattern is common among AWS services API calls and then corresponding AWS CLI commands the one-liner should be re-usable given minor changes for other services.
The Bash one-liner will be used in the CloudShell but should be portable to be used in other installations of Bash.
I have read about && and sleep but would like some direction as to how to integrate these commands (or any alternative ones) in a idiomatic Bash one-liner. Also I am concerned about not violating any throttling or request limiting AWS APIs have.
As jarmod commented with 'boto3 Waiters', I looked more into the AWS CLI's Github Issues and found this Issue
The underlying library that powers the CLI, botocore, has this capability already via a concept it calls "waiters", which is a data driven way to describe waiting for a resource to reach a certain state... We just need to come up with a suitable API to expose this in the CLI.
The capability was implemented in aws-cli but each service has to then use that capability for their own service.
It looks at the time of this writing the Cloud9 part of the aws-cli does not support wait.
I want to use boto3 to get list of available aws services in a specific region. Is there any way to do this.
I tried using Session object:
session = boto3.Session(region_name='ap-south-1').get_available_services()
but it is giving me all the AWS services. For eg: Cloudsearch is not present in ap-south-1, but this function still gives me the service in the output.
Also, I don't want to use ssm service get_parameters_by_path function as I don't want to give ssm permission.
Any other way?
To be frank, I reckon, your best bet actually is the Systems Manager Parameter Store.
For example, you can easily display a complete list of all available AWS services, sort them into alphabetical order, and, for the brevity, show the first 10.
$ aws ssm get-parameters-by-path \
--path /aws/service/global-infrastructure/services --output json | \
jq '.Parameters[].Name' | sort | head -10
Output:
"/aws/service/global-infrastructure/services/acm"
"/aws/service/global-infrastructure/services/acm-pca"
"/aws/service/global-infrastructure/services/alexaforbusiness"
"/aws/service/global-infrastructure/services/apigateway"
"/aws/service/global-infrastructure/services/application-autoscaling"
"/aws/service/global-infrastructure/services/appmesh"
"/aws/service/global-infrastructure/services/appstream"
"/aws/service/global-infrastructure/services/appsync"
"/aws/service/global-infrastructure/services/athena"
"/aws/service/global-infrastructure/services/autoscaling"
And here's how to get the list of services that are available in a given region. Show first 10 and sorted.
$ aws ssm get-parameters-by-path \
--path /aws/service/global-infrastructure/regions/us-east-1/services --output json | \
jq '.Parameters[].Name' | sort | head -10
But... if you want any other way you might want to try AWS Price List API.
With the AWS Price List Query API, you can query specific information about AWS services, products, and pricing using an AWS SDK or the AWS CLI.
This obviously can be narrowed down to a specific region. If there's a price, there is a service.
I got this by below code:
resp = boto3.Session().get_available_regions('cloudsearch')
This gave me the list of all the regions where cloudsearch service is available.
I really need to know about the stopped time of AWS EC2 instances. I have checked with AWS cloudtrail, but its not easy to find the exact stopped EC2 instance. Is possible to see exact time of stopped EC2 instances by aws-cli commands or any boto3 script?
You can get this info from StateTransitionReason in describe-instances AWS CLI when you search for stopped instances:
aws ec2 describe-instances --filter Name=instance-state-name,Values=stopped --query 'Reservations[].Instances[*].StateTransitionReason' --output text
Example output:
User initiated (2020-12-03 07:16:35 GMT)
AWS Config keeps track of the state of resources as they change over time.
From What Is AWS Config? - AWS Config:
AWS Config provides a detailed view of the configuration of AWS resources in your AWS account. This includes how the resources are related to one another and how they were configured in the past so that you can see how the configurations and relationships change over time.
Thus, you could look back through the configuration history of the Amazon EC2 instance and extract times for when the instance changed to a Stopped state.
Sometimes time is missing from StateTransitionReason, you can use CloudTrail and search for Resource Name = instance ID to find out StopInstance(s) API calls.
By default you can track back 90 days, or indefinitely if you create your own trail.
I am trying to develop an autoscaling policy with Cloud formation template to spinup multiple ec2-instances when my load surge happens in AWS cloud.
Every time the new nodes gets generated via autoscaling ;I need to allocate a hostname. The hostname should be in such a way that it should resemble the nodes application role ( such as web,db,or mail server etc....This is a requirement of my legacy application.The host name can be in sequence like web01,web02 for web servers and db01,db02... for db server .I am not sure how i can achieve it ,Any help would be appreciated.
It's actually a one-liner:
sudo hostname `curl http://169.254.169.254/latest/meta-data/iam/info | jq -r .InstanceProfileArn | cut -d "/" -f 2`
Just run this at instance startup (in cloud-init script, in /etc/rc.local, in /etc/init.d/hostname etc.).
curl http://169.254.169.254/latest/meta-data/iam/info receives information in JSON format about IAM instance profile (EC2 instances don't really have IAM roles, they have profiles derived from IAM roles).
jq -r .InstanceProfileArn extracts only profile ARN from that JSON.
cut -d "/" -f 2 removes everything from profile ARN except the last part, which matches IAM role.
hostname... Well, sets the hostname.
I am trying to come up with a script to automate the setting up of desired count of AutoScalingGroups based on some kind of profiles e.g., SHUTDOWN profile should set everything to zero.
We have lot of applications under single account itself. So when running below command, it gives all the resources.
aws ecs list-clusters
Is there a way to filter these by either tags or any other means? Apparently --filter is not a valid option for aws ecs or aws autoscaling commands.
I am utilizing the grep command for now.
aws ecs list-clusters | grep string1 | grep string2
Not sure that's exactly what you're asking, but if you want to play with the JSON output of these commands (or filter/transform any JSON string in general), there's no better tool than jq. Takes some time to get into, but this tool might become your best friend.
Once installed, you can issue commands such as:
aws ecs describe-clusters|jq -r '.clusters[]|{clusterName, status}'
To create a cluster name/status list from the info.
aws ecs describe-clusters|jq -r '.clusters[]|if .status == "INACTIVE" then .clusterArn else null end'
To list all the inactive clusters.
Add a delete command this way to delete all the inactive clusters (don't run it!!!):
aws ecs describe-clusters|jq -r '.clusters[]|if .status == "INACTIVE" then .clusterArn else null end'|xargs aws ecs delete-clusters --clusters
I have only one cluster at disposal, I didn't test if these commands still work with many clusters (JSON tables properly parsed), but you get the idea...
jq tutorial: https://stedolan.github.io/jq/tutorial/