I am using the AWS CLI to fetching EC2 instance details. My requirements are:
1) Fetch only running EC2 instances:
instance id
instance tag
instance type
availability zone
blockdevices name
platform
reserved or ondemand
However I can't able to fetch last 3 attributes (blockdevices name, platform and reserved or ondemand). When I add it in my query it is showing None. What do I need to change in query?
My query is:
aws ec2 describe-instances --filters "Name=instance-state-name,Values=running" --query "Reservations[].Instances[].[Tags[?Key==`Name`] | [0].Value,InstanceId,State.Name,InstanceType,Placement.AvailabilityZone]" --output table > output.txt
The output is:
| test1 | i-xxxxxxx | m1.small | running | us-east-1a |
| test2 | i-xxxxx | m1.large | running | us-east-1c |
| test3 | i-xxxxx | m1.xlarge | running | us-east-1c |
| test4 | i-xxxxxxx | m3.2xlarge | running | us-east-1a |
2) Need to fetch each instance price considering those attributes (region,type,ondemand/reserved) and put full instance details in a CSV file.
Blockdevices name
I'm not sure what you mean by "name", but there is a DeviceName field:
aws ec2 describe-instances --query Reservations[*].Instances[*].BlockDeviceMappings[*].DeviceName
Platform
The Platform field is either Windows or undefined (for Linux):
aws ec2 describe-instances --query Reservations[*].Instances[*].Platform
Reserved or On-Demand
This is a billing concept and is not available for an instance. In fact, an instance could be billed as On-Demand in one hour and as a Reserved Instance in another hour.
The basic concept is that, for each Reserved Instance owned by an account, the billing system looks for a running Amazon EC2 instance that matches the Reserved Instance parameters. The matching instance is not charged the hourly fee for that hour. This information can be seen in the billing files, but is not available as information against the instance.
Make sure you have turned on Detailed Billing Reports so that you can receive this level of billing information. It is only available after the reports have been activated (not back in time).
Fetch the price
There is no command to fetch the price of a running Amazon EC2 instance. There is a Price List API that provides pricing information similar to the pricing pages on the AWS website.
You could write a program to fetch the prices via the Price List API and determine which price is appropriate for an instance. However, as stated above, the program would not know whether an instance was running as a Reserved Instance.
Put full instance details in a CSV file
The AWS CLI does not output CSV format. You would have to post-process the output into the desired format. Note that some of your information (eg BlockDevices) might return multiple values for a single instance.
Related
I am trying to write a script that would save me going to the web Console every time I need checking how many RDS instances & S3 buckets I can create.
In the RDS section, there is a DB Instances (20/50) title, I need to get both the 20 and the 50
In the S3 bucket section, there is the total number of buckets. The implied max number is 100, but my company has increased it to 150. I need the number and the limit (as above).
Is that possible?
You can use AWS CLI to get all of these values.
In the RDS section, there is a DB Instances (20/50) title, I need to get both the 20 and the 50
Get the total number of RDS instances you can create:
aws service-quotas get-service-quota \
--service-code rds \
--quota-code L-7B6409FD
Get the current number of running RDS instances:
aws rds describe-db-instances --region us-east-1 | grep DBInstanceIdentifier | wc -l
In the S3 bucket section, there is the total number of buckets. The implied max number is 100, but my company has increased it to 150. I need the number and the limit (as above).
Get the total number of S3 buckets you can create:
aws service-quotas get-service-quota \
--service-code s3 \
--quota-code L-DC2B2D3D
Get the current number of S3 bucket:
aws s3api list-buckets | grep -w Name | wc -l
I am trying to find out the list of Empty/Unused Load Balancer with all hosts in out of service status for a specific account using AWS cli. I know this query will help in fetching the list of used SGs.
aws elb describe-load-balancers --query 'LoadBalancerDescriptions[*].SecurityGroups[*]' --output text | tr '\t' '\n' | sort | uniq
Also trying to print the cloudwatch metric for avg amount of errors returned from each load balancer.
Recently got an email titled, "Important News from AWS About Amazon EC2-Classic" describing some changes that need to occur. These emails from AWS usually reference the effected resources though and this one did not. I am having a hard time identifying what resources in our account are effected by this. All our EC2 instances are in a VPC and I am not even sure if anything needs to change or not.
Is there a way to identify that an EC2 instance is classic?
I have looked through their linked documentation and gone through the instances we have but I cannot tell if they are "classic" of not.
You can identify the EC2-Classic env by checking the instance has VPC ID or not.
EC2 console
VPC ID is not shown by default. Enable VPC ID from Preference -> Attribute columns.
Then if VPC ID attribute is -, that means the instance is EC2-Classic. (Except that the instance state is not terminated.)
CLI
2 ways for checking. Output is none unless EC2-classic instances exist.
Describe instance with EC2-Classic env.
aws ec2 describe-instances --filters Name=instance-state-name,Values=pending,running,shutting-down,stopping,stopped | jq '.Reservations[].Instances[] | select(.VpcId == null)'
Describe the instance if it is the EC2-Classic.
aws ec2 describe-instances --instance-id i-xxxxxxxxxxxx --filters Name=instance-state-name,Values=pending,running,shutting-down,stopping,stopped | jq '.Reservations[].Instances[] | select(.VpcId == null)'
jq select for terminated state
This is another way to filter the result of aws ec2 describe-instances.
Adding .State.Name != "terminated" and in jq select works the same as --filters .... This might be more readable.
aws ec2 describe-instances | jq '.Reservations[].Instances[] | select(.State.Name != "terminated" and .VpcId == null)'
Edit note: Thanks to the suggestion from #AUdden, I have modified the CLI code for filtering out the terminated state. When we terminated instances (not stop), the instances exist for a while in terminated state. The terminated instances are not associated with VPC anymore. To do that, I have added --filters Name=instance-state-name,Values=pending,running,shutting-down,stopping,stopped.
Amazon provides a script to identify all resources affected by the retirement, including resources you may not consider such as security groups.
Important: Check the file errors.txt after running the script. The script will happily run through its steps even if there is an error (such as missing/wrong credentials) without showing any hint of trouble in the console output.
I have some EC2 instances that I don't really know who launched them.
Is there a way to know who launched a specific instance?
Unfortunately this information is not directly available via an API call - you currently have two options:
depending on your needs, you could approximate your goal by using the DescribeInstances API action to look at the key-name used for starting that instance (if any, it's optional, though usually in place) - assuming you have followed security best practices and are using a dedicated EC2 key pair per IAM user (rather than sharing keys), the key should usually denote the user who started the instance ;)
that's most easily tested via the AWS Command Line Interface, specifically describe-instances
nowadays you could activate AWS CloudTrail, which records AWS API calls for your account and delivers log files to you and provides exactly the information you are after:
The recorded information includes the identity of the API caller, the
time of the API call, the source IP address of the API caller, the
request parameters, and the response elements returned by the AWS
service.
AWS CloudTrail is a long overdue and invaluable tool and provides unprecedented insight into your AWS usage; it is easy to get started and only has a few remaining caveats currently:
most importantly, it isn't available in all regions yet, but AWS has just extended it to 3 more for a total of 5, see AWS CloudTrail Expands Again - More Locations and Services, thus quickly approaching coverage of their entire Global Infrastructure
not all services are covered yet, but AWS hast just extended it to 7 more for a total of 15, see AWS CloudTrail Update - Seven New Services
depending on your needs, you most likely want to have a Logging as a Service (LaaS) solution in place to ease digging through the vast amount of logs, and provide alerts etc. - several providers already offer dedicated CloudTrail integration (and usually a free tier sufficient for that as well)
events are delivered within 15 minutes of the API call and the resulting logs to your S3 bucket approximately every 5 minutes, resulting in a maximum delay of 20 minutes, which is enough for post hoc and batch analysis, but not sufficient for near real-time alerting of course (see the AWS CloudTrail FAQ)
Assuming that you have configured CloudTrail already in your console (to monitor the right events) here's a possible solution to retrieve the username of who launched a certain EC2 instance, given the instance ID.
This solution uses the AWS CLI to retrieve information about which instances were launched, and jq for parsing.
When calling the aws cloudtrail lookup-events function, we can specify:
The name of the event. In this case, RunInstances.
The date from which to start searching, start-date. This can be when the EC2 instance was launched.
The date up to which to search, end-date. This can be just one minute after the instance was started.
First, let's define the instance id and the region variables:
$ instance_id="i-08dcc12c018737061"
$ region="us-east-1"
then, let's find out when the instance was launched:
$ instance_launch_date=$(aws ec2 describe-instances --instance-ids $instance_id --region $region | jq '.Reservations[].Instances[].LaunchTime')
$ echo $instance_launch_date
"2020-07-08T15:21:02.000Z"
now, let's sanitize the date so that it's in the right format for the CloudTrail API.
$ launch_date=$(sed -E 's/\"([0-9]{4})-([0-9]{2})-([0-9]{2}).([0-9]{2}):([0-9]{2}):([0-9]{2}).+/\2\/\3\/\1 \4:\5:\6/' <<< $instance_launch_date)
$ start_date=$(date -d "${launch_date}" +"%s")
$ echo $start_date
1594221662
now, let's define the end date as the date the instance was launched plus 1 minute:
$ end_date=$(date -d "${launch_date} 1 min" +"%s")
echo $end_date
1594221722
and now let's call the CloudTrail API, and filter the output with jq:
aws cloudtrail lookup-events --lookup-attributes AttributeKey=EventName,AttributeValue=RunInstances --start-time $start_date --end-time $end_date --region $region | jq --arg INSTANCE_ID "$instance_id" '.Events[] | select(.CloudTrailEvent | fromjson | .responseElements.instancesSet.items[].instanceId | contains($INSTANCE_ID)) | .Username '
"paolo"
Is there a way to see how many instances are in a availability zone for AWS? I want to use the runInstances api to bring up instances but call it on the least loaded zone which I don't see a obvious solution to. Thanks.
As far as i know there's no way to check the amount of instances in an AZ. What you could do is use an Autoscaling group and specify the AZs that you want when creating it. Autoscaling will then disperse the instance load evenly amongst listed AZs
AZ = Availability zone
The obvious solution seems like it would be to call DescribeInstances and use the availability-zone filter to request the details of instances in each zone you want to check and count the instances returned in the response.
Or don't use the filter, which will get all of them for the region, then examine the records to see where each of them is, since that information is all returned in the response.
<instancesSet>
<item>
<instanceId>i-1a2b3c4d</instanceId>
...
<placement>
<availabilityZone>us-west-2a</availabilityZone>
<groupName/>
<tenancy>default</tenancy>
</placement>
I had the same problem.
I solved it using aws cli:
aws cloudformation describe-stack-resources --stack-name STACKNAME\
--output text | grep 'AWS::AutoScaling::AutoScalingGroup' | cut -f3 > /tmp/tmpfile
ASGNAME="`cat /tmp/tmpfile`"
aws autoscaling describe-auto-scaling-groups --auto-scaling-group-names $ASGNAME \
--output text | egrep -e 'INSTANCES.*InService'