I am trying to write a script that would save me going to the web Console every time I need checking how many RDS instances & S3 buckets I can create.
In the RDS section, there is a DB Instances (20/50) title, I need to get both the 20 and the 50
In the S3 bucket section, there is the total number of buckets. The implied max number is 100, but my company has increased it to 150. I need the number and the limit (as above).
Is that possible?
You can use AWS CLI to get all of these values.
In the RDS section, there is a DB Instances (20/50) title, I need to get both the 20 and the 50
Get the total number of RDS instances you can create:
aws service-quotas get-service-quota \
--service-code rds \
--quota-code L-7B6409FD
Get the current number of running RDS instances:
aws rds describe-db-instances --region us-east-1 | grep DBInstanceIdentifier | wc -l
In the S3 bucket section, there is the total number of buckets. The implied max number is 100, but my company has increased it to 150. I need the number and the limit (as above).
Get the total number of S3 buckets you can create:
aws service-quotas get-service-quota \
--service-code s3 \
--quota-code L-DC2B2D3D
Get the current number of S3 bucket:
aws s3api list-buckets | grep -w Name | wc -l
Related
With the AWS CLI ec2 describe-instance-types I can get a list of all the EC2 instance types but Sagemaker Instance Types like ml.t3.medium, ml.t3.large, ml.m5.xlarge, etc are not there.
aws ec2 describe-instance-types --filters "Name=instance-type,Values=ml.*" --query "InstanceTypes[].{Type: InstanceType, MaxENI: NetworkInfo.MaximumNetworkInterfaces, IPv4addr: NetworkInfo.Ipv4AddressesPerInterface}" --output table
# returns no results
I know I can get the list of SageMaker Instance Types from https://aws.amazon.com/sagemaker/pricing/ but I really want to get it programmatically.
How can I get programatically the list of instance types supported in Sagemaker for a given region?
You can get the list of ml instances with this CLI call (doc):
aws pricing get-products --service-code AmazonSageMaker --filters Type=TERM_MATCH,Field=location,Value="US East (N. Virginia)"
You'll need to filter the results further.
Note that a particular ml instance type might be available for a certain SageMaker feature like training, but not for inference. And be available in one region but not another.
If you're end goal is to get technical details, you could first fine the relevant ml.* instances (maybe even with regex), then use the EC2 describe instance type to get more details (just strip down the ml. prefix).
You can find relevant Python code in the "Total Cost" section of my notebook here.
Building upon the answer from Gili.
Using the aws CLI and jq :
aws --region us-east-1 pricing get-products \
--service-code AmazonSageMaker \
--filters Type=TERM_MATCH,Field=regionCode,Value=eu-north-1 \
| jq -r '.PriceList[]|fromjson|select(.product.productFamily == "ML Instance")|.product.attributes.instanceName'\
|sort\
|uniq
ml.c5.12xlarge
ml.c5.18xlarge
ml.c5.24xlarge
ml.c5.2xlarge
ml.c5.4xlarge
ml.c5.9xlarge
ml.c5.large
...
ml.t3.2xlarge
ml.t3.large
ml.t3.medium
ml.t3.xlarge
it uses
aws pricing get-products
--region us-east-1 is important because the Pricing service is not widely available
--filters Type=TERM_MATCH,Field=regionCode,Values=eu-north-1 to restrict the listing to products in eu-north-1 region, the number of products for AmazonSageMaker alone across all regions is huge, so better to let AWS to filter out those early
jq is used to further filter the output, it seems it's not possible to filter by productFamily at the aws pricing get-products so we need to do it with jq
-r removes the quotes from the output
.PriceList[] will iterate over all the prices returned by aws pricing get-products
fromjson will parse the each string as JSON (.PriceList is an array of strings)
select(.product.productFamily == "ML Instance") will filter out all other products
.product.attributes.instanceName extracts the instance type from each product.
I want to use boto3 to get list of available aws services in a specific region. Is there any way to do this.
I tried using Session object:
session = boto3.Session(region_name='ap-south-1').get_available_services()
but it is giving me all the AWS services. For eg: Cloudsearch is not present in ap-south-1, but this function still gives me the service in the output.
Also, I don't want to use ssm service get_parameters_by_path function as I don't want to give ssm permission.
Any other way?
To be frank, I reckon, your best bet actually is the Systems Manager Parameter Store.
For example, you can easily display a complete list of all available AWS services, sort them into alphabetical order, and, for the brevity, show the first 10.
$ aws ssm get-parameters-by-path \
--path /aws/service/global-infrastructure/services --output json | \
jq '.Parameters[].Name' | sort | head -10
Output:
"/aws/service/global-infrastructure/services/acm"
"/aws/service/global-infrastructure/services/acm-pca"
"/aws/service/global-infrastructure/services/alexaforbusiness"
"/aws/service/global-infrastructure/services/apigateway"
"/aws/service/global-infrastructure/services/application-autoscaling"
"/aws/service/global-infrastructure/services/appmesh"
"/aws/service/global-infrastructure/services/appstream"
"/aws/service/global-infrastructure/services/appsync"
"/aws/service/global-infrastructure/services/athena"
"/aws/service/global-infrastructure/services/autoscaling"
And here's how to get the list of services that are available in a given region. Show first 10 and sorted.
$ aws ssm get-parameters-by-path \
--path /aws/service/global-infrastructure/regions/us-east-1/services --output json | \
jq '.Parameters[].Name' | sort | head -10
But... if you want any other way you might want to try AWS Price List API.
With the AWS Price List Query API, you can query specific information about AWS services, products, and pricing using an AWS SDK or the AWS CLI.
This obviously can be narrowed down to a specific region. If there's a price, there is a service.
I got this by below code:
resp = boto3.Session().get_available_regions('cloudsearch')
This gave me the list of all the regions where cloudsearch service is available.
I need to get information such as VPCs, subnets, security groups, etc for many AWS accounts at once. How can I go about this?
One solution is to use a for loop with the AWS CLI. Check out the CLI Documentation for the service that you're wanting to gather information for and find the appropriate commands then use a for loop to loop over the profiles in your ~/.aws/credentials file.
For example, if you're wanting to get the VPCs, subnets, and security groups, those are all described in the EC2 CLI docs.
Here is an example of getting information about those resources and outputting it into the current directory as .json (this assumes you didn't change the default output format when using aws configure
#!/usr/bin/env bash
region=us-east-1
for profile in `grep [[] ~/.aws/credentials | tr -d '[]'`
do
echo "getting vpcs, subnets, and security groups for $profile"
aws ec2 describe-vpcs --region $region --profile $profile > "$profile"_vpcs.json
aws ec2 describe-subnets --region $region --profile $profile > "$profile"_subnets.json
aws ec2 describe-security-groups --region $region --profile $profile > "$profile"_security_groups.json
done
I am using the AWS CLI to fetching EC2 instance details. My requirements are:
1) Fetch only running EC2 instances:
instance id
instance tag
instance type
availability zone
blockdevices name
platform
reserved or ondemand
However I can't able to fetch last 3 attributes (blockdevices name, platform and reserved or ondemand). When I add it in my query it is showing None. What do I need to change in query?
My query is:
aws ec2 describe-instances --filters "Name=instance-state-name,Values=running" --query "Reservations[].Instances[].[Tags[?Key==`Name`] | [0].Value,InstanceId,State.Name,InstanceType,Placement.AvailabilityZone]" --output table > output.txt
The output is:
| test1 | i-xxxxxxx | m1.small | running | us-east-1a |
| test2 | i-xxxxx | m1.large | running | us-east-1c |
| test3 | i-xxxxx | m1.xlarge | running | us-east-1c |
| test4 | i-xxxxxxx | m3.2xlarge | running | us-east-1a |
2) Need to fetch each instance price considering those attributes (region,type,ondemand/reserved) and put full instance details in a CSV file.
Blockdevices name
I'm not sure what you mean by "name", but there is a DeviceName field:
aws ec2 describe-instances --query Reservations[*].Instances[*].BlockDeviceMappings[*].DeviceName
Platform
The Platform field is either Windows or undefined (for Linux):
aws ec2 describe-instances --query Reservations[*].Instances[*].Platform
Reserved or On-Demand
This is a billing concept and is not available for an instance. In fact, an instance could be billed as On-Demand in one hour and as a Reserved Instance in another hour.
The basic concept is that, for each Reserved Instance owned by an account, the billing system looks for a running Amazon EC2 instance that matches the Reserved Instance parameters. The matching instance is not charged the hourly fee for that hour. This information can be seen in the billing files, but is not available as information against the instance.
Make sure you have turned on Detailed Billing Reports so that you can receive this level of billing information. It is only available after the reports have been activated (not back in time).
Fetch the price
There is no command to fetch the price of a running Amazon EC2 instance. There is a Price List API that provides pricing information similar to the pricing pages on the AWS website.
You could write a program to fetch the prices via the Price List API and determine which price is appropriate for an instance. However, as stated above, the program would not know whether an instance was running as a Reserved Instance.
Put full instance details in a CSV file
The AWS CLI does not output CSV format. You would have to post-process the output into the desired format. Note that some of your information (eg BlockDevices) might return multiple values for a single instance.
I have some EC2 instances that I don't really know who launched them.
Is there a way to know who launched a specific instance?
Unfortunately this information is not directly available via an API call - you currently have two options:
depending on your needs, you could approximate your goal by using the DescribeInstances API action to look at the key-name used for starting that instance (if any, it's optional, though usually in place) - assuming you have followed security best practices and are using a dedicated EC2 key pair per IAM user (rather than sharing keys), the key should usually denote the user who started the instance ;)
that's most easily tested via the AWS Command Line Interface, specifically describe-instances
nowadays you could activate AWS CloudTrail, which records AWS API calls for your account and delivers log files to you and provides exactly the information you are after:
The recorded information includes the identity of the API caller, the
time of the API call, the source IP address of the API caller, the
request parameters, and the response elements returned by the AWS
service.
AWS CloudTrail is a long overdue and invaluable tool and provides unprecedented insight into your AWS usage; it is easy to get started and only has a few remaining caveats currently:
most importantly, it isn't available in all regions yet, but AWS has just extended it to 3 more for a total of 5, see AWS CloudTrail Expands Again - More Locations and Services, thus quickly approaching coverage of their entire Global Infrastructure
not all services are covered yet, but AWS hast just extended it to 7 more for a total of 15, see AWS CloudTrail Update - Seven New Services
depending on your needs, you most likely want to have a Logging as a Service (LaaS) solution in place to ease digging through the vast amount of logs, and provide alerts etc. - several providers already offer dedicated CloudTrail integration (and usually a free tier sufficient for that as well)
events are delivered within 15 minutes of the API call and the resulting logs to your S3 bucket approximately every 5 minutes, resulting in a maximum delay of 20 minutes, which is enough for post hoc and batch analysis, but not sufficient for near real-time alerting of course (see the AWS CloudTrail FAQ)
Assuming that you have configured CloudTrail already in your console (to monitor the right events) here's a possible solution to retrieve the username of who launched a certain EC2 instance, given the instance ID.
This solution uses the AWS CLI to retrieve information about which instances were launched, and jq for parsing.
When calling the aws cloudtrail lookup-events function, we can specify:
The name of the event. In this case, RunInstances.
The date from which to start searching, start-date. This can be when the EC2 instance was launched.
The date up to which to search, end-date. This can be just one minute after the instance was started.
First, let's define the instance id and the region variables:
$ instance_id="i-08dcc12c018737061"
$ region="us-east-1"
then, let's find out when the instance was launched:
$ instance_launch_date=$(aws ec2 describe-instances --instance-ids $instance_id --region $region | jq '.Reservations[].Instances[].LaunchTime')
$ echo $instance_launch_date
"2020-07-08T15:21:02.000Z"
now, let's sanitize the date so that it's in the right format for the CloudTrail API.
$ launch_date=$(sed -E 's/\"([0-9]{4})-([0-9]{2})-([0-9]{2}).([0-9]{2}):([0-9]{2}):([0-9]{2}).+/\2\/\3\/\1 \4:\5:\6/' <<< $instance_launch_date)
$ start_date=$(date -d "${launch_date}" +"%s")
$ echo $start_date
1594221662
now, let's define the end date as the date the instance was launched plus 1 minute:
$ end_date=$(date -d "${launch_date} 1 min" +"%s")
echo $end_date
1594221722
and now let's call the CloudTrail API, and filter the output with jq:
aws cloudtrail lookup-events --lookup-attributes AttributeKey=EventName,AttributeValue=RunInstances --start-time $start_date --end-time $end_date --region $region | jq --arg INSTANCE_ID "$instance_id" '.Events[] | select(.CloudTrailEvent | fromjson | .responseElements.instancesSet.items[].instanceId | contains($INSTANCE_ID)) | .Username '
"paolo"