The EC2 Spot advisor which is available in the url - https://us-west-2.console.aws.amazon.com/ec2sp/v1/spot/advisor?region=us-west-2 provides recommendation on which instance-type in an availability zone has the least likelihood of getting intervened(Terminated) and the spot price.
Is there a programmable way(api/sdk, etc) to get this data from a script to automate provisioning instances using this data?
I am having the same issue, I noticed in the network tab that the data is coming from an API get_advice, but I cannot find that API anywhere in the AWS SDK/CLI docs.
GET https://us-west-2.console.aws.amazon.com/ec2sp/services/get_advice?product=linux&cheapest=90&min_cpu=2&min_ram=3&same_size=false¶llelability=90&az=any
It returns machine-parseable JSON data like this:
[
{
"instanceType": "t2.medium",
"azs": [
"us-west-2a",
"us-west-2b",
"us-west-2c"
],
"averagePrice": 0.0139,
"pricePerUnit": 0.0139,
"downtime": null,
"terminationProbability": 0,
"averageQuality": 0.01251
},
{
"instanceType": "t3.2xlarge",
"azs": [
"us-west-2a",
"us-west-2b",
"us-west-2c"
],
"averagePrice": 0.1002,
"pricePerUnit": 0.0251,
"downtime": null,
"terminationProbability": 0,
"averageQuality": 0.024372972972972973
},
{
"instanceType": "t2.large",
"azs": [
"us-west-2a",
"us-west-2b"
],
"averagePrice": 0.0278,
"pricePerUnit": 0.0278,
"downtime": null,
"terminationProbability": 0,
"averageQuality": 0.02502
}
]
This could be used to populate LaunchSpecifications in request-spot-fleet, however the API is probably not stable, and needs a console authentication token.
Related
I have private snapshots in one account (source) that I have shared with another account (target). I am able to see the snapshots themselves from the target account, but the tags are not available, neither on the console nor via the cli. This makes it impossible to filter for a desired snapshot from the target account. For background, the user in the target account has the following policy in effect:
"Effect": "Allow",
"Action": "ec2:*",
"Resource": "*"
Here's an example of what I'm seeing; from the source account:
$ aws --region us-east-2 ec2 describe-snapshots --snapshot-ids snap-XXXXX
{
"Snapshots": [
{
"Description": "snapshot for testing",
"VolumeSize": 50,
"Tags": [
{
"Value": "test-snapshot",
"Key": "Name"
}
],
"Encrypted": true,
"VolumeId": "vol-XXXXX",
"State": "completed",
"KmsKeyId": "arn:aws:kms:us-east-2:XXXXX:key/mrk-XXXXX",
"StartTime": "2022-04-19T18:29:36.069Z",
"Progress": "100%",
"OwnerId": "XXXXX",
"SnapshotId": "snap-XXXXX"
}
]
}
but from the target account
$ aws --region us-east-2 ec2 describe-snapshots --owner-ids 012345678900 --snapshot-ids snap-11111111111111111
{
"Snapshots": [
{
"Description": "snapshot for testing",
"VolumeSize": 50,
"Encrypted": true,
"VolumeId": "vol-22222222222222222",
"State": "completed",
"KmsKeyId": "arn:aws:kms:us-east-2:012345678900:key/mrk-00000000000000000000000000000000",
"StartTime": "2022-04-19T18:29:36.069Z",
"Progress": "100%",
"OwnerId": "012345678900",
"SnapshotId": "snap-11111111111111111"
}
]
}
Any ideas on what's going on here?
Cheers!
From https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Using_Tags.html#tag-restrictions
When you tag public or shared resources, the tags you assign are
available only to your AWS account; no other AWS account will have
access to those tags. For tag-based access control to shared
resources, each AWS account must assign its own set of tags to control
access to the resource.
I'm looking for a way to understand if we are making use of a specific VPC
The easy way is to review resources 1-by-1 like:
EC2 Machines
RDS
Client-VPN-Endpoint
Other resources - What else do I need to check?
And check manually.
is there another way to determine what is relying on a specific VPC before I'll delete it?
You can do it in two ways: AWS CLI or AWS console.
AWS CLI
You can use AWS CLI to list all ENIs associated with the VPC and prettify the output using the --query parameter to get a resource list with the desired fields (AZ, instance-id, etc.).
`aws ec2 describe-network-interfaces --filters Name=vpc-id,Values=<vpc-id> --query 'NetworkInterfaces[*].[AvailabilityZone, OwnerId, Attachment.InstanceId, PrivateIpAddresses[*].Association.PublicIp]'
`aws ec2 describe-network-interfaces --filters Name=vpc-id,Values=<vpc-id> --query 'NetworkInterfaces[*].[RequesterId,Description]'
A sample of the raw output (only one instance on the VPC):
"NetworkInterfaces": [
{
"Association": {
"IpOwnerId": "amazon",
"PublicDnsName": "ec2-54-196-57-169.compute-1.amazonaws.com",
"PublicIp": "54.196.57.169"
},
"Attachment": {
"AttachTime": "2020-08-24T10:59:16+00:00",
"AttachmentId": "eni-attach-047e562690aabbffd",
"DeleteOnTermination": true,
"DeviceIndex": 0,
"InstanceId": "i-0fe495a6c17bd0f82",
"InstanceOwnerId": "570398916848",
"Status": "attached"
},
"AvailabilityZone": "us-east-1d",
"Description": "",
"Groups": [
{
"GroupName": "launch-wizard-1",
"GroupId": "sg-0aa7d8257bb487e1b"
}
],
"InterfaceType": "interface",
"Ipv6Addresses": [],
"MacAddress": "0e:58:38:33:9a:31",
"NetworkInterfaceId": "eni-0b20855178d276783",
"OwnerId": "570398916848",
"PrivateDnsName": "ip-172-31-34-30.ec2.internal",
"PrivateIpAddress": "172.31.34.30",
"PrivateIpAddresses": [
{
"Association": {
"IpOwnerId": "amazon",
"PublicDnsName": "ec2-54-196-57-169.compute-1.amazonaws.com",
"PublicIp": "54.196.57.169"
},
"Primary": true,
"PrivateDnsName": "ip-172-31-34-30.ec2.internal",
"PrivateIpAddress": "172.31.34.30"
}
],
"RequesterManaged": false,
"SourceDestCheck": true,
"Status": "in-use",
"SubnetId": "subnet-e2bc5fbd",
"TagSet": [],
"VpcId": "vpc-6ad2e110"
}
]
And now filtered:
For the first --query
[
"us-east-1d",
"57039816848",
"i-0fe495a6c17bd0f82",
[
"44.196.57.169"
]
]
And for the second --query (another VPC):
[
"amazon-elasticache",
"ElastiCache alon-001"
],
[
"amazon-elasticache",
"ElastiCache alon-002"
],
[
"975289786086",
"arn:aws:ecs:us-east-2:57039916848:attachment/22a90802-fae7-4afb-9a7e-43e6f4be8ca4"
],
[
"074689309192",
"Interface for NAT Gateway nat-069344579d8bda20"
],
[
"amazon-elb",
"ELB app/EC2Co-EcsEl-YX74WCWEGOK/0b6d7bc60b540b1"
],
[
"amazon-elb",
"ELB app/EC2Co-EcsEl-YX74WCWGGOK/0b6bd7c60b540b1"
],
[
"amazon-elasticache",
"ElastiCache alon-003"
]
AWS Console
You can do the same using the AWS console.
Under EC2->Network Interfaces, search for the desired vpc-id in the search bar.
I am looking to see if there is an official API for retrieving metadata for information related to IAM permissions, services, etc.
The policy simulator console has pretty much everything I need. Looking at the HTTP requests made, there are requests that return, what looks like, a well defined JSON object.
For instance, to retrieve all of the available services, it calls:
https://policysim.aws.amazon.com/home/data/resource
and returns (shorted for brevity):
[
{
"name": "Amazon EC2",
"actionPrefix": "ec2",
"hasResource": true,
"arnFormat": "arn:aws:ec2:<region>:<account>:<resourceType>/<resourcePath>"
},
{
"name": "Amazon Sumerian",
"actionPrefix": "sumerian",
"hasResource": false,
"arnFormat": "arn:aws:sumerian:<region>:<account-id>:<resource-type>/<resource_name>"
},
{
"name": "Amazon Comprehend",
"actionPrefix": "comprehend",
"hasResource": false,
"arnFormat": "arn:${Partition}:comprehend:${Region}:${AccountId}:${ResourceType}/${ResourceName}"
},
...
{
"name": "AWS Elemental MediaStore",
"actionPrefix": "mediastore",
"hasResource": true,
"arnFormat": "arn:aws:mediastore:<Region>:<Account>:<Resource>"
}
]
And then for example, API Gateway, calling
https://policysim.aws.amazon.com/home/data/action?serviceName=Amazon+API+Gateway&servicePrefix=execute-api
returns:
[
{
"name": "InvalidateCache",
"evaluated": false,
"allowed": false,
"prefix": "execute-api",
"service": "Amazon API Gateway",
"serviceAware": true,
"resourceEnabled": true,
"deniedByOrganization": false,
"requiredResourceNames": [
"execute-api-general"
],
"supportedResources": [
{
"Name": "execute-api-general",
"ARN": "arn:aws:execute-api:${Region}:${Account}:${ApiId}/${Stage}/${Method}/${ApiSpecificResourcePath}",
"RegEx": "^arn:aws:execute-api:.+",
"isRequired": true
}
],
"scenarios": [],
"contextKeys": []
},
{
"name": "Invoke",
"evaluated": false,
"allowed": false,
"prefix": "execute-api",
"service": "Amazon API Gateway",
"serviceAware": true,
"resourceEnabled": true,
"deniedByOrganization": false,
"requiredResourceNames": [
"execute-api-general"
],
"supportedResources": [
{
"Name": "execute-api-general",
"ARN": "arn:aws:execute-api:${Region}:${Account}:${ApiId}/${Stage}/${Method}/${ApiSpecificResourcePath}",
"RegEx": "^arn:aws:execute-api:.+",
"isRequired": true
}
],
"scenarios": [],
"contextKeys": []
}
]
This is all of the information I am after. The list of service name and then the actions for each of them, along with some of that other metadata.
Obviously, I want to use an official API that would support IAM credentials and policies but wasn't able to find anything outside of the supported basic SimulatePolicy and context related API endpoints.
Thanks!
Pink
Just as a follow up in case anyone is looking for the same thing, according to AWS support, there is currently no official or supported API to retrieve this information.
I have added an EMR cluster to a stack. After updating the stack successfully (CloudFormation), I can see the master and slave nodes in EC2 console and I can SSH into the master node. But AWS console does not show the new cluster. Even aws emr list-clusters doesn't show the cluster. I have triple checked the region and I am certain I'm looking at the right region.
Relevant CloudFormation JSON:
"Spark01EmrCluster": {
"Type": "AWS::EMR::Cluster",
"Properties": {
"Name": "Spark01EmrCluster",
"Applications": [
{
"Name": "Spark"
},
{
"Name": "Ganglia"
},
{
"Name": "Zeppelin"
}
],
"Instances": {
"Ec2KeyName": {"Ref": "KeyName"},
"Ec2SubnetId": {"Ref": "PublicSubnetId"},
"MasterInstanceGroup": {
"InstanceCount": 1,
"InstanceType": "m4.large",
"Name": "Master"
},
"CoreInstanceGroup": {
"InstanceCount": 1,
"InstanceType": "m4.large",
"Name": "Core"
}
},
"Configurations": [
{
"Classification": "spark-env",
"Configurations": [
{
"Classification": "export",
"ConfigurationProperties": {
"PYSPARK_PYTHON": "/usr/bin/python3"
}
}
]
}
],
"BootstrapActions": [
{
"Name": "InstallPipPackages",
"ScriptBootstrapAction": {
"Path": "[S3 PATH]"
}
}
],
"JobFlowRole": {"Ref": "Spark01InstanceProfile"},
"ServiceRole": "MyStackEmrDefaultRole",
"ReleaseLabel": "emr-5.13.0"
}
}
The reason is missing VisibleToAllUsers property, which defaults to false. Since I'm using AWS Vault (i.e. using STS AssumeRole API to authenticate), I'm basically a different user every time, so I couldn't see the cluster. I couldn't update the stack to add VisibleToAllUsers either as I was getting Job flow ID does not exist.
The solution was to login as root user and fix things from there (I had to delete the cluster manually, but removing it from the stack template JSON and updating the stack would probably have worked if I hadn't messed things up already).
I then added the cluster back to the template (with VisibleToAllUsers set to true) and updated the stack as usual (AWS Vault).
I'm studying AWS api to retrieve requisite information about my EC2 instances.
So, I'm on AWS Cost Explorer Service.
It has function 'GetCostAndUsage' that, for example, sends information below. (this is an example from official AWS document)
{
"TimePeriod": {
"Start":"2017-09-01",
"End": "2017-10-01"
},
"Granularity": "MONTHLY",
"Filter": {
"Dimensions": {
"Key": "SERVICE",
"Values": [
"Amazon Simple Storage Service"
]
}
},
"GroupBy":[
{
"Type":"DIMENSION",
"Key":"SERVICE"
},
{
"Type":"TAG",
"Key":"Environment"
}
],
"Metrics":["BlendedCost", "UnblendedCost", "UsageQuantity"]
}
and retrieve information below. (this is an example from official AWS document)
{
"GroupDefinitions": [
{
"Key": "SERVICE",
"Type": "DIMENSION"
},
{
"Key": "Environment",
"Type": "TAG"
}
],
"ResultsByTime": [
{
"Estimated": false,
"Groups": [
{
"Keys": [
"Amazon Simple Storage Service",
"Environment$Prod"
],
"Metrics": {
"BlendedCost": {
"Amount": "39.1603300457",
"Unit": "USD"
},
"UnblendedCost": {
"Amount": "39.1603300457",
"Unit": "USD"
},
"UsageQuantity": {
"Amount": "173842.5440074444",
"Unit": "N/A"
}
}
},
{
"Keys": [
"Amazon Simple Storage Service",
"Environment$Test"
],
"Metrics": {
"BlendedCost": {
"Amount": "0.1337464807",
"Unit": "USD"
},
"UnblendedCost": {
"Amount": "0.1337464807",
"Unit": "USD"
},
"UsageQuantity": {
"Amount": "15992.0786663399",
"Unit": "N/A"
}
}
}
],
"TimePeriod": {
"End": "2017-10-01",
"Start": "2017-09-01"
},
"Total": {}
}
]
}
The retrieved data in key 'Metrics' I guess, it is total cost. not each.
So, How can I get each usage and cost for each EC2 instance??
This was way harder than I had imagined so I'm sharing in case someone else needs it.
aws ce get-cost-and-usage \
--filter file://filters.json \
--time-period Start=2021-08-01,End=2021-08-14 \
--granularity DAILY \
--metrics "BlendedCost" \
--group-by Type=TAG,Key=Name
Contents of filters.json:
{
"Dimensions": {
"Key": "SERVICE",
"Values": [
"Amazon Elastic Compute Cloud - Compute"
]
}
}
--- Available Metrics ---
AmortizedCost
BlendedCost
NetAmortizedCost
NetUnblendedCost
NormalizedUsageAmount
UnblendedCost
UsageQuantity
Descriptions for most metrics except for usage: https://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/ce-advanced.html
Know this question is old, but you will need to use the GetCostAndUsageWithResources call, as opposed to GetCostAndUsage.
https://awscli.amazonaws.com/v2/documentation/api/latest/reference/ce/get-cost-and-usage-with-resources.html
It's going to be difficult to associate an exact cost with each instance - simple example, you have 2 instances of the same size - one reserved and one on-demand - you run both for 1/2 the month and then turn off one of them for the second 1/2 of the month.
You will pay for a reserved instance for the entire month and an on-demand instance for 1/2 the month - but which instance was reserved and which was on-demand? You can't tell; the concept of a reserved instance is just a billing concept, and is not associated with a particular instance.
You might be able to approximate what you are looking for - but there are limitations.
You can use tags to track the cost of resources. In the case of EC2 you can assign tags like Project: myprojcet or Application: myapp and in cost explorer then filter the expenses by tags and use the tag that has been put to track the expenses. If the instance at some point was covered by a reservation plan, the tag will only show you the cost of the period in which your expenses were not covered.