I am trying to clean up my AWS configuration and I want to know if particular subnets are actually used/have any resources in them.
I'm aware you can filter a list of a particular resource type (e.g. EC2 instances) by subnet id, through the AWS web interface, but I am not yet aware of all of the different resource types that may be used - so I am concerned I may miss something.
I have tried inspecting the subnet via the AWS CLI, but I can't see anything that clearly differentiates subnets that are in use and those that are not:
aws ec2 describe-subnets
This question deals with enumerating all IP addresses within a particular subnet's CIDR block, but it doesn't reveal how to show only active IP addresses (which I could presumably use to find the attached AWS resources and confirm a subnet is indeed in use).
This seems like it would be a common task, but I can find no AWS documentation or SO posts on how to do this. Perhaps there is something flawed in my approach.
aws ec2 describe-network-interfaces --filters Name=subnet-id,Values=subnet-id-here | grep Description (replace subnet-id-here with the subnet id in mind)
The above command will give you the names of resources in that subnet.
Take a look at aws ec2 describe-network-interfaces.
This returns a list of Elastic Network Interfaces (ENIs) and supports a subnet-id filter. EC2 instances aren't the only thing that can be on a subnet -- RDS instances, Elastic Load Balancers, Lambda functions, Elastic File System mount targets, NAT Gateways, and other resources consume IP addresses on a subnet, but in each case I can think of, they do this by allocating ENIs. In some cases, like load balancers (ALB and Classic), the number of addresses grows and shrinks as the balancer scales up and down in capacity. In the case of Lambda, a lack of allocated ENIs may only mean that no Lambda container hosts are currently using the subnet, due to a lack of traffic... so if you have VPC Lambda functions, bear that in mind.
You can also see ENIs in the EC2 console, under "Network Interfaces" in the left hand navigation pane.
Thank you for the responses - they were both helpful and indeed did help me identify whether particular subnets were in use or not.
The thing I found most useful to understanding what was in each subnet, however, was the open source Python visualisation tool, CloudMapper (I'm in no way affiliated - I discovered it after asking my question and scrolling through commercial visualisers).
The AWS CLI is a great tool but, if you're just trying to see what's in each subnet, AWS added a Network Interfaces section to the EC2 console. From there, you can filter by subnetID
AWS CLI filter and query switches
In the --filters switch Values argument, replace <<Subnet ID>> with your Subnet ID.
aws ec2 describe-network-interfaces \
--filters Name=subnet-id,Values=<<Subnet ID>> \
--query 'NetworkInterfaces[*].Description'
Related
I able to find that in my research there are some of the ways to find all our aws resources but will you suggest any solution to find public resources in aws our account.
Unfortunately, there is no easy way that you can find all resources that are allowed to communicate with outside world. However, the only way for most of the resources in AWS can connect to the internet through a VPC that is configured to allow access to 0.0.0.0/0 network with Security Group and ACL. So what you can do is to search for VPC ID on resources categories and if any results you get can connect to the internet.
This can be a very tedious task and this is NOT 100% guaranteed to find all resources as some resources has self-sustained VPC per resource such as S3. So my suggestion is you get used to using tags for resources.
Also maybe think about using infrastructure as code approach using something like Terraform.
If you have all of your instances in a VPC you could enable VPC Flow Logs on your VPC and then monitor all VPC traffic that way.
For the instances that are not in VPC's you could have CloudWatch logs enabled for that instance and then monitor the traffic that way.
Your instances should have security groups attached to them so that you could monitor the rules, and see if you have an OutBound Set to 0.0.0.0/0
You can give Netflix's Security Monkey a try. The tool will make a complete report of all secure and insecure AWS assets including Security Groups, IAM Policies and S3 Buckets. In fact, it also provides a ticketing like system to collab on such problems.
You can use IAM Access Analyzer console
I'm trying to secure a pipeline for analyzing controlled-access genomic data with Amazon Elastic MapReduce (EMR), and it would help to know the minimal set of outbound rules required of the master and slave security groups of an EMR cluster. I'm sure it differs from region to region, and the IP ranges given at http://docs.aws.amazon.com/general/latest/gr/aws-ip-ranges.html probably subsume them, but it would be great to know exactly which CIDR blocks we should worry about. It looks like EMR pokes just the right holes among the inbound rules for everything to work, but I've found the cluster gets stuck on provisioning if the outbound rules are anything other than "allow all traffic."
We had the identical problem. The way we addressed this problem is by doing the following.
From the ip-ranges.json, use the EC2 CIDR block & AMAZON service cidr block. You may substract CLOUDFRONT & ROUTE53 blocks.
The reason is you need to be able to talk to EMR webservice endpoints that are hsoted outside your VPC. EMR uses a subset of EC2 instances to spin up cluster.
If you have a support contract, ask Amazon to provide you with the CIDR block (we paid for a consulting engagement and this was one of the things they did).
Also, as the EMR webservice is on a public DNS endpoint (not 10.*), there should be a route to the internet gateway.
I have one VPC with an RDS instance in it. They are both located in the same region.
I want to use the RDS instance in another VPC, that is in another region on another AWS account (we have multiple AWS accounts). If that's not complicated enough the 2nd VPC comes up via CloudFormation (i.e. dynamic). Whenever I am bringing up a CloudFormation stack I want to attach the RDS instance automatically.
I have looked at:
exposing the RDS instance on the public internet :(
an ELB w/ TCP transport to put the database instance behind
VPC peering but the different regions and the approval workflow in the AWS console make little sense in the case we are using CloudFormation
All of these seem suboptimal to me and was wondering if somebody already did this before. If yes, please share what you did and what the though process behind it was.
Use a VPN tunnel from one VPC to the other. You could build your own or look at Vyatta. Ideally the two VPCs do not have overlapping CIDRs. Note that you cannot use VPC peering inter-region.
For anyone who stumbles around here, it looks like AWS VPC Peering can now be done cross region: https://docs.aws.amazon.com/vpc/latest/peering/what-is-vpc-peering.html
I am faced with a chicken and egg problem. I currently have a server in EC2 classic, as well as an RDS instance -- in EC2 classic as well. The EC2 instances also interact with Cassandra cluster, which also resides in EC2 classic.
However, I need to move RDS into the VPC. Now, in an ideal world, I'd have all of my stuff in VPC at this point. However, that presents a major migration challenge and I'd like to minimize impact on users and keep steps to minimum -- this is mainly because of the Cassandra cluster.
It turns out that I cannot create security group rules between VPC and Non-VPC security groups.
So, how can I have RDS in VPC that my EC2 instances can access w/o having to open up my RDS to the entire world ?
Any help is greatly appreciated.
UPDATE: So, one idea I had is to assign elastic IPs to my EC2 instances and add IPs explicitly to the security group for RDS within VPC. Would that work ? (trying it now using https://github.com/skymill/aws-ec2-assign-elastic-ip)
Yes, unfortunately that's the only way to do it. You cannot use DNS in security groups, so you're stuck with IP address.
So, I ended up solving it exactly like I described -- assign elastic IPs to my EC2 instances and add IPs explicitly to the security group for RDS within VPC. It ended up working great.
I have a pretty standard stack, RDS, 2 EC2 instances using ELB. Because I wanted the ELB to be restricted to a particular IP range I've launched the stack in VPC, for DR reasons across 2 subnets.
I use several ephemeral ec2 machines, which when not in VPC I allowed to startup in any availability zone. Which (as far as I understand) would mean that if an AZ were unavailable then machines would be started in other zones.
Is there a way to emulate this in VPC? Is there a way of saying launch a machine in any subnet in a VPC?
If not its fairly easy to workaround by picking a subnet at random, and if it fails trying another. Just wondered if there was a supported method that's cleaner?
I'm using python and boto.
thanks
Which (as far as I understand) would mean that if an AZ were unavailable then machines would be started in other zones.
That's correct and it would indeed be nice to have this option available within an Amazon VPC as well when running instances directly via the available Amazon EC2 API actions.
Unfortunately both the RunInstances and the RequestSpotInstances API actions only allow to specify the optional parameters SubnetId or LaunchSpecification.SubnetId respectively (The ID of the subnet in which to launch the [Spot] Instance), thus won't have any information into which VPC you would want to launch the instance if no subnet is specified.
Workaround
You can achieve the desired behavior indirectly via Auto Scaling by means of its CreateAutoScalingGroup API action, see parameter VPCZoneIdentifier:
A comma-separated list of subnet identifiers of Amazon Virtual Private Clouds (Amazon VPCs).
This feature is also available via the AutoScalingGroup resource type within AWS CloudFormation.