Is there any way to find unused VPCs in an AWS account?
I mean the VPCs that don't have any EC2 instances, RDS and other services associated with it.
One way is to just search with VPC ID in running instances, RDS and for other services to find out whether it is in use or not. Is there any other way or AWS CLI to find unused VPCs?
There are many resources that be included in a VPC, such as:
Amazon EC2 instances
Amazon RDS instances
Amazon Redshift instances
Amazon Elasticache instances
Elastic Load Balancers
Elastic Network Interfaces
and so on!
Rather than trying to iterate through each of these services, you could iterate through the Elastic Network Interfaces (ENIs), since everything connects to a VPC via an ENI.
Here's a command you could run using the AWS Command-Line Interface (CLI) that shows ENIs attached to a given VPC:
aws ec2 describe-network-interfaces --filters 'Name=vpc-id,Values=vpc-abcd1234' --query 'NetworkInterfaces[*].NetworkInterfaceId'
If no ENIs are returned, then you'd probably call it an unused VPC.
This might sound crazy, but I am pretty sure you can attempt to delete the VPC. It should protect from deletion any VPC that has resources running in it. Of course, you should give this a quick try before you do it. But its probably the fastest/cleanest.
Please use the following script to identify the unused Subnets for your AWS accounts in all regions:
USAGE:
Please add Account list in accounts variable as accounts=["a1","a2","a3"]
It will query and provide the list of subnets in all the regions for
respective accounts
A single CSV file will be created at end of each run for one account
Logic:
Query all the subnets across all the regions for an AWS account
Get currently available IP details for the subnet(It is provided by AWS API)
Get Subnet CIDR, calculate total IPs count, and subtract 5 counts (5
because 2 are used for Network and Broadcast and the other 3 are
reserved by AWS by default)
Then, Subtract Total IPs - Available = Currently used IP. If Used IP
= 0 , subnet can be cleaned
import boto3
import sys
import csv
import ipaddress
def describe_regions(session):
try:
aws_regions = []
ec2_client = session.client('ec2')
response_regions = ec2_client.describe_regions()['Regions']
for region in response_regions:
aws_regions.append(region['RegionName'])
return aws_regions
except Exception:
print("Unexpected error:", sys.exc_info()[0])
def describe_vpc(ec2,aws_region,writer,profile_name):
try:
response_vpc = ec2.describe_vpcs()['Vpcs']
for vpc in response_vpc:
print('=' * 50)
count = 0
filters = [
{'Name': 'vpc-id',
'Values': [vpc['VpcId']]}
]
response_subnets = ec2.describe_subnets(Filters=filters)['Subnets']
for subnets in response_subnets:
count += 1
total_count = (ipaddress.ip_network(subnets['CidrBlock']).num_addresses) - 5
Used_IP = total_count - subnets['AvailableIpAddressCount']
writer.writerow({"Account": profile_name, "VpcId": vpc['VpcId'], "VpcCidr": vpc['CidrBlock'], "Region": aws_region,
"Subnet": subnets['CidrBlock'], "SubnetId": subnets['SubnetId'], "AvailableIPv4": subnets['AvailableIpAddressCount'], "Total_Network_IP": str(total_count),
"AvailabilityZone": subnets['AvailabilityZone'],"Used_IP": str(Used_IP)})
print({"Account": profile_name, "VpcId": vpc['VpcId'], "VpcCidr": vpc['CidrBlock'], "Region": aws_region,
"Subnet": subnets['CidrBlock'], "SubnetId": subnets['SubnetId'], "AvailableIPv4": subnets['AvailableIpAddressCount'], "Total_Network_IP": str(total_count),
"AvailabilityZone": subnets['AvailabilityZone'],"Used_IP": str(Used_IP)})
print('='*50)
except Exception:
print("Unexpected error:", sys.exc_info()[0])
def main():
try:
accounts=["<Account names here as list>"]
for profile in accounts:
session = boto3.session.Session(
profile_name=profile
)
file_name = profile
print("File Name: " +file_name)
profile_name = profile
print("Profile_name: " +profile_name)
with open(file_name + ".csv", "w", newline="") as csvfile:
fieldnames = [
"Account", "VpcId",
"VpcCidr", "Region",
"Subnet", "SubnetId",
"AvailableIPv4","Total_Network_IP",
"AvailabilityZone","Used_IP"
]
writer = csv.DictWriter(csvfile, fieldnames=fieldnames)
writer.writeheader()
aws_regions = describe_regions(session)
for aws_region in aws_regions:
ec2 = session.client('ec2', region_name=aws_region)
print("Scanning region: {}".format(aws_region))
describe_vpc(ec2,aws_region, writer, profile_name)
except Exception:
print("Unexpected error:", sys.exc_info()[0])
raise
if __name__ == "__main__":
main()
This AWS Knowledge Center post will give good help. It contains even better aws-cli commands to use. https://aws.amazon.com/premiumsupport/knowledge-center/troubleshoot-dependency-error-delete-vpc/
Related
So, I have this code, and I will love to grab the public IP address of the new windows instance that will be created when I adjust the desired capacity.
The launch template assigns an automatic tag name when I adjust the desired_capacity. I want to be able to grab the public IP address of that tag name.
import boto3
session = boto3.session.Session()
client = session.client('autoscaling')
def set_desired_capacity(asg_name, desired_capacity):
response = client.set_desired_capacity(
AutoScalingGroupName=asg_name,
DesiredCapacity=desired_capacity,
)
return response
def lambda_handler(event, context):
asg_name = "test"
desired_capacity = 1
return set_desired_capacity(asg_name, desired_capacity)
if __name__ == '__main__':
print(lambda_handler("", ""))
I took a look at the EC2 client documentation, and I wasn't sure what to use. I just need help modifying my code
If you know the tag that you are assigning in the autoscaling group, then you can just use a describe_instances method. The Boto3 docs have an example with filtering. Something like this should work, replacing TAG, VALUE, and TOPICARN with the appropriate values.
import boto3
ec2_client = boto3.client('ec2', 'us-west-2')
sns_client = boto3.client('sns', 'us-west-2')
response = ec2_client.describe_instances(
Filters=[
{
'Name': 'tag:TAG',
'Values': [
'VALUE'
]
}
]
)
for reservation in response["Reservations"]:
for instance in reservation["Instances"]:
ip = instance["PublicIpAddress"]
sns_publish = sns_client.publish(
TopicArn='TOPICARN',
Message=ip,
)
print(sns_publish)
Objective:
After an EC2 instance starts
Obtain the IP address
Send a message via Amazon SNS
It can take some time for a Public IP address to be assigned to an Amazon EC2 instance. Rather than continually calling DescribeInstances(), it would be easier to Run commands on your Linux instance at launch - Amazon Elastic Compute Cloud via a User Data script.
The script could:
Obtain its Public IP address via Instance metadata and user data - Amazon Elastic Compute Cloud:
IP=$(curl 169.254.169.254/latest/meta-data/public-ipv4)
Send a message to an Amazon SNS topic with:
aws sns publish --topic-arn xxx --message $IP
If you also want the message to include a name from a tag associated with the instance, the script will need to call aws ec2 describe-instances with its own Instance ID (which can be obtained via the Instance Metadata) and then extra the name from the tags returned.
I am trying to write a query to return results of any referenced security group not owned by the current account.
This means I am trying to show security groups that are being used as part of a peering connection from another VPC.
There are a couple of restrictions.
Show the entire security group details (security group id, description)
Only show security groups where IpPermissions.UserIdGroupPairs has a Value and where that value is not equal to the owner of the security group
I am trying to write this using a single AWS CLI cmd vs a bash script or python script.
Any thoughts?
Heres what I have so far.
aws ec2 describe-security-groups --query "SecurityGroups[?IpPermissions.UserIdGroupPairs[*].UserId != '`aws sts get-caller-identity --query 'Account' --output text`']"
Following is Python 3.8 based AWS Lambda, but you can change a bit to use as python script file to execute on any supported host machine.
import boto3
import ast
config_service = boto3.client('config')
# Refactor to extract out duplicate code as a seperate def
def lambda_handler(event, context):
results = get_resource_details()
for resource in results:
if "configuration" in resource:
config=ast.literal_eval(resource)["configuration"]
if "ipPermissionsEgress" in config:
ipPermissionsEgress=config["ipPermissionsEgress"]
for data in ipPermissionsEgress:
for userIdGroupPair in data["userIdGroupPairs"]:
if userIdGroupPair["userId"] != "123456789111":
print(userIdGroupPair["groupId"])
elif "ipPermissions" in config:
ipPermissions=config["ipPermissions"]
for data in ipPermissions:
for userIdGroupPair in data["userIdGroupPairs"]:
if userIdGroupPair["userId"] != "123456789111":
print(userIdGroupPair["groupId"])
def get_resource_details():
query = "SELECT configuration.ipPermissions.userIdGroupPairs.groupId,configuration.ipPermissionsEgress.userIdGroupPairs.groupId,configuration.ipPermissionsEgress.userIdGroupPairs.userId,configuration.ipPermissions.userIdGroupPairs.userId WHERE resourceType = 'AWS::EC2::SecurityGroup' AND configuration <> 'null'"
results = config_service.select_resource_config(
Expression=query,
Limit=100
) # you might need to refacor to add support huge list of records using NextToken
return results["Results"]
How can I get the instance ID and private IP for EC2 instance deployed with AutoscalingGroup (AWS CDK Python) ?
The AutoscalingGroup Construct is like this:
from aws_cdk import (
core,
aws_ec2,
aws_autoscaling
)
autoscaling_group = aws_autoscaling.AutoScalingGroup(
self,
id="AutoscalingGroup",
instance_type=aws_ec2.InstanceType('m5.xlarge'),
machine_image=aws_ec2.MachineImage.latest_amazon_linux(),
vpc=Myvpc,
vpc_subnets=aws_ec2.SubnetSelection(subnet_type=aws_ec2.SubnetType.PUBLIC),
associate_public_ip_address=True,
desired_capacity=1,
key_name='MySSHKey'
)
Thank you very much.
You can retrieve them using boto3.
Here is an example to get them only for the running instances :
ec2_res = boto3.resource('ec2')
instances = ec2_res.instances.filter(
Filters=[
{'Name': 'instance-state-name', 'Values': ['running']}
]
)
for instance in instances:
print(instance.id, instance.instance_type, instance.private_ip_address)
You can check the doc here for available parameters and here for the boto3 call.
If you want to filter on a specific name, you have to check in the tags of the instances:
for instance in instances:
for tag in instance.tags:
if (tag.get('Key') == 'Name') and (tag.get('Value') == '<The name of your instance>'):
print(instance.id, instance.instance_type, instance.private_ip_address)
I am creating new fargate task by using the following python script.
import boto3
import json
def handler():
client = boto3.client('ecs')
response = client.run_task(
cluster='fargate-learning', # name of the cluster
launchType = 'FARGATE',
taskDefinition='fargate-learning:1', # replace with your task definition name and revision
count = 1,
platformVersion='LATEST',
networkConfiguration={
'awsvpcConfiguration': {
'subnets': [
'subnet-0a024d8ac87668b64', # replace with your public subnet or a private with NAT
],
'assignPublicIp': 'ENABLED'
}
})
print(response)
return str(response)
if __name__ == '__main__':
handler()
And here is the response I am getting from boto3.
https://jsonblob.com/5faf3ae6-bc31-11ea-8cae-53bd90c38587
I can not see the public ip address in response although the script is assigning the public ip address and I can see it on website.
So, how can I get this public ip address by using boto3?
Thanks
This can be done in two steps:
Use describe_tasks to get ENI id associated with your fargate awsvpc interface. The value of eni, e.g. eni-0c866df3faf8408d0, will be given in attachments and details from the result of the call.
Once you have the eni, then you can use EC2.NetworkInterface. For example:
eni_id = 'eni-0c866df3faf8408d0' # from step 1
eni = boto3.resource('ec2').NetworkInterface(eni_id)
print(eni.association_attribute['PublicIp'])
Tried implementing #Marcin 's answers as a function. Hope this can be helpful
def get_service_ips(cluster, tasks):
tasks_detail = ecs.describe_tasks(
cluster=cluster,
tasks=tasks
)
# first get the ENIs
enis = []
for task in tasks_detail.get("tasks", []):
for attachment in task.get("attachments", []):
for detail in attachment.get("details", []):
if detail.get("name") == "networkInterfaceId":
enis.append(detail.get("value"))
# now the ips
ips = []
for eni in enis:
eni_resource = boto3.resource("ec2").NetworkInterface(eni)
ips.append(eni_resource.association_attribute.get("PublicIp"))
return ips
as a gist here.
rds_client = boto3.client('rds', 'us-east-1')
instance_info = rds_client.describe_db_instances( DBInstanceIdentifier='**myinstancename**')
But the instance_info doesn't contain any tags I set in the RDS instance. I want to fetch the instances that has env='production' in them and want to exclude env='test'. Is there any method in boto3 that fetched the tags as well?
Only through boto3.client("rds").list_tags_for_resource
Lists all tags on an Amazon RDS resource.
ResourceName (string) --
The Amazon RDS resource with tags to be listed. This value is an Amazon Resource Name (ARN). For information about creating an ARN, see Constructing an RDS Amazon Resource Name (ARN) .
import boto3
rds_client = boto3.client('rds', 'us-east-1')
db_instance_info = rds_client.describe_db_instances(
DBInstanceIdentifier='**myinstancename**')
for each_db in db_instance_info['DBInstances']:
response = rds_client.list_tags_for_resource(
ResourceName=each_db['DBInstanceArn'],
Filters=[{
'Name': 'env',
'Values': [
'production',
]
}])
Either use a simple exclusion over the simple filter, or you can dig through the documentation to build complicated JMESPath filter using
paginators.
Notes : AWS resource tags is not a universal implementation. So you must always refer to the boto3 documentation.
Python program will show you how to list all rds instance, there type and status.
list_rds_instances.py
import boto3
#connect ot rds instance
client = boto3.client('rds')
#rds_instance will have all rds information in dictionary.
rds_instance = client.describe_db_instances()
all_list = rds_instance['DBInstances']
print('RDS Instance Name \t| Instance Type \t| Status')
for i in rds_instance['DBInstances']:
dbInstanceName = i['DBInstanceIdentifier']
dbInstanceEngine = i['DBInstanceClass']
dbInstanceStatus = i['DBInstanceStatus']
print('%s \t| %s \t| %s' %(dbInstanceName, dbInstanceEngine, dbInstanceStatus))
Important Note: While working with boto3 you need to setup your credentials in two files ~/.aws/credentials and ~/.aws/config
~/.aws/credentials
[default]
aws_access_key_id=<ACCESS_KEY>
aws_secret_access_key=<SECRET_KEY>
~/.aws/config
[default]
region=ap-south-1