Using Powershell to Report AWS Security Groups - amazon-web-services

I am new to powershell and the AWS CLI and have look and looked for someone else who has posted this...
What I'm trying to do is get the code right to create a report of each AWS Security Group that shows the inbound rules.
Something like this I would image the output would be.
SecurityGroupName GroupID, Type, Protcol, PortRange, Source, Description
SSH & HTTP gs-1111 SSH TCP 22 0.0.0.0/0 Inbound SSH & HTTP
SSH & HTTP gs-1111 HTTP TCP 80 1.2.34 Inbound SSH & HTTP
HTTPS gs-2222 HTTPS TCP 443 0.0.0.0/0
'
But I can't figure it out.
I can use $GroupID=Get-EC2SecurityGroup -Region us-east-1 |Select-Object -ExpandProperty GroupID
ForEach ($item in $GroupID) {
(Get-EC2SecurityGroup -Region us-east-1 -GroupId $item).IpPermissions | Select-Object IPProtocol,IpRange, FromPort,ToPort}
to get the basic rule but can't seem to combine all the properties you need. This is where my lack of powershell really hurts. Any help would be great!

First, you need to create a new Powershell array that will hold all the objects.
Later, run over all the security groups in the specific region and add relevant data to the hash-table object.
Then, just add the object to the array we created earlier.
Code snippet:
$region = 'YOUR_REGION'
$allSG = #()
Get-EC2SecurityGroup -Region $region | % {
$obj = #{
Description = $_.IpPermission.Ipv4Ranges.Description
GroupId = $_.GroupId
SecurityGroupName = $_.GroupName
Source = $_.IpPermission.Ipv4Ranges.CidrIp
FromPort = $_.IpPermission.FromPort
ToPort = $_.IpPermission.ToPort
Protocol = $_.IpPermission.IpProtocol
}
$object = new-object psobject -Property $obj
$allSG += $object
}
Output:
Insights:
Some Descriptions might be missing.
The From/To Port represent the Type (e.g - 80 = HTTP, 443 = HTTPS)
More about Get-EC2SecurityGroup.

Related

powercli site recovery manager: determine which protection group an unprotected VM will be in. SRM uses array based replication

I use automation to deply VM's to various vcenter clusters.
I then confgiure SRM network mapping to create a network map between the cluster that the VM is in and the cluster which is used for DR purposes, in the protection group for those two clusters.
SRM is set up for array based replication, so as long as the VM is placed on replicated storage in the right cluster it will appear in SRM under the protection group, if a network mapping is in place then the VM will be auto protected by SRM or via my SRM config script.
I currently have the primary cluster, DR cluster and protection group hard coded, but would like to determine the protection group a VM is in and the name of the two clusters which the protection group is set up for, that way any changes to cluster configuration is automatically picked up and doesn't require manual changes to the SRM config script.
I've looked in the SRM API docs but it's not something I have worked out yet!
I have solved the issue:
$credential = Get-Credential
$server_name = "test-server"
Connect-VIServer -Server $primaryDC -Credential $credential
$srmConnection = Connect-SrmServer -Credential $credential -RemoteCredential $credential
Connect-VIServer -Server $secondaryDC -Credential $credential
$srmApi = $srmConnection.ExtensionData
$protectionGroups = $srmApi.Protection.ListProtectionGroups()
foreach ($protectionGroup in $protectionGroups){
$associatedVms = $protectionGroup.ListProtectedDatastores() | Get-VIObjectByVIView | Get-VM | Where-Object {($_.name -eq $server_name) -and($_.ExtensionData.Config.ManagedBy.ExtensionKey -ne 'com.vmware.vcDr' )}
foreach ($vm in $associatedVms) {
if ($vm.Name -eq $server_name) {
$protection_group_name = $protectionGroup.GetInfo().Name
$primary_cluster = get-vm -name $server_name | get-cluster
$primary_cluster_res_group = $primary_cluster.ExtensionData.ResourcePool
$srm_resource_groups = $srmApi.inventoryMapping.getResourcePoolMappings()
foreach ($resource_group in $srm_resource_groups){
if ($resource_group.PrimaryObject -eq $primary_cluster_res_group){
$secondary_res_group = $resource_group.SecondaryObject
}
}
}
}
}
$secondary_cluster = Get-Cluster | Where-Object {$_.ExtensionData.ResourcePool -eq $secondary_res_group}
Write-Host "VM: $vm - Protection Group: $protection_group_name - Primary cluster: $primary_cluster - Secondary cluster: $secondary_cluster - Primary ResGrp: $primary_cluster_res_group - Secondary ResGrp: $secondary_res_group"

Unable to register targets on AWS Target group

I am trying to setup an automated DNS deployment using powershell. I have written a powershell script that creates TargetGroup, registers instances to the TG, creates an ALB and adds a listener to it. Once, that is done, it creates R53 RecordSet and creates an A record to the ALB DNS.
I am having issues in having the instances registered to the TargetGroup.
This is my code snippet to that section:
$searchFor1 =#( #{name = 'tag:Name'; values = $target1})
$searchFor2 =#( #{name = 'tag:Name'; values = $target2})
$id1 = (Get-EC2Instance -Filter $searchFor1).Instances | select InstanceId
$id2 = (Get-EC2Instance -Filter $searchFor2).Instances | select InstanceId
# Create Target Group
$tg = New-ELB2TargetGroup -TargetType "instance" -HealthyThresholdCount 4 -Name $custname -Port $siteport -Protocol "HTTP" -UnhealthyThresholdCount 4 -VpcId $vpcid
Start-Sleep -s 120
$addid1 = New-Object Amazon.ElasticLoadBalancingV2.Model.TargetDescription
$addid2 = New-Object Amazon.ElasticLoadBalancingV2.Model.TargetDescription
$addid1.Id = $id1.InstanceId
$addid2.Id = $id2.InstanceId
$addid1.Port = $siteport
$addid2.Port = $siteport
$tgarn = (Get-ELB2TargetGroup -Name $custname).TargetGroupArn
Register-ELB2Target -TargetGroupArn $tgarn -Target #($addid1)
Register-ELB2Target -TargetGroupArn $tgarn -Target #($addid2)
It throws below error:
Register-ELB2Target : An instance ID must be specified
At C:\scripts\Distinct-DNS-Deployment.ps1:107 char:1
+ Register-ELB2Target -TargetGroupArn $tgarn -Target #($addid1)
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidOperation: (Amazon.PowerShe...LB2TargetCmdlet:RegisterELB2TargetCmdlet) [Register
-ELB2Target], InvalidOperationException
+ FullyQualifiedErrorId : Amazon.ElasticLoadBalancingV2.AmazonElasticLoadBalancingV2Exception,Amazon.PowerShell.Cm
dlets.ELB2.RegisterELB2TargetCmdlet
I have checked a similar post here. And the corresponding posts, so far nothing helped. I am wondering if anyone can guide me what am I doing wrong?
I have tried to run each line one by one and that happens to register the instance to the TargetGroup, just the script fails.
Instances are t2.micro and they are in running state.
As per the https://docs.aws.amazon.com/sdkfornet/v3/apidocs/items/ElasticLoadBalancingV2/TTargetDescription.html -
the Amazon.ElasticLoadBalancingV2.Model.TargetDescription is about 'Information about a target' -
which means, you should be assigning a single instance id Also, if you take a close look at the properties:
AvailabilityZone System.String
Id System.String
Port System.Int32
The result of your instance search may or may not be single output - you should keep them in loop to create each target via TargetDescription
$Instances = (Get-EC2Instance -Filter #{Name="tag:auto-delete";Value="no"}).instances |select instanceid
$theVpc = get-ec2vpc -VpcId vpc-4565e5c4
$name = "new-tg"
$port = "80"
$protocol = "HTTP"
$tg = New-ELB2TargetGroup -TargetType "instance" -HealthyThresholdCount 4 -Name $name -Port $port -Protocol "HTTP" -UnhealthyThresholdCount 4 -VpcId $theVpc.VpcId
$tgarn = (Get-ELB2TargetGroup -Name $name).TargetGroupArn
If($instances -ne $null){
foreach ($instance in $instances ){
$addid1 = New-Object Amazon.ElasticLoadBalancingV2.Model.TargetDescription
$addid1.Id = $Instance.instanceid
$addid1.Port = $port
Register-ELB2Target -TargetGroupArn $tgarn -Target #($addid1)
Remove-Variable addid1
}
}
else {
echo "There were no instances with the matching filter"
}

How to specify a port range using Grant-EC2SecurityGroupIngress in AWS PowerShell Tools

I'm using AWS Powershell Tools and AWS CLI.
I am able to add inbound rules to a security group using code like the following:
$IpRange = New-Object -TypeName Amazon.EC2.Model.IpRange
$IpRange.CidrIp = "102.196.30.33/32"
$IpRange.Description = "RDP"
$IpPermission = New-Object Amazon.EC2.Model.IpPermission
$IpPermission.IpProtocol = "tcp"
$IpPermission.FromPort = 2089
$IpPermission.ToPort = 2089
$IpPermission.Ipv4Ranges = $IpRange
Grant-EC2SecurityGroupIngress -GroupId sg-9773d0bb -IpPermission $IpPermission
# Verify inbound rule has been correctly added
aws ec2 describe-security-groups --group-ids sg-9773d0bb
However, if I try to add a second call to Grant-EC2SecurityGroupIngress and specify a port range, then the second call doesn't work:
$IpRange.Description = "FTP PASV"
$IpPermission.FromPort = 2025
$IpPermission.ToPort = 2030
Grant-EC2SecurityGroupIngress -GroupId sg-9773d0bb -IpPermission $IpPermission
# Verify inbound rule has been correctly added
aws ec2 describe-security-groups --group-ids sg-9773d0bb
No error is returned, however the rule is not added. Why not?
UPDATE
Some people have suggested using the AWS CLI authorize-security-group-ingress rule instead, however that returns an "Invalid JSON" error message if I add a description to the rule:
Usually something like this would indicate trouble with incorrect escaping of quotes, but I don't think it is. Here it is with double quotes around the string, and the internal double quotes escaped either using back-ticks, or double-double quotes:
Answer in two parts:
PART 1: Grant-EC2SecurityGroupIngress.
Internally this command is modifying $IpPermission. If you try to change the $IpRange.CidrIp to a different address then you will see the error:
Grant-EC2SecurityGroupIngress : Cannot set values for both Ipv4Ranges and IpRanges properties on
the IpPermission type which is part of the request. Consider using only Ipv4Ranges as IpRanges has
been marked obsolete.
The solution is to reallocate $IpPermission.
$IpRange.CidrIp = "102.196.30.33/32"
$IpRange.Description = "FTP PASV"
$IpPermission = New-Object Amazon.EC2.Model.IpPermission
$IpPermission.IpProtocol = "tcp"
$IpPermission.FromPort = 2025
$IpPermission.ToPort = 2030
$IpPermission.Ipv4Ranges = $IpRange
Grant-EC2SecurityGroupIngress -GroupId $sg -IpPermission $IpPermission
PART 2: For PowerShell (Windows) AWS CLI: you need to escape the double-quotes and surround the string with double-quotes.
Here is a working example:
aws ec2 authorize-security-group-ingress --group-id sg-12345678 --ip-permissions "[{\"IpProtocol\": \"tcp\", \"FromPort\": 2025, \"ToPort\": 2030, \"IpRanges\": [{\"CidrIp\": \"102.196.30.33/32\", \"Description\": \"FTP PASV\"}]}]"
If you're happy sticking with the AWS Command-Line Interface (CLI), this worked for me:
aws ec2 authorize-security-group-ingress --group-id sg-xxx --ip-permissions '[{"IpProtocol": "tcp", "FromPort": 2025, "ToPort": 2030, "IpRanges": [{"CidrIp": "102.196.30.33/32", "Description": "FTP PASV"}]}]'

Is that possible to create bulk aws ALBs using powershell script?

Is that possible to create bulk aws ALBs using powershell script?
If someone can provide Powershell script template, that would be great.
Absolutely, you can install AWS Tools for PowerShell. Check link below, there are examples there.
https://aws.amazon.com/powershell/
`# Create HTTP Listener
$HTTPListener = New-Object -TypeName ‘Amazon.ElasticLoadBalancing.Model.Listener’
$HTTPListener.Protocol = ‘http’
$HTTPListener.InstancePort = 80
$HTTPListener.LoadBalancerPort = 80
#Create HTTPS Listener
$HTTPSListener = New-Object -TypeName ‘Amazon.ElasticLoadBalancing.Model.Listener’
$HTTPSListener.Protocol = ‘http’
$HTTPSListener.InstancePort = 443
$HTTPSListener.LoadBalancerPort = 80
$HTTPSListener.SSLCertificateId = ‘YourSSL’
# Create Load Balancer
New-ELBLoadBalancer -LoadBalancerName ‘YourLoadBalancerName’ -Listeners
#($HTTPListener, $HTTPSListener) -SecurityGroups #($sgId) -Subnets #($sn1Id, $sn2Id)
-Scheme ‘internet-facing’
# Create Load Balancer
New-ELBLoadBalancer -LoadBalancerName ‘YourLoadBalancerName’ -Listeners
#($HTTPListener, $HTTPSListener) -SecurityGroups #(‘SecurityGroupId’) -Subnets
#(‘subnetId1’, ‘subnetId2’) -Scheme ‘internet-facing’
# Associate Instances with Load Balancer
Register-ELBInstanceWithLoadBalancer -LoadBalancerName ‘YourLoadBalancerName’ -
Instances #(‘instance1ID’, ‘instance2ID’)
# Create Application Cookie Stickiness Policy
New-ELBAppCookieStickinessPolicy -LoadBalancerName ‘YourLoadBalancerName’ -
PolicyName ‘SessionName’ -CookieName ‘CookieName’
# Set the Application Cookie Stickiness Policy to Load Balancer
Set-ELBLoadBalancerPolicyOfListener -LoadBalancerName ‘YourLoadBalancerName’ -
LoadBalancerPort 80 -PolicyNames ‘SessionName’`
This script is just for one elb...how to transform this scripts to create bulk elbs?
Also, where to mention AWS account credentials?

How to find Unused Amazon EC2 Security groups

I'm try to find a way to determine orphan security groups so I can clean up and get rid of them. Does anyone know of a way to discover unused security groups.
Either through the console or with the command line tools will work (Running command line tools on linux and OSX machines).
Note: this only considers security use in EC2, not other services like RDS. You'll need to do more work to include security groups used outside EC2. The good thing is you can't easily (might not even be possible) to delete active security groups if you miss one associated w/another service.
Using the newer AWS CLI tool, I found an easy way to get what I need:
First, get a list of all security groups
aws ec2 describe-security-groups --query 'SecurityGroups[*].GroupId' --output text | tr '\t' '\n'
Then get all security groups tied to an instance, then piped to sort then uniq:
aws ec2 describe-instances --query 'Reservations[*].Instances[*].SecurityGroups[*].GroupId' --output text | tr '\t' '\n' | sort | uniq
Then put it together and compare the 2 lists and see what's not being used from the master list:
comm -23 <(aws ec2 describe-security-groups --query 'SecurityGroups[*].GroupId' --output text | tr '\t' '\n'| sort) <(aws ec2 describe-instances --query 'Reservations[*].Instances[*].SecurityGroups[*].GroupId' --output text | tr '\t' '\n' | sort | uniq)
If you select all of your security groups in the EC2 console, then press actions -> Delete Security Groups, a popup will appear telling you that you cannot delete security groups that are attached to instances, other security groups, or network interfaces, and it will list the security groups that you can delete; ie the unused security groups.
NOTE: according to #andrewlorien’s comment this does not work for all types of AWS services.
This is the sample code written in boto (Python SDK for AWS) to list the Security Group against number of instances it is associated with.
You may use this logic to obtain the same in command line as well
Boto Code
import boto
ec2 = boto.connect_ec2()
sgs = ec2.get_all_security_groups()
for sg in sgs:
print sg.name, len(sg.instances())
Output
Security-Group-1 0
Security-Group-2 1
Security-Group-3 0
Security-Group-4 3
After about a year of unaudited use, I found it necessary to audit my AWS EC2 security groups and clean up legacy, unused groups.
This was a daunting task to perform via the web GUI, so I looked to the AWS CLI to make the task easier. I found a start on how to do this at StackOverflow, but it was far from complete. So I decided to write my own script. I used the AWS CLI, MySQL and some “Bash-foo” to perform the following:
Get a list of all EC2 security groups.
I store the group-id, group-name and description in a tabled called “groups” in a MySQL database called aws_security_groups on the localhost. The total number of groups found is reported to the user.
Get a list of all security groups associated with each of the following services and exclude them from the table:
EC2 Istances
EC2 Elastic Load Balancers
AWS RDS Instances
AWS OpsWorks (shouldn’t be removed per Amazon)
Default security groups (Can’t be deleted)
ElastiCache
For each service I report a count of the number of groups left in the table after the exclusion is complete.
Finally I display the group-id, group-name and description for the groups that are left. These are the “unused” groups that need to be audited and/or deleted. I’ve found that SG’s between instances and Elastic Load Balancers (ELBs) often refer to each other. It’s best practice to do some manual investigation to ensure they are truly not in use prior to removing the cross references and deleting the security groups. But my script at least pares this down to something mor manageable.
NOTES:
1. You will want to create a file to store your MySQL host, username and password and point the $DBCONFIG variable to it. It should be structured like this:
[mysql]
host=your-mysql-server-host.com
user=your-mysql-user
password=your-mysql-user-password
You can change the name of the database if you wish – make sure to change the $DB variable in the script
Let me know if you find this useful or have any comments,fixes or enhancements.
Here is the script.
#!/bin/bash
# Initialize Variables
DBCONFIG="--defaults-file=mysql-defaults.cnf"
DB="aws_security_groups"
SGLOOP=0
EC2LOOP=0
ELBLOOP=0
RDSLOOP=0
DEFAULTLOOP=0
OPSLOOP=0
CACHELOOP=0
DEL_GROUP=""
# Function to report back # of rows
function Rows {
ROWS=`echo "select count(*) from groups" | mysql $DBCONFIG --skip-column-names $DB`
# echo -e "Excluding $1 Security Groups.\nGroups Left to audit: "$ROWS
echo -e $ROWS" groups left after Excluding $1 Security Groups."
}
# Empty the table
echo -e "delete from groups where groupid is not null" | mysql $DBCONFIG $DB
# Get all Security Groups
aws ec2 describe-security-groups --query "SecurityGroups[*].[GroupId,GroupName,Description]" --output text > /tmp/security_group_audit.txt
while IFS=$'\t' read -r -a myArray
do
if [ $SGLOOP -eq 0 ];
then
VALUES="(\""${myArray[0]}"\",\""${myArray[1]}"\",\""${myArray[2]}"\")"
else
VALUES=$VALUES",(\""${myArray[0]}"\",\""${myArray[1]}"\",\""${myArray[2]}"\")"
fi
let SGLOOP="$SGLOOP + 1"
done < /tmp/security_group_audit.txt
echo -e "insert into groups (groupid, groupname, description) values $VALUES" | mysql $DBCONFIG $DB
echo -e $SGLOOP" security groups total."
# Exclude Security Groups assigned to Instances
for groupId in `aws ec2 describe-instances --output json | jq -r ".Reservations[].Instances[].SecurityGroups[].GroupId" | sort | uniq`
do
if [ $EC2LOOP -eq 0 ];
then
DEL_GROUP="'$groupId'"
else
DEL_GROUP=$DEL_GROUP",'$groupId'"
fi
let EC2LOOP="$EC2LOOP + 1"
done
echo -e "delete from groups where groupid in ($DEL_GROUP)" | mysql $DBCONFIG $DB
Rows "EC2 Instance"
DEL_GROUP=""
# Exclude groups assigned to Elastic Load Balancers
for elbGroupId in `aws elb describe-load-balancers --output json | jq -c -r ".LoadBalancerDescriptions[].SecurityGroups" | tr -d "\"[]\"" | sort | uniq`
do
if [ $ELBLOOP -eq 0 ];
then
DEL_GROUP="'$elbGroupId'"
else
DEL_GROUP=$DEL_GROUP",'$elbGroupId'"
fi
let ELBLOOP="$ELBLOOP + 1"
done
echo -e "delete from groups where groupid in ($DEL_GROUP)" | mysql $DBCONFIG $DB
Rows "Elastic Load Balancer"
DEL_GROUP=""
# Exclude groups assigned to RDS
for RdsGroupId in `aws rds describe-db-instances --output json | jq -c -r ".DBInstances[].VpcSecurityGroups[].VpcSecurityGroupId" | sort | uniq`
do
if [ $RDSLOOP -eq 0 ];
then
DEL_GROUP="'$RdsGroupId'"
else
DEL_GROUP=$DEL_GROUP",'$RdsGroupId'"
fi
let RDSLOOP="$RDSLOOP + 1"
done
echo -e "delete from groups where groupid in ($DEL_GROUP)" | mysql $DBCONFIG $DB
Rows "RDS Instances"
DEL_GROUP=""
# Exclude groups assigned to OpsWorks
for OpsGroupId in `echo -e "select groupid from groups where groupname like \"AWS-OpsWorks%\"" | mysql $DBCONFIG $DB`
do
if [ $OPSLOOP -eq 0 ];
then
DEL_GROUP="'$OpsGroupId'"
else
DEL_GROUP=$DEL_GROUP",'$OpsGroupId'"
fi
let OPSLOOP="$OPSLOOP + 1"
done
echo -e "delete from groups where groupid in ($DEL_GROUP)" | mysql $DBCONFIG $DB
Rows "OpsWorks"
DEL_GROUP=""
# Exclude default groups (can't be deleted)
for DefaultGroupId in `echo -e "select groupid from groups where groupname like \"default%\"" | mysql $DBCONFIG $DB`
do
if [ $DEFAULTLOOP -eq 0 ];
then
DEL_GROUP="'$DefaultGroupId'"
else
DEL_GROUP=$DEL_GROUP",'$DefaultGroupId'"
fi
let DEFAULTLOOP="$DEFAULTLOOP + 1"
done
echo -e "delete from groups where groupid in ($DEL_GROUP)" | mysql $DBCONFIG $DB
Rows "Default"
DEL_GROUP=""
# Exclude Elasticache groups
for CacheGroupId in `aws elasticache describe-cache-clusters --output json | jq -r ".CacheClusters[].SecurityGroups[].SecurityGroupId" | sort | uniq`
do
if [ $CACHELOOP -eq 0 ];
then
DEL_GROUP="'$CacheGroupId'"
else
DEL_GROUP=$DEL_GROUP",'$CacheGroupId'"
fi
let CACHELOOP="$CACHELOOP + 1"
done
echo -e "delete from groups where groupid in ($DEL_GROUP)" | mysql $DBCONFIG $DB
Rows "ElastiCache"
# Display Security Groups left to audit / delete
echo "select * from groups order by groupid" | mysql $DBCONFIG $DB | sed 's/groupid\t/groupid\t\t/'
And here is the sql to create the database.
-- MySQL dump 10.13 Distrib 5.5.41, for debian-linux-gnu (x86_64)
--
-- Host: localhost Database: aws_security_groups
-- ------------------------------------------------------
-- Server version 5.5.40-log
/*!40101 SET #OLD_CHARACTER_SET_CLIENT=##CHARACTER_SET_CLIENT */;
/*!40101 SET #OLD_CHARACTER_SET_RESULTS=##CHARACTER_SET_RESULTS */;
/*!40101 SET #OLD_COLLATION_CONNECTION=##COLLATION_CONNECTION */;
/*!40101 SET NAMES utf8 */;
/*!40103 SET #OLD_TIME_ZONE=##TIME_ZONE */;
/*!40103 SET TIME_ZONE='+00:00' */;
/*!40014 SET #OLD_UNIQUE_CHECKS=##UNIQUE_CHECKS, UNIQUE_CHECKS=0 */;
/*!40014 SET #OLD_FOREIGN_KEY_CHECKS=##FOREIGN_KEY_CHECKS, FOREIGN_KEY_CHECKS=0 */;
/*!40101 SET #OLD_SQL_MODE=##SQL_MODE, SQL_MODE='NO_AUTO_VALUE_ON_ZERO' */;
/*!40111 SET #OLD_SQL_NOTES=##SQL_NOTES, SQL_NOTES=0 */;
--
-- Table structure for table `groups`
--
DROP TABLE IF EXISTS `groups`;
/*!40101 SET #saved_cs_client = ##character_set_client */;
/*!40101 SET character_set_client = utf8 */;
CREATE TABLE `groups` (
`groupid` varchar(12) DEFAULT NULL,
`groupname` varchar(200) DEFAULT NULL,
`description` varchar(200) DEFAULT NULL
) ENGINE=InnoDB DEFAULT CHARSET=latin1;
/*!40101 SET character_set_client = #saved_cs_client */;
--
-- Dumping data for table `groups`
--
LOCK TABLES `groups` WRITE;
/*!40000 ALTER TABLE `groups` DISABLE KEYS */;
/*!40000 ALTER TABLE `groups` ENABLE KEYS */;
UNLOCK TABLES;
/*!40103 SET TIME_ZONE=#OLD_TIME_ZONE */;
/*!40101 SET SQL_MODE=#OLD_SQL_MODE */;
/*!40014 SET FOREIGN_KEY_CHECKS=#OLD_FOREIGN_KEY_CHECKS */;
/*!40014 SET UNIQUE_CHECKS=#OLD_UNIQUE_CHECKS */;
/*!40101 SET CHARACTER_SET_CLIENT=#OLD_CHARACTER_SET_CLIENT */;
/*!40101 SET CHARACTER_SET_RESULTS=#OLD_CHARACTER_SET_RESULTS */;
/*!40101 SET COLLATION_CONNECTION=#OLD_COLLATION_CONNECTION */;
/*!40111 SET SQL_NOTES=#OLD_SQL_NOTES */;
-- Dump completed on 2015-01-27 16:07:44
Among other functions, both ScoutSuite and Prowler report unused EC2 Security Groups. Both are open source.
A boto example printing the Group IDs and Names only of the security groups that have no current instances.
It also shows how to specify which region you are concerned with.
import boto
import boto.ec2
EC2_REGION='ap-southeast-2'
ec2region = boto.ec2.get_region(EC2_REGION)
ec2 = boto.connect_ec2(region=ec2region)
sgs = ec2.get_all_security_groups()
for sg in sgs:
if len(sg.instances()) == 0:
print ("{0}\t{1}".format(sg.id, sg.name))
To confirm which security groups are still being used you should reverse or remove the if len(sg.instances()) == 0 test and print the len(sg.instances()) value out.
E.g.
print ("{0}\t{1}\t{2} instances".format(sg.id, sg.name, len(sg.instances())))
Using the node.js AWS SDK I can confirm that AWS doesn't allow you to delete security groups that are in use. I wrote a script that simply tries to delete all groups and gracefully handles the errors. This works for classic and the modern VPC. The error message can be seen below.
Err { [DependencyViolation: resource sg-12345678 has a dependent object]
message: 'resource sg-12345678 has a dependent object',
code: 'DependencyViolation',
time: Mon Dec 07 2015 12:12:43 GMT-0500 (EST),
statusCode: 400,
retryable: false,
retryDelay: 30 }
To the SGs attached to the network interfaces:
By name:
aws ec2 describe-network-interfaces --output text --query NetworkInterfaces[*].Groups[*].GroupName | tr -d '\r' | tr "\t" "\n" | sort | uniq
By id:
aws ec2 describe-network-interfaces --output text --query NetworkInterfaces[*].Groups[*].GroupId | tr -d '\r' | tr "\t" "\n" | sort | uniq
I was searching for the same info.
How to find all security groups that are not attached to any resource? And I found this:
Using AWS config rule "EC2_SECURITY_GROUP_ATTACHED_TO_ENI," I got a list of checks that non-default security groups are attached to Amazon Elastic Compute Cloud (EC2) instances or elastic network interfaces (ENIs). The rule returns NON_COMPLIANT if the security group is not associated with an EC2 instance or an ENI.
This is a very old question and I'm sure there are more ways to skin this AWS cat, but here's my solution in bash (you'll need jq for this to work):
REGION="eu-west-1"
SGLIST=$(aws ec2 describe-security-groups --query 'SecurityGroups[*].GroupId' | jq -r .[])
echo $SGLIST | xargs -n1 | while read SG; do [ "$(aws ec2 describe-network-interfaces --filters Name=group-id,Values=$SG --region $REGION | jq .NetworkInterfaces)" != '[]' ] || echo $SG; done
Remember to replace REGION with whatever region you're using.
The 1st step is to get a list of security groups.
Then we're checking for each security group if there's a network interface associated with it - this is not limited to EC2 instances, it checks anything that has a network interface (LBs, RDS, etc).
For reference see here.
Unfortunately the chosen answer is not as accurate as I need (I've tried to investigate the why, but I've preferred to implement it).
If I check ALL NetworkInterfaces, looking for attachments to any SecurityGroup, It gets me partial results. If I check only on EC2Instances, it gets me back partial results as well.
So that's my approach to the problem:
I get ALL EC2 SecurityGroups -> all_secgrp
I get ALL EC2 Instances -> all_instances
For each Instance, I get all SecurityGroups attached to it
I remove from all_secgrp each of these SecurityGroup (because attached)
For each SecurityGroup, I check an association with any NetworkInterfaces (using the filter function and filtering using that security-group-id)
IF no association is found, I remove the security-group from all_secgrp
Attached you can see a snippet of code. Don't complain for efficiency, but try to optimize it if you want.
all_secgrp = list(ec2_connector.security_groups.all())
all_instances = ec2_connector.instances.all()
for single_instance in all_instances:
instance_secgrp = ec2_connector.Instance(single_instance.id).security_groups
for single_sec_grp in instance_secgrp:
if ec2.SecurityGroup(id=single_sec_grp['GroupId']) in all_secgrp:
all_secgrp.remove(ec2.SecurityGroup(id=single_sec_grp['GroupId']))
all_secgrp_detached_tmp = all_secgrp[:]
for single_secgrp in all_secgrp_detached_tmp:
try:
print(single_secgrp.id)
if len(list(ec2_connector.network_interfaces.filter(Filters=[{'Name': 'group-id', 'Values': [single_secgrp.id]}]))) > 0:
all_secgrp.remove(single_secgrp)
except Exception:
all_secgrp.remove(single_secgrp)
return all_secgrp_detached
There's a tool in the AWS marketplace that makes this a lot easier. It shows you which groups are attached/detached for easy deletion, but it also compares your VPC Flow Logs against the security group rules and shows you which SG rules are in use or unused. AWS posted an ELK-stack solution to do this, but it was ridiculously complex.
Here's the tool, and a disclaimer that I worked on it. But I hope you all find it pertinent:
https://www.piasoftware.net/single-post/2018/04/24/VIDEO-Watch-as-we-clean-up-EC2-security-groups-in-just-a-few-minutes
This is a difficult problem, if you have security groups that reference other security groups in the rules. If so, you'll have to resolve DependencyErrors, which is not trivial.
If you are only using IP addresses, then this solution will work, after you create a boto3 client:
# pull all security groups from all vpcs in the given profile and region and save as a set
all_sgs = {sg['GroupId'] for sg in client.describe_security_groups()['SecurityGroups']}
# create a new set for all of the security groups that are currently in use
in_use = set()
# cycle through the ENIs and add all found security groups to the in_use set
for eni in client.describe_network_interfaces()['NetworkInterfaces']:
for group in eni['Groups']:
in_use.add(group['GroupId'])
unused_security_groups = all_sgs - in_use
for security_group in unused_security_groups:
try:
response = client.delete_security_group(GroupId=security_group)
except ClientError as e:
if e.response['Error']['Code'] == 'DependencyViolation':
print('EC2/Security Group Dependencies Exist')
else:
print('Unexpected error: {}'.format(e))