Revoke all AWS security group ingress rules - amazon-web-services

Is it possible to revoke all the ingress rules in an AWS security group? Is it possible to revoke all the SSH ingress rules? I'm trying with the cli command below, but it's not working:
aws ec2 revoke-security-group-ingress --group-id GroupID --protocol tcp --port 22

Based on the suggestion of #kuboon, here is a simpler, working version of that script, tested in zsh. The key differences are:
Forcing the first command to return json (which is not always the default) by explicitly using --output json option
Passing that result to parameter --ip-permissions of revoke-security-group-ingress rather than as a fully-formatted command in json (which it is not) that --cli-input-json would require.
groupId="your group-id"
aws ec2 revoke-security-group-ingress --group-id $groupId \
--ip-permissions \
"`aws ec2 describe-security-groups --output json --group-ids $groupId --query "SecurityGroups[0].IpPermissions"`"

I ended up writing a Powershell script that uses the AWS CLI to do that.
The script goes through all the regions, looks for the default security group, and deletes all rules in it.
Here it is:
# get all regions
write-host "Getting all regions.."
$regions = ([string](aws ec2 describe-regions --region eu-west-2) | ConvertFrom-Json).Regions.RegionName
write-host "Got them:"
$regions
write-host "-------------------"
write-host ""
# for all regions
foreach ($region in $regions)
{
write-host "Getting default security groups for $region.."
# get all subnets
$groupIds = ([string](aws ec2 describe-security-groups --filters Name=group-name,Values=default --region $region) | ConvertFrom-Json).SecurityGroups.GroupId
foreach ($groupId in $groupIds)
{
write-host "Got it: $groupId"
write-host "Getting all rules.."
$rules = [string](aws ec2 describe-security-groups --group-id $groupId --query "SecurityGroups[0].IpPermissions" --region $region) | ConvertFrom-Json
foreach ($rule in $rules)
{
$protocol = $rule.IpProtocol
$cidr = $rule.IpRanges.CidrIp
$fromPort = $rule.FromPort
$toPort = $rule.ToPort
$cidrIpv6 = $rule.Ipv6Ranges.CidrIpv6
$sourceGroup = $rule.UserIdGroupPairs.GroupId
$sourceGroupUserId = $rule.UserIdGroupPairs.UserId
if ($protocol -eq "icmpv6") {
$protocol = "icmp"
}
if (($protocol -eq "tcp") -Or ($protocol -eq "udp") -Or ($protocol -eq "icmp"))
{
if ($cidr){
if ($fromPort -eq -1){
write-host "Removing rule from security group using this command:"
write-host "aws ec2 revoke-security-group-ingress --group-id $groupId --protocol $protocol --port "$fromPort" --cidr $cidr --region $region"
aws ec2 revoke-security-group-ingress --group-id $groupId --protocol $protocol --port "$fromPort" --cidr $cidr --region $region
write-host "Done!"
query
}
else {
write-host "Removing rule from security group using this command:"
write-host "aws ec2 revoke-security-group-ingress --group-id $groupId --protocol $protocol --port "$fromPort-$toPort" --cidr $cidr --region $region"
aws ec2 revoke-security-group-ingress --group-id $groupId --protocol $protocol --port "$fromPort-$toPort" --cidr $cidr --region $region
write-host "Done!"
}
}
if ($cidrIpv6){
$json = ('{"IpProtocol": "'+$protocol+'", "FromPort": '+$fromPort+', "ToPort": '+$toPort+', "Ipv6Ranges": [{"CidrIpv6": "'+$cidrIpv6+'"}]}') | ConvertTo-Json
write-host "Removing Ipv6 version of rule from security group using this command:"
write-host "aws ec2 revoke-security-group-ingress --group-id $groupId --ip-permissions $json --region $region"
aws ec2 revoke-security-group-ingress --group-id $groupId --ip-permissions $json --region $region
write-host "Done!"
}
if ($sourceGroup -and $sourceGroupUserId)
{
write-host "Removing SourceGroup rule from security group using this command:"
write-host "aws ec2 revoke-security-group-ingress --group-id $groupId --protocol $protocol --port "$fromPort-$toPort" --cidr $cidr --region $region"
aws ec2 revoke-security-group-ingress --group-id $groupId --protocol $protocol --port "$fromPort-$toPort" --source-group $sourceGroup --group-owner $sourceGroupUserId --region $region
write-host "Done!"
}
}
else
{
if ($cidr){
write-host "Removing rule from security group using this command:"
write-host "aws ec2 revoke-security-group-ingress --group-id $groupId --protocol $protocol --cidr $cidr --region $region"
aws ec2 revoke-security-group-ingress --group-id $groupId --protocol $protocol --cidr $cidr --region $region
write-host "Done!"
}
if ($cidrIpv6){
$json = '{"IpProtocol": "-1", "Ipv6Ranges": [{"CidrIpv6": "'+$cidrIpv6+'"}]}' | ConvertTo-Json
write-host "Removing Ipv6 version of rule from security group using this command:"
write-host "aws ec2 revoke-security-group-ingress --group-id $groupId --ip-permissions $json --region $region"
aws ec2 revoke-security-group-ingress --group-id $groupId --ip-permissions $json --region $region
write-host "Done!"
}
if ($sourceGroup)
{
$json = '{ "IpProtocol": "-1", "UserIdGroupPairs":[{"GroupId":"'+$sourceGroup+'","UserId":"'+$sourceGroupUserId+'"}] }' | ConvertTo-Json
write-host "Removing SourceGroup rule from security group using this command:"
write-host "aws ec2 revoke-security-group-ingress --group-id $groupId --ip-permissions $json --region $region"
aws ec2 revoke-security-group-ingress --group-id $groupId --ip-permissions $json --region $region
write-host "Done!"
}
}
}
}
write-host "-------------------"
write-host ""
}
N.B.
This script cannot delete some rules (custom protocols, Custom ICMP Rule - IPv6), but it works for most rules.
I tested the script on the following set of rules:
This is what is left after running the script:
Hope this helps somebody out there!

It looks like you have to specify each source individually, for example --cidr 0.0.0.0/0 or --source-group sg-12345678.

Get list of rules and revoke all.
groupId="your group-id"
json=`aws ec2 describe-security-groups --group-id $groupId --query "SecurityGroups[0].IpPermissions"`
aws ec2 revoke-security-group-ingress --cli-input-json "{\"GroupId\": \"$groupId\", \"IpPermissions\": $json}"

Just a minor improvement to demonicdaron's script, more than 1 cidr can be returned to the same rule, in this case you just do a foreach() on the $cidr to loop over everyone.
But it works like a charm

Related

Terraform throwing this error Getting Extra characters after interpolation expression;

I'm trying to associate the Elastic IP address with Auto scaling group, so whenever the autoscaling triggers it will automatically associate with the EIP.
For this I'm trying to add the script in user data.
My intention is to we have 2 servers so its associated with 2 EIP's, whenever the autoscaling triggers it has to check whether the EIP is free or not if its free it has to associate with that instance using the instance id.
Below is my script where I'm getting the error
In EIP_LIST im getting this error Extra characters after interpolation expression; Expected a closing brace to end the interpolation expression, but found extra characters.
INSTANCE_ID=$(ec2-metadata --instance-id | cut -d " " -f 2);
MAXWAIT=10
# Get list of EIPs
EIP_LIST=${"eipalloc-09e7274dd3c641ae6" "eipalloc-05e8e926926f9de55"}
# Iterate over EIP list
for EIP in ${EIP_LIST}; do
echo "Checking if EIP with ALLOC_ID[$EIP] is free...."
ISFREE=$(aws ec2 describe-addresses --allocation-ids $EIP --query Addresses[].InstanceId --output text --region ap-south-1)
STARTWAIT=$(date +%s)
while [ ! -z "$ISFREE" ]; do
if [ "$(($(date +%s) - $STARTWAIT))" -gt $MAXWAIT ]; then
echo "WARNING: We waited 30 seconds, we're forcing it now."
ISFREE=""
else
echo "Waiting for EIP with ALLOC_ID[$EIP] to become free...."
sleep 3
ISFREE=$(aws ec2 describe-addresses --allocation-ids $EIP --query Addresses[].InstanceId --output text --region ap-south-1)
fi
done
echo Running: aws ec2 associate-address --instance-id $INSTANCE_ID --allocation-id $EIP --allow-reassociation --region ap-south-1
aws ec2 associate-address --instance-id $INSTANCE_ID --allocation-id $EIP --allow-reassociation --region ap-south-1
$ is TF symbol used for interpolation of variables. It clashes with the same symbol used in bash. You have to escape it using $$ if you want to us $ in bash, not in TF, e.g.
$${EIP_LIST}
will result in ${EIP_LIST} in your script.

deploy ecs-cli fargate cluster with load balancer

I am trying to create a Fargate cluster with ecs-cli using a load balancer
I came up so far with a script to deploy it without, so far my script is
building image
pushing it to ECR
echo ""
echo "creating task execution role"
aws iam wait role-exists --role-name $task_execution_role 2>/dev/null || \ aws iam --region $REGION create-role --role-name $task_execution_role \
--assume-role-policy-document file://task-execution-assume-role.json || return 1
echo ""
echo "adding AmazonECSTaskExecutionRole Policy"
aws iam --region $REGION attach-role-policy --role-name $task_execution_role \
--policy-arn arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy || return 1
echo ""
echo "creating task role"
aws iam wait role-exists --role-name $task_role 2>/dev/null || \
aws iam --region $REGION create-role --role-name $task_role \
--assume-role-policy-document file://task-role.json
echo ""
echo "adding AmazonS3ReadOnlyAccess Policy"
aws iam --region $REGION attach-role-policy --role-name $task_role \
--policy-arn arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess || return 1
echo ""
echo "configuring cluster"
ecs-cli configure --cluster $CLUSTER --default-launch-type FARGATE --config-name $CLUSTER --region $REGION || return 1
ecs-cli down --force --cluster-config $CLUSTER --ecs-profile $profile_name || return 1
ecs-cli up --force --cluster-config $CLUSTER --ecs-profile $profile_name || return 1
echo ""
echo "adding ingress rules to security groups"
aws ec2 authorize-security-group-ingress --group-id $SGid --protocol tcp \
--port 80 --cidr 0.0.0.0/0 --region $REGION || return
ecs-cli compose --project-name $SERVICE_NAME service up --create-log-groups \
--cluster-config $CLUSTER --ecs-profile $profile_name
ecs-cli compose --project-name $SERVICE_NAME service ps \
--cluster-config $CLUSTER --ecs-profile $profile_name
aws ec2 describe-instances --query 'Reservations[].Instances[].[InstanceId,InstanceType,PublicIpAddress,Tags[?Key==`Name`]| [0].Value]' --output table
this works. service is up and I can access it from the public ip.
I now would like to add a load balancer
so I can expose a DNS with route53
Following a few other questions’ advice (this one in particular)
I came up with this
echo ""
echo "configuring cluster"
ecs-cli compose --project-name $CLUSTER create
ecs-cli configure --cluster $CLUSTER --default-launch-type FARGATE --config-name $CLUSTER --region $REGION
echo ""
echo "creating a new AWS CloudFormation stack called amazon-ecs-cli-setup-"$CLUSTER
ecs-cli up --force --cluster-config $CLUSTER --ecs-profile $profile_name
echo "create elb & add a dns CNAME for the elb dns"
aws elb create-load-balancer --load-balancer-name $SERVICE_NAME --listeners Protocol="TCP,LoadBalancerPort=8080,InstanceProtocol=TCP,InstancePort=80" --subnets $subnet1 $subnet2 --security-groups $SGid --scheme internal
echo "create service with above created task definition & elb"
aws ecs create-service \
--cluster $CLUSTER \
--service-name ecs-simple-service-elb \
--cli-input-json file://ecs-simple-service-elb.json
ecs-cli compose --project-name $SERVICE_NAME service up --create-log-groups \
--cluster-config $CLUSTER --ecs-profile $profile_name
echo ""
echo "here are the containers that are running in the service"
ecs-cli compose --project-name $SERVICE_NAME service ps --cluster-config $CLUSTER --ecs-profile $profile_name
and I get the following error messages:
create elb & add a dns CNAME for the elb dns
An error occurred (InvalidParameterException) when calling the CreateService operation: Unable to assume role and validate the listeners configured on your load balancer. Please verify that the ECS service role being passed has the proper permissions.
INFO[0002] Using ECS task definition TaskDefinition="dashboard:4"
WARN[0003] Failed to create log group dashboard-ecs in us-east-1: The specified log group already exists
INFO[0003] Auto-enabling ECS Managed Tags
ERRO[0003] Error creating service error="InvalidParameterException: subnet cannot be blank." service=dashboard
INFO[0003] Created an ECS service service=dashboard taskDefinition="dashboard:4"
FATA[0003] InvalidParameterException: subnet cannot be blank.
here are the containers that are running in the service
Name State Ports TaskDefinition Health
dashboard/4d0ebb65b20e4010b93cb99fb5b9e21d/web STOPPED ExitCode: 137 80->80/tcp dashboard:4 UNKNOWN
My task execution role and task role have this policy attached
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "",
"Effect": "Allow",
"Principal": {
"Service": "ecs-tasks.amazonaws.com"
},
"Action": "sts:AssumeRole"
}
]
}
and the JSON I pass to create service is (copied from the documentation):
{
"serviceName": "dashboard",
"taskDefinition": "dashboard",
"loadBalancers": [
{
"loadBalancerName": "dashboard",
"containerName": "dashboard",
"containerPort": 80
}
],
"desiredCount": 10,
"role": "ecsTaskExecutionRole"
}
what permissions am I missing and what should I change?
IIRC, your ECS service role should have AmazonEC2ContainerServiceRole role permissions to access your ELB and validate the listeners.
See here - https://aws.amazon.com/premiumsupport/knowledge-center/assume-role-validate-listeners/
and here - https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs_managed_policies.html#AmazonEC2ContainerServiceRole

AWS SSM Agent - Using the aws cli, is there a way to list all the AWS instances that are missing the SSM agent?

I need to audit a large number of AWS accounts to determine which EC2 instances are missing the SSM agent. Then I need have all those instances and their tags outputted.
Running aws ssm describe-instance-information lists all the instances that have the agent installed and are running, but it doesn't list instances that are missing the agent or systems that might be turned off.
#!/bin/bash
for instance in $(aws ec2 describe-instances --query 'Reservations[*].Instances[*].[InstanceId]' --output text )
do
managed=$(aws ssm describe-instance-information --filters "Key=InstanceIds,Values=$instance" --query 'InstanceInformationList[*].[AssociationStatus]' --output text)
if [[ "$managed" != "Success" ]]; then
managed="Not Managed";
fi
aws ec2 describe-instances --instance-id $instance --output text --query 'Reservations[*].Instances[*].[InstanceId, Placement.AvailabilityZone, [Tags[?Key==`Name`].Value] [0][0], [Tags[?Key==`App`].Value] [0][0], [Tags[?Key==`Product`].Value] [0][0], [Tags[?Key==`Team`].Value] [0][0] ]'
echo "$managed"
done
Save and make the script executable, then run
script.sh > file.tsv
And finally import it into excel
This will print a list of all your instances with "success" printed beneath the ones which are managed.
for instance in $(aws ec2 describe-instances --query 'Reservations[*].Instances[*].[InstanceId]' --output text )
do;
managed=$(aws ssm describe-instance-information --filters "Key=InstanceIds,Values=$instance" --query 'InstanceInformationList[*].[AssociationStatus]' --output text)
echo "$instance $managed";
done
To add a simple but not well-formatted set of tags, replace the echo line with
if [[ "$managed" != "Success" ]]; then
managed="Fail";
fi
echo "$instance $managed"
aws --profile GC-Staging ec2 describe-instances --instance-id $instance --query 'Reservations[*].Instances[*].[Tags[*].Value]' --output text

How to list only the instance id and related tags

I using the CLI tools and I want to list only the instance ID and related tags of an instance. The command that I am using is ec2-describe instances. I have tried certain filters but nothing is working out.
Can anybody help ??
error :
./Instance_Audit.sh: line 24: $: command not found
./Instance_Audit.sh: line 25: syntax error near unexpected token `do'
./Instance_Audit.sh: line 25: ` do echo $ID ; echo $(aws ec2 describe-instances --query "Reservations[].Instances[?InstanceId==\`$ID\`].Tags[]") done'
Using the aws cli, I can produce a few different alternatives of the tag output:
aws ec2 describe-instances --output text --query 'Reservations[*].Instances[*].[InstanceId,Tags[]]'
Result: i-xxxxx [{u'Value': 'key1value', u'Key': 'key1name'}, {u'Value': 'key2value', u'Key': 'key2name'}]
aws ec2 describe-instances --output text --query 'Reservations[*].Instances[*].[InstanceId,Tags[].*]'
Result: i-xxxxx [['key1value', 'key1name'], ['key2value', 'key2name']]
aws ec2 describe-instances --output text --query 'Reservations[*].Instances[*].[InstanceId,Tags[].Key,Tags[].Value]'
Result: i-xxxx ['key1name', 'key2name'] ['key1value' 'key2value']
Is any of those formats what you're looking for?
The AWS CLI command
aws ec2 describe-instances --region us-west-2 --query 'Reservations[].Instances[].[InstanceId,Tags[]]' --output text
produces these results:
i-789c55c3
Name Instance1
aws:cloudformation:logical-id Instance1
aws:cloudformation:stack-id arn:aws:cloudformation:us-west-2:012345678901:stack/test10-test10-foobarfoobar/a6e545a0-af52-11e4-a0be-50d5020578e0
aws:cloudformation:stack-name test10-test10-foobarfoobar
i-77c108cc
Name Instance2
aws:cloudformation:logical-id Instance2
aws:cloudformation:stack-id arn:aws:cloudformation:us-west-2:012345678901:stack/test10-test10-foobarfoobar/a6e545a0-af52-11e4-a0be-50d5020578e0
aws:cloudformation:stack-name test10-test10-foobarfoobar
If that is not in the format you are looking for, can you provide an example of they format you are expecting?
If you want to only display the instance id and the tags associated to that instance, you can use something like :
$ for ID in $(aws ec2 describe-instances --region eu-west-1 --query "Reservations[].Instances[].InstanceId" --output text); do echo $ID ; echo $(aws ec2 describe-instances --region eu-west-1 --query "Reservations[].Instances[?InstanceId==\`$ID\`].Tags[]") done
i-60****2a
[ [ { "Value": "SB", "Key": "Name" } ] ]
i-1a****56
[ [ { "Value": "Course and Schedule on Apache 2.4", "Key": "Name" } ] ]
i-88eff3cb
[ [ { "Value": "secret.com", "Key": "Name" } ] ]
i-e7****a5
[ [ { "Value": "2014.03 + Docker", "Key": "Name" } ] ]
I could not find a way to do that with only one AWS CLI call. Should someone come up with a shorter solution, I would love to hear from you.
If you want to filter to certain tag key/value only, you can edit the aws ec2 describe-instances to add a --filter option.
For example :
aws ec2 describe-instances --region eu-west-1 --filter "Name=tag:Name,Values=SB"
--Seb
simple answer:
aws ec2 describe-instances \
--query 'Reservations[*].Instances[*].[InstanceId,Tags[*]]'
not so simple answer .. the same principle could be applied to both instances and vpc's
# how-to get all the tags per instance-id with aws ec2 for prd env
while read -r i ; do \
aws ec2 describe-tags --filters "Name=resource-id,Values=$i" \
--profile prd; done < <(aws ec2 describe-instances \
--query 'Reservations[*].Instances[*].InstanceId' --profile prd)
# how-to get all the tags per virtual-private-cloud-id
while read -r v; do aws ec2 describe-tags \
--filters "Name=resource-id,Values=$v" ; \
done < <(aws ec2 describe-vpcs \
--query 'Vpcs[*].VpcId'|perl -ne 's/\t/\n/g;print')

Clear rules of AWS security group for a particular port

How to remove all rules for a given port using the "aws ec2"?
aws ec2 revoke-security-group-ingress --group-name MySecurityGroup --protocol tcp --port 22 **--ALL-IP**
As per the documentation, This commands works on either --cidr or --source-group. so If you have multiple IP addresses then I would say the only option is to run the same command multiple times for the individual IP address (which would take the form of 1.1.1.1/32).
Or,
You can list all the ipadress in cidr format (1.1.1.1/32) in a file (each ip address on a new line) and then run a for loop over it running above command for each iteration. e.g.
for i in `cat ip_address_cidr.txt`; do aws ec2 revoke-security-group-ingress --group-name MySecurityGroup --protocol tcp --port 22 $i; done
I have not tested above command syntax but that should do it so that you can revoke the rules in a single one-liner command.
I think that this is what you are looking for: How to Close All Open SSH Ports in AWS Security Groups
Here a solution for a specific security group-id:
#!/bin/bash
sg = {security group}
# get the cidrs for the ingress rule
rules=$(aws ec2 describe-security-groups --group-ids $sg --output text --query 'SecurityGroups[*].IpPermissions')
# rules will contain something like:
# 22 tcp 22
# IPRANGES 108.42.177.53/32
# IPRANGES 10.0.0.0/16
# 80 tcp 80
# IPRANGES 0.0.0.0/0
# luckily, aws returns all ipranges per port grouped together
# flag for if we are reading ipranges
reading=0
# loop returned lines
while read -r line; do
# split the line up
rulebits=($line)
# check if if we are reading ssh port ipranges
if [ $reading -eq 0 ] ; then
# we are not reading ipranges
# check if '22 tcp 22'
if [ ${rulebits[0]} == "22" ] && [ ${rulebits[1]} == "tcp" ] && [ ${rulebits[2]} == "22" ] ; then
# found it
reading=1
fi
else
# we are reading ipranges
# check if first word is 'IPRANGES'
if [ ${rulebits[0]} == "IPRANGES" ] ; then
# found a cidr for open ssh port
cidr=${rulebits[1]}
echo -n found port 22 open cidr $cidr closing...
# close it
result=$(aws ec2 revoke-security-group-ingress --group-id $sg --protocol tcp --port 22 --cidr $cidr --output text)
if [ "$result" == "true" ] ; then
echo " OK"
else
echo " ERROR"
fi
else
# new port
reading=0
fi
fi
done
revoke-security-group-ingress in version 2 gives us to specify multiple IP CIDRS. See below solution written in PHP which I am trying to clean in multiple region and multiple prots.
To specify multiple rules in a single command use the --ip-permissions option https://awscli.amazonaws.com/v2/documentation/api/latest/reference/ec2/revoke-security-group-ingress.html
$cleanports = [22,5984];
$sgids = [["sgid"=>"sg1","region"=>"us-east-1"],["sgid"=>"sg1","region"=>"us-east-1"]];
foreach($sgids as $sgidDetail){
$iprules = json_decode(shell_exec("/usr/bin/aws ec2 describe-security-groups --group-ids {$sgidDetail['sgid']} --region {$sgidDetail['region']} --query 'SecurityGroups[*].IpPermissions'"), true)[0];
foreach ($iprules as $key => $ips) {
if(!empty($ips['FromPort']) && !empty($ips['ToPort']) && in_array($ips['FromPort'], $cleanports) && in_array($ips['ToPort'], $cleanports)){
echo "\n\n";
echo shell_exec("/usr/bin/aws ec2 revoke-security-group-ingress --group-id {$sgidDetail['sgid']} --region {$sgidDetail['region']} --ip-permissions '".json_encode($ips)."'");
}
}
}