I did the AWS misfits tutorial and I thought the cleanup would be as simple as deleting the Cloud Formation stack. However, items failed to delete so I am trying to clean them up manually to assist the Cloud Formation stack deletion.
I keep getting an error when trying to Detach the network interfaces (I am logged into the root account):
Error detaching network interfaces:
eni-0047gfhfgh8ab0e: You are not allowed to manage 'ela-attach' attachments.
eni-0f4a46hgfha757e: You are not allowed to manage 'ela-attach' attachments.
I am unable to delete my VPC without these being deleted.
The stack as a whole is failing to delete because of the following:
The following resource(s) failed to delete: [InternetGateway,
PublicSubnetTwo, VPC, GatewayAttachement, PublicSubnetOne].
I had the same issue, which gave me the following message:
This could be because there is a service in use which still uses the network interface. You could try some of the following things:
Remove unused VPC links from API gateway
Remove unused VPC Endpoint services
Remove unused NAT gateways
Remove unused ECS/EKS clusters
Remove unused load balancers
Remove unused EFS mounts
If that doesn't help, there is something wrong/stuck on the underlying OS, you should wait for it to resolve by itself or report it. I had an ENI deployed by a Lambda function after deleting the lambda function, the ENI got stuck. After some time I was able to detach the ENI.
Update: for the people using the aws cli, AWS support posted a bash script and documentation to easily identify ENIs that are still hanging around somewhere (source).
AWS offers this bash script as help to find out the dependencies:
#!/bin/bash
vpc="vpc-xxxxxxxxxxxxx"
region="yy-yyyy-y"
aws ec2 describe-internet-gateways --region $region --filters 'Name=attachment.vpc-id,Values='$vpc | grep InternetGatewayId
aws ec2 describe-subnets --region $region --filters 'Name=vpc-id,Values='$vpc | grep SubnetId
aws ec2 describe-route-tables --region $region --filters 'Name=vpc-id,Values='$vpc | grep RouteTableId
aws ec2 describe-network-acls --region $region --filters 'Name=vpc-id,Values='$vpc | grep NetworkAclId
aws ec2 describe-vpc-peering-connections --region $region --filters 'Name=requester-vpc-info.vpc-id,Values='$vpc | grep VpcPeeringConnectionId
aws ec2 describe-vpc-endpoints --region $region --filters 'Name=vpc-id,Values='$vpc | grep VpcEndpointId
aws ec2 describe-nat-gateways --region $region --filter 'Name=vpc-id,Values='$vpc | grep NatGatewayId
aws ec2 describe-security-groups --region $region --filters 'Name=vpc-id,Values='$vpc | grep GroupId
aws ec2 describe-instances --region $region --filters 'Name=vpc-id,Values='$vpc | grep InstanceId
aws ec2 describe-vpn-connections --region $region --filters 'Name=vpc-id,Values='$vpc | grep VpnConnectionId
aws ec2 describe-vpn-gateways --region $region --filters 'Name=attachment.vpc-id,Values='$vpc | grep VpnGatewayId
aws ec2 describe-network-interfaces --region $region --filters 'Name=vpc-id,Values='$vpc | grep NetworkInterfaceId
aws ec2 describe-carrier-gateways --region $region --filters Name=vpc-id,Values=$vpc | grep CarrierGatewayId
aws ec2 describe-local-gateway-route-table-vpc-associations --region $region --filters Name=vpc-id,Values=$vpc | grep LocalGatewayRouteTableVpcAssociationId
My issue was a Transit Gateway Attachment and I could figure it out at some point.
Source: https://aws.amazon.com/premiumsupport/knowledge-center/troubleshoot-dependency-error-delete-vpc/
If you're working with Lambda Functions in VPC.
This is caused by old Lambda version is still referring to the ENI.
Try below to see if it's this case:
aws lambda list-versions-by-function --function-name FUNCTIONNAME | jq -r '.Versions | .[] | [.LastModified, .Version, .VpcConfig.VpcId] | #tsv' | sort
I had this same issue. For me, I had to delete an Endpoint Service associated with my ELB first, then I could start to get rid of the network interfaces that were set up.
Try to verify if you have a EFS mount target asociated at the ENI and delete it, after try to delete the network interface that's work for me.
Modify the VPC Endpoint Service to reject VPC Endpoint (deselect
Require acceptance for endpoint) https://docs.aws.amazon.com/vpc/latest/userguide/modify-endpoint-service.html
Delete the VPC Endpoint Service
Continue deleting other resources or try again the stack deletion task in
Cloudformation.
The VPC Endpoint Service deletion is restricted by the no longer existing VPC Endpoint.
In my case I had an EFS mount that was preventing my subnets from being deleted. My solution was to visit the AWS Management Console, search for EFS, and delete the corresponding mount. After that I was able to delete the subnets and VPC.
I had the same issue. The root problem is that it won't let you delete anything because everything has something else dependent on it. The solution was to scroll down below the list of each type of resource and find the right tab to edit/delete the connections to other resources. Once those are gone, you will be able to delete the resource at the top of the page. I'm not sure which break ultimately enabled the cloudformation to delete successfully, but just going through the VPC dashboard and hitting each of these should fix the problem.
The problem for me was that my load balancer was created with "Deletion protection"
enabled. I logged on to AWS console, found my load balancer and edited its attributes setting "Deletion protection" to false. This allowed me to delete the load balancer and everything else.
if error occurs on network interface deletion , wait until NAT gateway and internet gateway entry removes on the respective page. Otherwise remove public or static IP associate with service in the vpc
In my case I had a DataSync instance - when I removed that I could delete these resources
Check whether any lambda function/db instance/ec2 instance still there in the VPC-> any of the subnet.
Delete that particular instance first and then the network interface will get deleted automatically.
For identifying the instance you can follow the following steps :
1.Search for the network interface in the AWS Network interface console.
2.Click on that open the detailed view and then check for interface type and delete the mentioned resource from there.
Error:
Failed to detach the network interface. API error: "You do not have
permission to access the specified resource."
Solution: In order to find the resources that are still using the Elastic Network Interface, pull up the ENI details by clicking on the Network Interface ID, and use these two fields:
Description: should indicate which service is using the ENI (ie ECS, ELB, Lambda, etc), but you might have to use some clues in the attachment ARN.
Security groups: should help you associate the specific cluster that's associated with it.
Once I deleted the ECS cluster that was associated with the ENI, the ENI resources went away automatically.
Also delete any Global Accelerator instances associated with the VPC.
I'd like to add to this conversation.
I had a similar issue and had to clear out some roles and 2x cloud9 environments, that I had ignored.
Worked the charm as my network interface was no longer associated with my cloud9 environments.
Related
Recently got an email titled, "Important News from AWS About Amazon EC2-Classic" describing some changes that need to occur. These emails from AWS usually reference the effected resources though and this one did not. I am having a hard time identifying what resources in our account are effected by this. All our EC2 instances are in a VPC and I am not even sure if anything needs to change or not.
Is there a way to identify that an EC2 instance is classic?
I have looked through their linked documentation and gone through the instances we have but I cannot tell if they are "classic" of not.
You can identify the EC2-Classic env by checking the instance has VPC ID or not.
EC2 console
VPC ID is not shown by default. Enable VPC ID from Preference -> Attribute columns.
Then if VPC ID attribute is -, that means the instance is EC2-Classic. (Except that the instance state is not terminated.)
CLI
2 ways for checking. Output is none unless EC2-classic instances exist.
Describe instance with EC2-Classic env.
aws ec2 describe-instances --filters Name=instance-state-name,Values=pending,running,shutting-down,stopping,stopped | jq '.Reservations[].Instances[] | select(.VpcId == null)'
Describe the instance if it is the EC2-Classic.
aws ec2 describe-instances --instance-id i-xxxxxxxxxxxx --filters Name=instance-state-name,Values=pending,running,shutting-down,stopping,stopped | jq '.Reservations[].Instances[] | select(.VpcId == null)'
jq select for terminated state
This is another way to filter the result of aws ec2 describe-instances.
Adding .State.Name != "terminated" and in jq select works the same as --filters .... This might be more readable.
aws ec2 describe-instances | jq '.Reservations[].Instances[] | select(.State.Name != "terminated" and .VpcId == null)'
Edit note: Thanks to the suggestion from #AUdden, I have modified the CLI code for filtering out the terminated state. When we terminated instances (not stop), the instances exist for a while in terminated state. The terminated instances are not associated with VPC anymore. To do that, I have added --filters Name=instance-state-name,Values=pending,running,shutting-down,stopping,stopped.
Amazon provides a script to identify all resources affected by the retirement, including resources you may not consider such as security groups.
Important: Check the file errors.txt after running the script. The script will happily run through its steps even if there is an error (such as missing/wrong credentials) without showing any hint of trouble in the console output.
I created stack with nested stacks, there is some network interfaces, VPC etc.
I try to remove network interface but I can't because I'm getting an error
Error detaching network interface
eni-0d3be6d4c7869686a: You are not allowed to manage 'ela-attach' attachments.
Do you have any ideas how to force remove?
I had the same issue with multiple CF Stacks.
Stacks fail deleting when there are AWS constructs in use attach to the VPC. One approach that worked for me was to use the following script to find the dependancies and then delete them manually before deleting the VPC. (Delete all dependancies that come out of the script manually, and try deleting Network Interfaces at last). Once done, Then tried deleting the CF stacks from the mgmt console, which worked without any issue.
Let us know if this worked.
#!/bin/bash
vpc="vpc-xxxxxxxxxxxxx"
aws ec2 describe-internet-gateways --filters 'Name=attachment.vpc-id,Values='$vpc | grep InternetGatewayId
aws ec2 describe-subnets --filters 'Name=vpc-id,Values='$vpc | grep SubnetId
aws ec2 describe-route-tables --filters 'Name=vpc-id,Values='$vpc | grep RouteTableId
aws ec2 describe-network-acls --filters 'Name=vpc-id,Values='$vpc | grep NetworkAclId
aws ec2 describe-vpc-peering-connections --filters 'Name=requester-vpc-info.vpc-id,Values='$vpc | grep VpcPeeringConnectionId
aws ec2 describe-vpc-endpoints --filters 'Name=vpc-id,Values='$vpc | grep VpcEndpointId
aws ec2 describe-nat-gateways --filter 'Name=vpc-id,Values='$vpc | grep NatGatewayId
aws ec2 describe-security-groups --filters 'Name=vpc-id,Values='$vpc | grep GroupId
aws ec2 describe-instances --filters 'Name=vpc-id,Values='$vpc | grep InstanceId
aws ec2 describe-vpn-connections --filters 'Name=vpc-id,Values='$vpc | grep VpnConnectionId
aws ec2 describe-vpn-gateways --filters 'Name=attachment.vpc-id,Values='$vpc | grep VpnGatewayId
aws ec2 describe-network-interfaces --filters 'Name=vpc-id,Values='$vpc | grep NetworkInterfaceId
Reference : https://aws.amazon.com/premiumsupport/knowledge-center/troubleshoot-dependency-error-delete-vpc/
Find the resource that the ENI is attached to. It could be a Lambda function or ELB, for example. Was that resource created outside of your CloudFormation stack? If so, you'll need to delete that resource. If it was created within the CloudFormation stack, then you might just need to wait and retry (e.g. if a warm Lambda function was holding on to the ENI).
Steps are described in more detail here. Other ideas here.
eksctl delete cluster --region=us-west-2 --name=myeks-2
[ℹ] using region us-west-2
[ℹ] deleting EKS cluster "myeks-2"
[ℹ] cleaning up LoadBalancer services
[✖] cannot delete orphan ELB Security Groups: cannot delete security group k8s-elb-aaa: DependencyViolation: resource sg-yyy has a dependent object
I created an EKS cluster, I helm installed a few things, now I want to blow it away, and recreate, and try again. It seems there is an orphan ENI + SG that can't be deleted and I can't understand why.
aws ec2 delete-security-group --group-id "sg-yyy"
An error occurred (DependencyViolation) when calling the DeleteSecurityGroup operation: resource sg-yyy has a dependent object
I can find one network interface referencing this security group with:
aws ec2 describe-network-interfaces --filters Name=group-id,Values=sg-yyy | jq '.NetworkInterfaces[].NetworkInterfaceId'
If I try to delete that, I get
aws ec2 delete-network-interface --network-interface-id eni-xxx
An error occurred (InvalidParameterValue) when calling the DeleteNetworkInterface operation: Network interface 'eni-xxx' is currently in use.
If I do:
aws ec2 describe-instances > instances.json
aws ec2 describe-nat-gateways > nat-gateways.json
aws ec2 describe-network-acls > network-acls.json
aws ec2 describe-network-interfaces > network-interfaces.json
aws ec2 describe-security-groups > security-groups.json
and then grep eni-xxx *.json and grep sg-yyy *.json, I get three results: the two entity results themselves and one reference from eni-xxx to sg-yyy. No EC2 instances are referencing either... I'm not sure how to fix this.
FYI, I've not done any custom AWS stuff with this EKS cluster. No custom networking or AWS commands. I was just hoping to create and delete/recreate test clusters and am trying to resolve this blocking error.
the solution was to manually delete the ELB using the security group:
aws elb describe-load-balancers | jq '.LoadBalancerDescriptions[] | select (.SecurityGroups[] | contains("sg-yyy")) | .LoadBalancerName'
aws elb delete-load-balancer --load-balancer-name <load-balancer-name>
I am trying to streamline the process for creating VPC/EC2 environments without using the gui. I also want to automate it by telling a script what I want created with what properties.
I decided that the best place to start is to create a VPC and create an EC2 instance with in it.
I am using
aws ec2 create-vpc --cidr-block 10.0.0.0/16
But I wanted to name it something like myVPC. Is there a way to do things like this? I am very new to this so if you have any documentation regarding this please send it my way.
Thank you!
Generally, AWS resources don't have names. Instead they have IDs. What passes for a 'name' of Production is actually a tag with the key/value pair Name=Production.
To set a name tag for a VPC, use the CLI's ec2 create-tags command. For example:
aws ec2 create-tags --resources vpc-1a2b3c4d --tags Key=Name,Value=Production
If you really want a one liner:
aws ec2 create-vpc --cidr-block 10.0.0.0/16 --output text | awk '{print $NF}' | xargs aws ec2 create-tags --tags Key=Name,Value=MyVPC --resources
It is a concatenation of two commands explained below.
Adding a tag while creating a VPC is not supported yet. Create a VPC like the following. The last value is VPC ID.
aws ec2 create-vpc --cidr-block 10.3.0.0/16 --output text
VPC 10.3.0.0/16 dopt-a54153c7 default False pending vpc-f13d7295
Use create-tags to add a tag to the created VPC
aws ec2 create-tags --resources vpc-f13d7295 --tags Key=Name,Value=MyVPC
You could use --tag-specifications:
aws ec2 create-vpc --cidr-block 10.0.0.0/24 --tag-specifications "ResourceType=vpc,Tags=[{Key=Name,Value=MyVPC}]"
https://docs.aws.amazon.com/cli/latest/reference/ec2/create-vpc.html#options
Unrelated to your specific question, but allow me to highly recommend AWS CloudFormation for managing these resources. It's a nicer method of definition that just the CLI, allows you to group resources or delete a stack. I use the CLI to call the Cloudformation, specifying a template.
http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-ec2-vpc.html
Is there any way to find out status of AWS EC2 instances, which are running on various different regions, from one EC2 instance which is present in any one of region by using EC2 API tool ?
How this is possible ?
I got the answer :-
ec2-describe-instances instance-ID --region region
Example :-
ec2-describe-instances i-f82d5ca0 --region eu-west-1
Where instance ID is EC2 instance ID which is located in region eu-west-1
Thats all .
Or in the new unified AWS CLI, this is slightly different:
aws ec2 describe-instances --instance-id i-f82d5ca0
You can also change the --output into json, text, or a table