I have a problem with AWS Elastic Beanstalk. I tried to delete an environment. It started the process, but after a few minutes the environment "health bar" went to grey and gave me the following errors:
"Deleting security group named: XXXXX failed Reason: resource YYYYY has a dependent object"
"Stack deletion failed: The following resource(s) failed to delete: [AWSEBSecurityGroup]."
I tried to delete the security group from the error message, but I got this:
"XXXXXX: resource XXXXX has a dependent object"
After this I wanted to delete the dependence from the EB environment, but because it's Grey, it didn't allow me to do that.
I browsed the internet for hours, found a possible solution, where I need to do something at the EC2's Network Interfaces page, but it doesn't say any specific option or information.
Try this, I was suggested by AWS support and it worked for me.
You need to go to your CloudFormation console and retry deletion of the CloudFormation stack which the Beanstalk environment used.
The deletion may fail, but after retrying it will prompt you if you want to skip the "AWSEBRDSDatabase" resource that failed to delete. You can just confirm that you want to skip deletion (since you have actually already deleted it).
This should remove the CloudFormation stack
Then you can retry deletion of the Beanstalk environment from the Beanstalk console.
Is the security group being referenced by RDS or something in S3? If that is the case, you'll have to delete the dependency in either RDS or S3.
The error message is saying something outside of your environment is still using the security group and it can't be deleted for this reason.
Go to EC2 under AWS console's Compute
Go to Security Groups under NETWORK & SECURITY on the sidebar
Find your misbehaving security group on the page
Check it and choose Delete Security Group from the Actions menu
You will be presented with a link that will lead you to the security group or instances it depends on.
Keep going until you get to the parent Security Group or instances and delete them.
Make sure you don't delete any important Security Groups or Instances!
In case this happens due to a similar error but due to RDS attached to this,
Stack deletion failed: The following resource(s) failed to delete: [AWSEBRDSDatabase].
This happens when you terminate the RDS instance manually from RDS listing console. I resolved this by launching another instance with the same DB instance ideIntifier name.
Once this is created, you can terminate the Elastic Beanstalk environment successfully. This works.
Use https://github.com/mingbowan/sgdeps to find your security group dependencies and then break the dependencies.
Had this happen where I was using the security group created by EB with a non-eb created RDS instance.
I modified the RDS instance to stop using the EB-created security group and was able to do a successful termination of the environment and application. I used the eb cli 3.x and eb terminate --all --force to get a fresh start on the application.
In my case i have white list EBS instance into RDS security group , so deleted from RDS solved problem.
Related
I'm hoping I can get some help with this deployment issue that I'm facing:
I have created an RDS instance and can see it is "Available" by looking at the dashboard. I then use the Elastic Beanstalk CLI to deploy my application and the deployment is successful.
However, when I access the endpoint I am getting a 502 Bad Gateway from nginx. After checking the logs I can see the following error from my Node.js app:
Error: connect ETIMEDOUT x.x.x.x:5432 (ip ommitted)
As per the AWS documentation on this I have tried to assign the auto generated security group from my Elastic Beanstalk instance to my RDS instance, but I am still getting the same error.
Is there something I have misunderstood in the documentation here? I would be very grateful if anyone can point me in the right direction here.
Thank you in advance.
Managed to figure this out after a lot of trial and error. Turns out that it wasn't too tricky.
Go to your EB environment -> Configuration
Click "Edit" next to "Instances"
Note down the security group ID that is selected at the bottom
Create a new security group e.g. "my-eb-instance-rds-access"
Under "Inbound rules" select "Add rule". Choose whichever DB service you are using and it should automatically fill the port. Set source to "Custom" and then click in the search box. Select the security group that your EB instance has that you noted down earlier.
Click "Create security group"
Find your RDS instance and click "Modify"
Scroll down and find "Connectivity". Then select the security group that you just created from the drop down box.
Scroll all the way to the bottom and hit continue. Here I found there to be two options: one that updates the changes immediately and the other that waits for regular scheduled maintenance. I'm no expert but I selected the "immediately" option since the database is not being used in production yet so some downtime was not a problem.
Your EB instance should now be able to connect! This worked for me even after re-deploying.
Disclaimer: I am by no means an expert. This was done purely by trial and error. If anyone has any tips or improvements I'd be happy to hear them and edit the answer.
In AWS Elastic Beanstalk, I created an environment and through it a database.
I went to RDS and renamed the instance name and then delete the RDS instance
Now I am trying to delete the environment and the procedure fails:
ERROR Stack deletion failed: The following resource(s) failed to delete: [AWSEBRDSDatabase].
What can I do in order to detach the environment from the RDS?
Any ideas?
Thank you very much
#Andreas What you can do is rename the RDS instance back to the original one and takes a Snapshot of it.
if you don't rename and try to delete the beanstalk what happens is beanstalk will keep on looking for the RDS Instance.
Now you can delete the Beanstalk, after successful deletion of the environment you can again create the RDS Instance via that Snapshot (which you have created previously)
What else you can try is go to the Configuration tab of the beanstalk and change the configuration of Database and update the beanstalk. This will preserve your beanstalk env also and changes the RDS configuration also.
For me helped to going to Cloud Formation and deleting stack there
https://aws.amazon.com/premiumsupport/knowledge-center/elastic-beanstalk-deleted-rds-database/
I wanted to delete an ASP.NET MVC application on the Elastic Beanstalk. But before deleting the app I first deleted the RDS server instance linked to the application. The RDS instance got deleted succesfully but now when I try to delete the Elastic Beanstalk application I get the following error message in the console -
Stack deletion failed: The following resources failed to delete AWSEBRDSDatabase
How do I proceed to delete this? I am not able to find any help regarding this issue!
ran across the same issue myself today.. there doesn't appear to be a way to delink the deleted RDS instance in the Elastic Beanstalk configuration sadly. I did however manage to delete the application though by first recreating the RDS instance using the same name as before.
Go to AWS RDS > Intances > Launch DB Instance
recreate the deleted instance using the exact name that was previously used. Once successfully created, go back to Elastic Beanstalk and it allows you to terminate the environment. Bit of a pain but works none the less.
Thanks
TJ
It happened to me and the way I was able to resume the removal of Elasticbeanstalk sucked on this status was to go through CloudFormation panel and remove the stack with failed status as described on: https://aws.amazon.com/premiumsupport/knowledge-center/cloudformation-stack-delete-failed/
After that I was able to remove the Environment and Application under Elasticbeanstalk.
I had this error "The following resource(s) failed to delete: [AWSEBRDSDatabase]" this helped me to resolve the problem.
Open the AWS CloudFormation console.
Choose the stack that's stuck in DELETE_FAILED status.
Choose Delete.
Note: A window opens and lists the resources that failed to delete.
4. In the window, select all the resources that you want to retain, and then choose Delete stack.
I checked to retain resource, even though the resource that was retained didn't exist but it helped me to delete it, if I didn't check it it was not deleted
https://aws.amazon.com/premiumsupport/knowledge-center/cloudformation-stack-delete-failed/
https://aws.amazon.com/premiumsupport/knowledge-center/elastic-beanstalk-deleted-rds-database/
Below the screenshot, it seems to be successful only in one Availability Zone.
I checked the codedeploy logs for a failed instance, and I found that there was an error, I think it is recognized as an on-premise instance.
2018-01-10 04:40:22 INFO [codedeploy-agent(2696)]: On Premises config file does not exist or not readable
2018-01-10 04:40:43 ERROR [codedeploy-agent(2696)]: CodeDeploy Instance Agent Service: CodeDeploy Instance Agent Service: error during start or run: InstanceMetadata::InstanceMetadataError - Not an EC2 instance and region not provided in the environment variable AWS_REGION. Please specify your region using environment variable AWS_REGION.......
I've searched for about three days for this issue, but there was no mention in the AWS documentation. In the production env, I plan to use two Availability Zones attached to the auto scaling group. I wonder if I'm overlooking the other thing except CodeDeploy... What should I check? Thank you in advance.
[Updated]
I update with ASG and ASG Config screent shot. There's no special, it's vanila and default process. I waiting 5 days from AWS support center but still pending response.
Auto Scaling Group -----
Auto Scaling Group Launch Config -----
Finally, I found out why codedeploy failed across multiple availability zones on Windows 2016. Also, this problem seems to be an issue with Windows 2016 EC2 itself rather than ASG or codedeploy(I have not tested it on linux). There are 2 solutions I found,
Shut down the server safely by clicking the button "Shutdown with Syspre" in Ec2LaunchSettings. And then you can create AMI as usal.
Run the C:\ProgramData\Amazon\EC2-Windows\Launch\Scripts\InitializeInstance.ps1 -Schedule script manually. The argument "-Schedule" is required. And then you can create AMI as usal.
The first method is an intuitive and convenient way(GUI), and the second method is appropriate for automate a powershell script. I have confirmed that both methods succeed in deploying to multiple AZs. There were no errors in the logs recorded by codedeployagent.
To be more specific, codedeployagent leaves various logs at the time of deployment, and I found that the agent seems to use meta-info from 169.254.169.254. When I failed, the log say "You are On-Premise Instance.". Probably the deployment fails because the instance can not get meta-info. In the following document, I have received a lot of help and all of my solutions are listed.
https://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/ec2launch.html
Especially, in the document
.....In Windows PowerShell, run the following command so that the system schedules the script to run as a Windows Scheduled Task. The script runs one time during the next boot and then disables these tasks from running again....
C:\ProgramData\Amazon\EC2-Windows\Launch\Scripts\InitializeInstance.ps1 -Schedule
I created AWS EB environments with RDS MySQL DB 1st time. But, I realized that I don't need to use MySQL so, I switched to sqlite3 and terminate RDS MySQL DB. After this, I can't update application version by using git aws.push. AWS EB shows the below error message all the time.
Service:AmazonCloudFormation, Message:Stack named 'awseb-e-xxxxxx-stack' aborted operation.
Current state: 'UPDATE_ROLLBACK_IN_PROGRESS' Reason: The following resource(s) failed to update: [AWSEBAutoScalingGroup].
Updating Auto Scaling group failed Reason: Template error: DBInstance xxxxxxx doesn't exist
How can I fix this issue? Thanks.
As some have noted, you actually cannot manually delete your RDS instance from an elasticbeanstalk environment.
Per another answer you can change this via some options using one of the aws apis:AWS Elastic Beanstalk change RDS Endpoint
My solution was to terminate the environment and just start a new one from a saved configuration.
I was unable to terminate an EB environment of mine due to previously having manually deleted the RDS instance associated with the environment.
I was able to solve this issue by creating a new RDS instance with the same identifier as the deleted instance, then terminating the environment like normal.
If you deleted your RDS instance manually, Elastic BeansTalk does not know about that change and must be updated too.
You can go to Elastic Beanstalk console, pick up your Application and your Environment, go to "Configuration" and delete your RDS instance from the Elastic Beanstalk configuration.