I'm hoping I can get some help with this deployment issue that I'm facing:
I have created an RDS instance and can see it is "Available" by looking at the dashboard. I then use the Elastic Beanstalk CLI to deploy my application and the deployment is successful.
However, when I access the endpoint I am getting a 502 Bad Gateway from nginx. After checking the logs I can see the following error from my Node.js app:
Error: connect ETIMEDOUT x.x.x.x:5432 (ip ommitted)
As per the AWS documentation on this I have tried to assign the auto generated security group from my Elastic Beanstalk instance to my RDS instance, but I am still getting the same error.
Is there something I have misunderstood in the documentation here? I would be very grateful if anyone can point me in the right direction here.
Thank you in advance.
Managed to figure this out after a lot of trial and error. Turns out that it wasn't too tricky.
Go to your EB environment -> Configuration
Click "Edit" next to "Instances"
Note down the security group ID that is selected at the bottom
Create a new security group e.g. "my-eb-instance-rds-access"
Under "Inbound rules" select "Add rule". Choose whichever DB service you are using and it should automatically fill the port. Set source to "Custom" and then click in the search box. Select the security group that your EB instance has that you noted down earlier.
Click "Create security group"
Find your RDS instance and click "Modify"
Scroll down and find "Connectivity". Then select the security group that you just created from the drop down box.
Scroll all the way to the bottom and hit continue. Here I found there to be two options: one that updates the changes immediately and the other that waits for regular scheduled maintenance. I'm no expert but I selected the "immediately" option since the database is not being used in production yet so some downtime was not a problem.
Your EB instance should now be able to connect! This worked for me even after re-deploying.
Disclaimer: I am by no means an expert. This was done purely by trial and error. If anyone has any tips or improvements I'd be happy to hear them and edit the answer.
Related
I'm incredibly confused about how health checks work for a docker container running in ECS using AWS Fargate. I think what makes this confusing is that there's three core components working in tandem, each of which I've seen have its own "health check" concerns:
ECS
EC2
ALB
First, if I check the health check docs, it makes it very clear that the built-in HEALTHCHECK in my docker image won't be used. However, I've seen comments from others on SO that they are used, so which is it?
Concerning the health check setup for ECS, I'm not seeing any way to configure health check commands when I create a Task Definition for my ECS service via Fargate in the AWS dashboard (web interface). I'm setting up the infrastructure using the CDK in C#, but for learning purposes I look at the AWS dashboard and edit things from there. I figure I need to learn how to set things up manually before I try to automate it.
I'll mention what I do see, but I'm not sure how it all pieces together.
ECS -> Clusters -> Click cluster name -> Click service name: I see "Healthy Targets" and "Unhealthy Targets"
ECS -> Clusters -> Click cluster name -> Click service name -> Deployments and events tab: There's a log that says "service X port 80 is unhealthy in target-group Y due to (reason Health checks failed with these codes: [404]). If I click the link for Y, it takes me to "EC2 -> Target groups -> Y (fargate)", which has a "Health checks" tab. There, I can click "Edit" and specify the health check "Path". This seems to eliminate the error.
ECS -> Task definitions -> Click task def name -> Click revision name -> JSON tab: No mention of "health" anywhere in this file
From the CDK, it looks like you can set up health checks after creating ApplicationLoadBalancedFargateService, at which point you can invoke ApplicationLoadBalancedFargateService.TargetGroup.ConfigureHealthCheck(), which takes an IHealthCheck that I haven't figured out how to create yet.
Also in the CDK there is QueueProcessingFargateService (not sure how that's different from the ALB version of FargateService) that has a HealthCheck property I can initialize, whereas the ALB version does not. Just adds more confusion. I don't necessarily care about QueueProcessingFargateService itself, but it does show up in the code example for HealthCheck in the CDK docs
All of this is very confusing. The AWS web UI is absolutely horrid and difficult to navigate. I'm seeing a lot of conflicting information on SO and google search results in general about how to set up health checks. Can someone please help make sense of all of this?
Concerning the health check setup for ECS, I'm not seeing any way to configure health check commands when I create a Task Definition for my ECS service via Fargate in the AWS dashboard
You would have to do that by editing the Task Definition JSON manually, instead of using the point-and-click features of the ECS web console. The ECS web console is currently missing a lot of features.
All of this is very confusing. The AWS web UI is absolutely horrid and difficult to navigate. I'm seeing a lot of conflicting information on SO and web search results in general about how to set up health checks. What can I try next?
I recommend not using the web UI at all. Use the CDK, or use Terraform for creation of resources. Use the web UI for just looking at what was created.
As for exactly how to setup health checks, it depends on what you are trying to do. If you are using a load balancer, then the target group health checks are required. You set those up on the Load Balancer's Target Group, and you could do that through the UI since that is over in the EC2 web UI and is fully featured. Target Group health checks will perform a network request to the ECS task periodically, and ensure that is is receiving a proper response.
If you are not using a load balancer, or if you just want extra health checks in addition to the Target Group checks, you can setup health check commands in the ECS task definition. These run a command inside the container periodically. You can't really setup these via the web UI and even the higher level CDK constructs probably mask this or make it less than obvious. This is an optional, and advanced feature of ECS that most people don't use, and I believe you would have to drop down to lower-level CDK constructs if you were using the CDK.
We had a working env yesterday morning. One person was working with some VPC peering stuff. And somehow it seems blocked the ELB from being able to talk to the application servers. So we can't deploy.
We get this error
Command failed on instance. An unexpected error has occurred [ErrorCode: 0000000001].
And EB can't retrieve any logs from the app server either. So clearly it is getting blocked. The IAM of the instance has AWSElasticBeanstalkWebTier and is the same IAM used in another env that works.
Also, I can RDP to the app instance from my laptop. Though I added my IP to one of the security groups for the instance. I audited the SG's for the working env, and don't see anything specific for EB.
Everything points to something with the VPC... but what should I look at?
So turns out that that error message doesn't always mean it can't talk to the instance. We had updated the AMI a while back and used the wrong one... it wasn't compatible with the platform. So nothing worked right, including getting the logs. So the fix was to sync up the AMI and the EB platform.
Recently I've been testing AWS CodeDeploy to validate that it will be useful, and so far so good. But after seeing its workflow I started to wonder: "How can someone validate that the new environment is good, in a human way?"
Explaining in more detail:
On my "Deployment Group > Deployment Settings" using the Traffic Rerouting policy of "I will choose whether to reroute traffic", when the new environment boots up, the deployment pauses waiting for me to verify that everything is fine in this new environment. Then, after validation, I can push the "reroute traffic" button and it will proceed as expected.
To validate that the new environment is good I, as someone who has access to the machines, can SSH into one of them and do some tests. Or I can grab the Public DNS of one new machine and access it through the browser and verify that it is OK.
But is there a simpler way of validating the application on these new machines? Like having a Load Balancer that always points to the soon to be new environment that I can send to QA people. Or will I have to, for each and every deploy, manually grab information about the new environment and then send to the QA people?
In order to validate the new environment, you can add scripts as validation hooks in the Appspec that would be run after installing the new revision in the new hosts. Also, the new environment will be registered behind any load balancer you specify in the deployment configuration.
I'm trying to setup http load balancer, with auto-scaling enabled managed instance group for the back-end service.
The problem occurs when I tried to add instance groups while creating backend service, I already made my managed instance group from an instance template, but I can't see it from adding instance group panel. I chose 'select from existing groups'(not sure it's exact phrase for I'm using foreign language) and I selected asia region, but I couldn't see/select the group i created.
It gives "no instance groups in this region" message when I choose other regions like US/EU, so I guess it recognizes my group but it just doesn't show up.
http load balancer doesn't support asia region or should I have to modify some other settings to see my group?
fyi, I tried all above action on the cloud console (not with command line)
Any helps would really be appreciated!
Best,
JP
There's currently a bug preventing existing instance groups from appearing in the drop-down menu. Until it's fixed, you can add them via gcloud using this command: gcloud compute backend-services add-backend [backend-service-name] --instance-group [instance-group-name].
I have a problem with AWS Elastic Beanstalk. I tried to delete an environment. It started the process, but after a few minutes the environment "health bar" went to grey and gave me the following errors:
"Deleting security group named: XXXXX failed Reason: resource YYYYY has a dependent object"
"Stack deletion failed: The following resource(s) failed to delete: [AWSEBSecurityGroup]."
I tried to delete the security group from the error message, but I got this:
"XXXXXX: resource XXXXX has a dependent object"
After this I wanted to delete the dependence from the EB environment, but because it's Grey, it didn't allow me to do that.
I browsed the internet for hours, found a possible solution, where I need to do something at the EC2's Network Interfaces page, but it doesn't say any specific option or information.
Try this, I was suggested by AWS support and it worked for me.
You need to go to your CloudFormation console and retry deletion of the CloudFormation stack which the Beanstalk environment used.
The deletion may fail, but after retrying it will prompt you if you want to skip the "AWSEBRDSDatabase" resource that failed to delete. You can just confirm that you want to skip deletion (since you have actually already deleted it).
This should remove the CloudFormation stack
Then you can retry deletion of the Beanstalk environment from the Beanstalk console.
Is the security group being referenced by RDS or something in S3? If that is the case, you'll have to delete the dependency in either RDS or S3.
The error message is saying something outside of your environment is still using the security group and it can't be deleted for this reason.
Go to EC2 under AWS console's Compute
Go to Security Groups under NETWORK & SECURITY on the sidebar
Find your misbehaving security group on the page
Check it and choose Delete Security Group from the Actions menu
You will be presented with a link that will lead you to the security group or instances it depends on.
Keep going until you get to the parent Security Group or instances and delete them.
Make sure you don't delete any important Security Groups or Instances!
In case this happens due to a similar error but due to RDS attached to this,
Stack deletion failed: The following resource(s) failed to delete: [AWSEBRDSDatabase].
This happens when you terminate the RDS instance manually from RDS listing console. I resolved this by launching another instance with the same DB instance ideIntifier name.
Once this is created, you can terminate the Elastic Beanstalk environment successfully. This works.
Use https://github.com/mingbowan/sgdeps to find your security group dependencies and then break the dependencies.
Had this happen where I was using the security group created by EB with a non-eb created RDS instance.
I modified the RDS instance to stop using the EB-created security group and was able to do a successful termination of the environment and application. I used the eb cli 3.x and eb terminate --all --force to get a fresh start on the application.
In my case i have white list EBS instance into RDS security group , so deleted from RDS solved problem.