The overall deployment failed because too many individual instances failed deployment, too few healthy instances are available for deployment, or some instances in your deployment group are experiencing problems. (Error code: HEALTH_CONSTRAINTS)
Note : 1. Already installed AWS code deploy agent
2. Roles created
Please assist and let me know for any more information.
You can see more information about the error by accessing the deployment ID events:
Here you can check all the steps and their detailed status:
If there are any errors you can click on the Logs column.
There can be different reasons why your deployment failed. As instructed above, you can see click on the deployment ID to go to the deployment details page and see which instances failed. You can then click on the view events for each instance to see why deployment to that instance failed. If you do not see any link to "View events", and no event details for that instance, it is likely that the agent is not running properly. Otherwise, you should be able to click on "View events" to see which lifecycle event failed. You can also log in to the failed instance and view the host agent logs to get more information.
Related
I am using the AWS interface to configure my services on ECS. Before the interface change, I used to be able to access a screen that would allow me to see why the task had failed (like in the example below), that interface could be accessed from the ECS service events by clicking on the taskid. Does anyone know how to get the task stopped reason data with the new interface?
You can see essentially the same message if you do the following steps:
Select your service from your ECS cluster:
Go to Configuration and tasks tab:
Scroll down and select a task. You would want to chose one which was stopped by the failing deployment:
You should have the Stopped reason message:
After eb deploy the environment gets stuck in Health: 'Severe'.
It show the following warning in recent events:
Environment health has transitioned from Info to Severe. ELB processes
are not healthy on all instances. Application update in progress on 1
instance. 0 out of 1 instance completed (running for 3 minutes). None
of the instances are sending data. ELB health is failing or not
available for all instances.
I'm not able to ssh into the instance: connection reset by peer. (I'm normally able to ssh into the instance without any issues).
The request logs function doesn't work because:
An error occurred requesting logs: Environment named
portal-api-staging is in an invalid state for this operation. Must be
Ready.
Cloudwatch logs only contain the same message from 'recent events'.
How do I figure out what why the deployment fails?
AWS documentation says I should check the logs or ssh into the instance, but none of those options work.
We found the issue. The deployment was somehow getting stuck and rolled back. Changing deployment policy to AllAtOnce and disabling RollingUpdateEnabled fixed it.
I can't deploy a new version on Elastic Beanstalk.
Everything was working fine until I tried to deploy a new version where I have lots of issues (It is not the first time I deploy a new version on this environment, I already have deployed dozens). When I manage to fixe all of them I got those errors:
Failed to deploy application.
During an aborted deployment, some instances may have deployed the new application version. To ensure all instances are running the same version, re-deploy the appropriate application version.
Unsuccessful command execution on instance id(s) 'i-...'. Aborting the operation
I redeploy the version which does not work.
Here is the Elastic Beanstalk console:
Elastic Beanstalk console
Elastic Beanstalk events
The request logs button from Elastic Beanstalk return nothing.
The system log from EC2 instance shows the last working version logs.
I enable the CloudWatch logs from Configuration navigation pane. It added 4 files to CloudWatch logs:
/var/log/eb-activity.log -> empty so far
/var/log/httpd/access_log -> empty so far
/var/log/httpd/error_log -> empty so far
/environment-health.log -> Command is executing on all instances (56 minutes or more elapsed).", "Incorrect application version found on all instances. Expected version \"prod-v1.7.28-0\" (deployment 128).
It is an Amazon Linux, t2.medium instance with Apache as web server
What I already try:
Change the name of .zip each time to be different of other zip already deploy
Terminate the instance and the loadBalancer automatically create a new one
Reboot the instance
Rebuild Elastic Beanstalk environment
Deploy a simplest code
I tried to deploy just a zip with the code below but I got same errors.
<html>
<head>
<title>This is the title of the webpage!</title>
</head>
<body>
<p>This is an example paragraph. Anything in the <strong>body</strong> tag will appear on the page, just like this <strong>p</strong> tag and its contents.</p>
</body>
</html>
It always go back to last working version and when I tried to deploy the new version it does not work.
On some post I see some people telling it is maybe because the instance is too small but before it was working perfectly and the size does not change since then.
If you have some questions or ideas I will be very thankful.
Have a nice day !
Answer:
The issue was in the logs like you said. I had to ssh into my EC2 instance to reached them. The error was in the file cfn-init-cmd.log.
One of the command was waiting for an input so it timed out with no error message.
You should check the logs of the EBS for any hints as to what goes wrong with your deployment. The AWS console
can be helpful for that.
There are also the logs that can be acquired from EC2:
CloudWatch logs is another thing to check.
You should also check the autoscaling group, and see if there are any health checks there. What kind of checks are these? What's the grace period?
Here's a list of reasons that an EC2 health check could fail.
You could launch a better ec2 instance for troubleshooting.
Instance status checks.
The following are examples of problems that can cause instance status checks to fail:
Failed system status checks
Incorrect networking or startup configuration
Exhausted memory
Corrupted file system
Incompatible kernel
Also rebuilding is really a drastic step as it destroys and rebuilds all your resources. Your ELB DNS for example will be gone, any associated EIP will be released. These things can't be reclaimed.
I also faced same issue and deleted the wrong application versions. And increased the command timeout.
Default max deployment time -Command timeout- is 600 (10 minutes)
Go to Your Environment → Configuration → Deployment preferences → Command timeout
Increase the Deployment preferences higher like 1800 and then try to deploy the previous working application version. It will work.
I'm hoping I can get some help with this deployment issue that I'm facing:
I have created an RDS instance and can see it is "Available" by looking at the dashboard. I then use the Elastic Beanstalk CLI to deploy my application and the deployment is successful.
However, when I access the endpoint I am getting a 502 Bad Gateway from nginx. After checking the logs I can see the following error from my Node.js app:
Error: connect ETIMEDOUT x.x.x.x:5432 (ip ommitted)
As per the AWS documentation on this I have tried to assign the auto generated security group from my Elastic Beanstalk instance to my RDS instance, but I am still getting the same error.
Is there something I have misunderstood in the documentation here? I would be very grateful if anyone can point me in the right direction here.
Thank you in advance.
Managed to figure this out after a lot of trial and error. Turns out that it wasn't too tricky.
Go to your EB environment -> Configuration
Click "Edit" next to "Instances"
Note down the security group ID that is selected at the bottom
Create a new security group e.g. "my-eb-instance-rds-access"
Under "Inbound rules" select "Add rule". Choose whichever DB service you are using and it should automatically fill the port. Set source to "Custom" and then click in the search box. Select the security group that your EB instance has that you noted down earlier.
Click "Create security group"
Find your RDS instance and click "Modify"
Scroll down and find "Connectivity". Then select the security group that you just created from the drop down box.
Scroll all the way to the bottom and hit continue. Here I found there to be two options: one that updates the changes immediately and the other that waits for regular scheduled maintenance. I'm no expert but I selected the "immediately" option since the database is not being used in production yet so some downtime was not a problem.
Your EB instance should now be able to connect! This worked for me even after re-deploying.
Disclaimer: I am by no means an expert. This was done purely by trial and error. If anyone has any tips or improvements I'd be happy to hear them and edit the answer.
I am trying to configure my docker hub image with aws ecs..I have created repository, cluster, and task while running task am getting an error as an essential container in task exited 1. while trying to get exact error details I have found that some of my variables are shown as not configured.
find the screenshot attached of errors.
cluster details
error detail
You should setup the "Log Configuration" by specifying a log configuration in your task definition. I would recommend the awslogs configuration type, as this lets you see the logs from your container right inside the console.
Once you do that you will get a new tab on the task details screen called "Logs" and you can click that to see the output from your container as it was starting up. You will probably see some kind of error or crash as the "Essential container exited" error means that the container was expected to stay up and running, but it just exited.
I had to expand the corresponding container details in the stopped task and check the "Details" --> "Status reason", which revealed the following issue:
OutOfMemoryError: Container killed due to memory usage
Exit Code 137
After increasing the available container memory, it worked fine.
I had a similar issue. You can setup the cloudwatch log, there you can get the full error log which will help you to debug and fix the issue. below are the part of the taken taken from aws official documentation.
Using the auto-configuration feature to create a log group
When registering a task definition in the Amazon ECS console, you have the option to allow Amazon ECS to auto-configure your CloudWatch logs. This option creates a log group on your behalf using the task definition family name with ecs as the prefix.
To use log group auto-configuration option in the Amazon ECS console
Open the Amazon ECS console at https://console.aws.amazon.com/ecs/.
In the left navigation pane, choose Task Definitions, Create a new Task Definition, alternatively, you can also do create a revision of the exiting task definition.
Select your compatibility option and choose Next Step.
Choose Add container.
In the Storage and Logging section, for Log configuration, choose Auto-configure CloudWatch Logs.
Enter your awslogs log driver options. For more information, see Specifying a log configuration in your task definition.
Complete the rest of the task definition wizard.
I have been stuck on the same error.
The problem ended up being "Task tagging configuration" > DISABLE "Enable ECS managed tags"
When this parameter is enabled, Amazon ECS automatically tags your tasks with two tags corresponding to the cluster and service names. These tags allow you to identify tasks easily in your AWS Cost and Usage Report.
Billing permissions are separate and not by default assigned when you create a new ECS cluster and task definition with the default setting. This is why ECS was failing with "STOPPED: Essential container in task exited"