I have an AWS Elastic Beanstalk application that I recently updated with an error in the my (Flask) code (one line of Python with an unrecognized named argument) that resulted in an expected server error. I then tried to redeploy a previously functioning version of my app, but got an error that EB was not in a state that allowed deployment. I then attempted to abort the current operation, but got
User initiated abort was received, however the current step of the operation in progress is not cancellable. The current operation will be aborted as soon as the non-cancellable step(s) complete.
Eventually the abort succeeded, but now all my attempts to update the EB app fail with
Environment update is starting.
Deploying new version to instance(s).
[Instance: i-xxxxxxxx] Command failed on instance. An unexpected error has occurred [ErrorCode: 0000000001].
Command execution completed on all instances. Summary: [Successful: 0, Failed: 1].
Unsuccessful command execution on instance id(s) 'i-xxxxxx'. Aborting the operation.
Failed to deploy application.
ERROR: ServiceError - Failed to deploy application.
followed by a warning in the events log
During an aborted deployment, some instances may have deployed the new application version. To ensure all instances are running the same version, re-deploy the appropriate application version.
I'm now stuck in an infinite loop (regardless of where I try to update from: command line, EB console, EB previous app versions), with all of by EB websites down. It doesn't seem like this should be possible and I must be missing come operation I can perform to restart my application with a previously functioning version.
UPDATE: Here's what seems to have worked:
Delete CloudFormation stack
Rebuild Elastic Beanstalk environment
Repair A records where necessary to use new EC2 instance IP
Why any of this was necessary — and why nothing less drastic touched the problem — remains a mystery.
Related
I'm trying to deploy a new version of my Python/Django application using CodePipeline.
I getting error while my code doploy using CodePipeline in AWS elastickbeanstlak. error is below.
Deployment completed, but with errors: During an aborted deployment, some instances may have deployed the new application version. To ensure all instances are running the same version, re-deploy the appropriate application version. Failed to deploy application. Unsuccessful command execution on instance id(s) 'i-0a0aff674bd64310b'. Aborting the operation. [Instance: i-0a0aff674bd64310b] Command failed on instance. Return code: 1 Output: Engine execution has encountered an error.. Instance deployment failed. For details, see 'eb-engine.log'.
The same says in the health status at AWS Console. Elastic Beanstalk marks it as Degraded.
I reboot instance, rebulid elastic beanstalk application.
How can i solve this issue. please help me.
I can't deploy a new version on Elastic Beanstalk.
Everything was working fine until I tried to deploy a new version where I have lots of issues (It is not the first time I deploy a new version on this environment, I already have deployed dozens). When I manage to fixe all of them I got those errors:
Failed to deploy application.
During an aborted deployment, some instances may have deployed the new application version. To ensure all instances are running the same version, re-deploy the appropriate application version.
Unsuccessful command execution on instance id(s) 'i-...'. Aborting the operation
I redeploy the version which does not work.
Here is the Elastic Beanstalk console:
Elastic Beanstalk console
Elastic Beanstalk events
The request logs button from Elastic Beanstalk return nothing.
The system log from EC2 instance shows the last working version logs.
I enable the CloudWatch logs from Configuration navigation pane. It added 4 files to CloudWatch logs:
/var/log/eb-activity.log -> empty so far
/var/log/httpd/access_log -> empty so far
/var/log/httpd/error_log -> empty so far
/environment-health.log -> Command is executing on all instances (56 minutes or more elapsed).", "Incorrect application version found on all instances. Expected version \"prod-v1.7.28-0\" (deployment 128).
It is an Amazon Linux, t2.medium instance with Apache as web server
What I already try:
Change the name of .zip each time to be different of other zip already deploy
Terminate the instance and the loadBalancer automatically create a new one
Reboot the instance
Rebuild Elastic Beanstalk environment
Deploy a simplest code
I tried to deploy just a zip with the code below but I got same errors.
<html>
<head>
<title>This is the title of the webpage!</title>
</head>
<body>
<p>This is an example paragraph. Anything in the <strong>body</strong> tag will appear on the page, just like this <strong>p</strong> tag and its contents.</p>
</body>
</html>
It always go back to last working version and when I tried to deploy the new version it does not work.
On some post I see some people telling it is maybe because the instance is too small but before it was working perfectly and the size does not change since then.
If you have some questions or ideas I will be very thankful.
Have a nice day !
Answer:
The issue was in the logs like you said. I had to ssh into my EC2 instance to reached them. The error was in the file cfn-init-cmd.log.
One of the command was waiting for an input so it timed out with no error message.
You should check the logs of the EBS for any hints as to what goes wrong with your deployment. The AWS console
can be helpful for that.
There are also the logs that can be acquired from EC2:
CloudWatch logs is another thing to check.
You should also check the autoscaling group, and see if there are any health checks there. What kind of checks are these? What's the grace period?
Here's a list of reasons that an EC2 health check could fail.
You could launch a better ec2 instance for troubleshooting.
Instance status checks.
The following are examples of problems that can cause instance status checks to fail:
Failed system status checks
Incorrect networking or startup configuration
Exhausted memory
Corrupted file system
Incompatible kernel
Also rebuilding is really a drastic step as it destroys and rebuilds all your resources. Your ELB DNS for example will be gone, any associated EIP will be released. These things can't be reclaimed.
I also faced same issue and deleted the wrong application versions. And increased the command timeout.
Default max deployment time -Command timeout- is 600 (10 minutes)
Go to Your Environment → Configuration → Deployment preferences → Command timeout
Increase the Deployment preferences higher like 1800 and then try to deploy the previous working application version. It will work.
Yesterday I attempted to follow this answer from stackoverflow to add multiple users to the authorized_keys in our Elastic Beanstalk environment for QA.
https://stackoverflow.com/a/46269015/1827986
However after doing so the deployment failed with the following error
2018-10-16 19:04:22 INFO Environment update is starting.
2018-10-16 19:05:05 INFO Deploying new version to instance(s).
2018-10-16 19:06:09 ERROR [Instance: i-05cc43b96ffc69145] Command
failed on instance. Return code: 1 Output: (TRUNCATED)...erform:
iam:GetGroup on resource: group BeanstalkAccess
declare -a users_array='()'
chmod: cannot access ‘/home/ec2-user/.ssh/authorized_keys’: No such
file or directory
chown: cannot access ‘/home/ec2-user/.ssh/authorized_keys’: No such
file or directory.
Hook /opt/elasticbeanstalk/hooks/appdeploy/post/980_beanstalk_ssh.sh
failed. For more detail, check /var/log/eb-activity.log using console
or EB CLI.
2018-10-16 19:06:09 INFO Command execution completed on all
instances. Summary: [Successful: 0, Failed: 1].
2018-10-16 19:06:10 ERROR Unsuccessful command execution on
instance id(s) 'i-05cc43b96ffc69145'. Aborting the operation.
2018-10-16 19:06:10 ERROR Failed to deploy application.
I then tried to deploy a previous application version that was known to work and that failed. I attempted to rebuild the environment so that I could deploy a working application version. However, it continues to try to deploy the same stuck version that is giving the errors.
Also now after the rebuild I am getting a bunch of errors about the instance not being reachable by the ELB.
100.0 % of the requests are erroring with HTTP 4xx. Insufficient request
rate (12.0 requests/min) to determine application health.
Command failed on all instances.
ELB health is failing or not available for all instances.
When I go into EC2 it shows the instance as running and it has green health.
I just tried to go to the Applications Versions and deleted the version that would not deploy hoping that it would restore from the working version. However, now it is just giving the following error when I try to deploy the working version.
Environment health has transitioned from Degraded to Severe. Command
failed on all instances. Incorrect application version found on all
instances. Expected version "app-v1_5_13-719-gc533-181016_092351"
(deployment 291). Application update failed 2 minutes ago and took 2
minutes. ELB health is failing or not available for all instances.
What else can I do to get this back to a working state?
This turned out to be a memory issue since we were running on the micro servers, I am not sure why we hit the limits all of the sudden, however, upgrading to a small instance solved the issues.
I'm in the process of migrating a Django app from Heroku to Elastic Beanstalk. Deploys were working and I was debugging issues to try to get the requirements.txt working then deploys stopped.
Now they will time out. The worst part of this is I'm unable to access logs, so I have no idea what's causing the issue.
Image of events:
When I try to access logs it either returns the error An error occurred retrieving logs: Rate exceeded or it will trigger the event requestEnvironmentInfo is starting. which itself will time out.
I'm not sure how you can move forward in debugging without access to the logs. I've cloned into a new environment and terminated the old one, but that didn't work. If you've encountered similar issues or know how to proceed please provide help!
I fixed the issue by terminating the main EC2 instance and redeploying in EB. in a larger instance. In this case small.
I try to deploy a zip on EBS. There is one EC2 and the environment health is green and seems to be fine. I try to deploy a zip and I get:
ERROR During an aborted deployment, some instances may have deployed the new application version. To ensure all instances are running the same version, re-deploy the appropriate application version.
ERROR Failed to deploy application.
ERROR Unsuccessful command execution on instance id(s) 'xxx'. Aborting the operation.
I deploy 10 zip's but all the same issue. I even tried to redeploy the zip which is also deployed at the moment (tried several times) but even that is not possible. What could be the issue?
Some configuration:
Environment type: Load balanced, auto scaling
Number instances: 1 - 1 (tried 1-2, and 0-2)
Instance type: t2.medium
I can only deploy succesfully if I terminate the ec2's manually and start a new deployment before the ec2's are scaled up again.
I was also facing the same issue. Couldn't deploy old builds or even the current one. It suddenly started working after a while, seems like an temporary issue with AWS.