I am using AWS beanstalk to host one of my application.
My application saves local configuration files constantly into .property and .csv files.
Everything works fine.
Till one day I noticed everything has been rolled back to the initial state:
The application is working fine but all the configurations I entered into .property and the history I saved into .csv are disappeared.
I looked at the beanstalk instant history and noticed that AWS implemented an update to the instance as the following. I guess when AWS implements the system maintenance it simply redeploy the application with the last .zip file I uploaded for the application deployment.
I know I can create a database to persist all the data I entered, however, that will be a big change.
Another option (which I am using right now) is to simply turn off the Managed updates feature. But my instance might slowly be faded out due to constant updates and patches nowadays.
Any idea how can I keep the managed updates open but instead of using the last .zip file to deploy the application, use the current working directory (I assume it is /app directory) after the update?
At the following is what happened during AWS managed updates:
2022-08-12 23:15:46 UTC+1000
INFO
Environment update completed successfully.
2022-08-12 23:15:46 UTC+1000
INFO
Successfully deployed new configuration to environment.
2022-08-12 23:14:38 UTC+1000
INFO
Updating environment xxx-env's configuration settings.
2022-08-12 23:14:30 UTC+1000
INFO
Environment update is starting.
2022-08-12 07:13:20 UTC+1000
INFO
Environment health has transitioned from Info to Ok. Configuration update completed 83 seconds ago and took 11 minutes.
2022-08-12 07:12:37 UTC+1000
INFO
Environment update completed successfully.
2022-08-12 07:12:37 UTC+1000
INFO
Successfully deployed new configuration to environment.
2022-08-12 07:12:21 UTC+1000
INFO
Removed instance [i-025394c6b5dc] from your environment.
2022-08-12 07:09:09 UTC+1000
INFO
Deployment succeeded. Terminating old instances and temporary Auto Scaling group.
2022-08-12 07:07:50 UTC+1000
INFO
Updating environment xxx-env's configuration settings.
2022-08-12 07:07:20 UTC+1000
INFO
Waiting for post-deployment configuration to complete.
2022-08-12 07:07:16 UTC+1000
INFO
Instance deployment completed successfully.
2022-08-12 07:06:39 UTC+1000
INFO
Starting post-deployment configuration on new instances.
2022-08-12 07:06:11 UTC+1000
INFO
Attached new instance(s) to the permanent auto scaling group awseb-e-wiuxjqswqj-stack-AWSEBAutoScalingGroup-AV.
2022-08-12 07:06:07 UTC+1000
INFO
Detached new instance(s) from temporary auto scaling group awseb-e-wiuxjqswqj-immutable-stack-AWSEBAutoScalingGroup-1I2W.
2022-08-12 07:04:29 UTC+1000
INFO
Waiting for instance(s) (i-0463ad18900a) to pass health checks.
2022-08-12 07:03:21 UTC+1000
INFO
Added instance [i-0463ad18900a] to your environment.
2022-08-12 07:03:21 UTC+1000
INFO
Environment health has transitioned from Ok to Info. Configuration update in progress on 1 instance. 0 out of 1 instance completed (running for 2 minutes). All instances are in same availability zone (us-west-2c).
2022-08-12 07:00:52 UTC+1000
INFO
Created temporary auto scaling group awseb-e-wiuxjqswqj-immutable-stack-AWSEBAutoScalingGroup-1I2.
2022-08-12 07:00:21 UTC+1000
INFO
Immutable deployment policy enabled. Launching one instance with the new settings to verify health.
2022-08-12 07:00:13 UTC+1000
INFO
Environment update is starting.
2022-08-12 07:00:11 UTC+1000
INFO
Managed platform update is in-progress.
My application saves local configuration files constantly into .property and .csv files. Everything works fine.
Any deployment, auto-scaling event, managed update, or failed instance recovery may delete the files you are storing to the local Elastic Beanstalk disk. You could never write files to the local Elastic Beanstalk disk!
You should change your application to write data to a persistent database, or use an Elastic File System (EFS) volume to write your files to.
Related
I can't deploy a new version on Elastic Beanstalk.
Everything was working fine until I tried to deploy a new version where I have lots of issues (It is not the first time I deploy a new version on this environment, I already have deployed dozens). When I manage to fixe all of them I got those errors:
Failed to deploy application.
During an aborted deployment, some instances may have deployed the new application version. To ensure all instances are running the same version, re-deploy the appropriate application version.
Unsuccessful command execution on instance id(s) 'i-...'. Aborting the operation
I redeploy the version which does not work.
Here is the Elastic Beanstalk console:
Elastic Beanstalk console
Elastic Beanstalk events
The request logs button from Elastic Beanstalk return nothing.
The system log from EC2 instance shows the last working version logs.
I enable the CloudWatch logs from Configuration navigation pane. It added 4 files to CloudWatch logs:
/var/log/eb-activity.log -> empty so far
/var/log/httpd/access_log -> empty so far
/var/log/httpd/error_log -> empty so far
/environment-health.log -> Command is executing on all instances (56 minutes or more elapsed).", "Incorrect application version found on all instances. Expected version \"prod-v1.7.28-0\" (deployment 128).
It is an Amazon Linux, t2.medium instance with Apache as web server
What I already try:
Change the name of .zip each time to be different of other zip already deploy
Terminate the instance and the loadBalancer automatically create a new one
Reboot the instance
Rebuild Elastic Beanstalk environment
Deploy a simplest code
I tried to deploy just a zip with the code below but I got same errors.
<html>
<head>
<title>This is the title of the webpage!</title>
</head>
<body>
<p>This is an example paragraph. Anything in the <strong>body</strong> tag will appear on the page, just like this <strong>p</strong> tag and its contents.</p>
</body>
</html>
It always go back to last working version and when I tried to deploy the new version it does not work.
On some post I see some people telling it is maybe because the instance is too small but before it was working perfectly and the size does not change since then.
If you have some questions or ideas I will be very thankful.
Have a nice day !
Answer:
The issue was in the logs like you said. I had to ssh into my EC2 instance to reached them. The error was in the file cfn-init-cmd.log.
One of the command was waiting for an input so it timed out with no error message.
You should check the logs of the EBS for any hints as to what goes wrong with your deployment. The AWS console
can be helpful for that.
There are also the logs that can be acquired from EC2:
CloudWatch logs is another thing to check.
You should also check the autoscaling group, and see if there are any health checks there. What kind of checks are these? What's the grace period?
Here's a list of reasons that an EC2 health check could fail.
You could launch a better ec2 instance for troubleshooting.
Instance status checks.
The following are examples of problems that can cause instance status checks to fail:
Failed system status checks
Incorrect networking or startup configuration
Exhausted memory
Corrupted file system
Incompatible kernel
Also rebuilding is really a drastic step as it destroys and rebuilds all your resources. Your ELB DNS for example will be gone, any associated EIP will be released. These things can't be reclaimed.
I also faced same issue and deleted the wrong application versions. And increased the command timeout.
Default max deployment time -Command timeout- is 600 (10 minutes)
Go to Your Environment → Configuration → Deployment preferences → Command timeout
Increase the Deployment preferences higher like 1800 and then try to deploy the previous working application version. It will work.
Yesterday I attempted to follow this answer from stackoverflow to add multiple users to the authorized_keys in our Elastic Beanstalk environment for QA.
https://stackoverflow.com/a/46269015/1827986
However after doing so the deployment failed with the following error
2018-10-16 19:04:22 INFO Environment update is starting.
2018-10-16 19:05:05 INFO Deploying new version to instance(s).
2018-10-16 19:06:09 ERROR [Instance: i-05cc43b96ffc69145] Command
failed on instance. Return code: 1 Output: (TRUNCATED)...erform:
iam:GetGroup on resource: group BeanstalkAccess
declare -a users_array='()'
chmod: cannot access ‘/home/ec2-user/.ssh/authorized_keys’: No such
file or directory
chown: cannot access ‘/home/ec2-user/.ssh/authorized_keys’: No such
file or directory.
Hook /opt/elasticbeanstalk/hooks/appdeploy/post/980_beanstalk_ssh.sh
failed. For more detail, check /var/log/eb-activity.log using console
or EB CLI.
2018-10-16 19:06:09 INFO Command execution completed on all
instances. Summary: [Successful: 0, Failed: 1].
2018-10-16 19:06:10 ERROR Unsuccessful command execution on
instance id(s) 'i-05cc43b96ffc69145'. Aborting the operation.
2018-10-16 19:06:10 ERROR Failed to deploy application.
I then tried to deploy a previous application version that was known to work and that failed. I attempted to rebuild the environment so that I could deploy a working application version. However, it continues to try to deploy the same stuck version that is giving the errors.
Also now after the rebuild I am getting a bunch of errors about the instance not being reachable by the ELB.
100.0 % of the requests are erroring with HTTP 4xx. Insufficient request
rate (12.0 requests/min) to determine application health.
Command failed on all instances.
ELB health is failing or not available for all instances.
When I go into EC2 it shows the instance as running and it has green health.
I just tried to go to the Applications Versions and deleted the version that would not deploy hoping that it would restore from the working version. However, now it is just giving the following error when I try to deploy the working version.
Environment health has transitioned from Degraded to Severe. Command
failed on all instances. Incorrect application version found on all
instances. Expected version "app-v1_5_13-719-gc533-181016_092351"
(deployment 291). Application update failed 2 minutes ago and took 2
minutes. ELB health is failing or not available for all instances.
What else can I do to get this back to a working state?
This turned out to be a memory issue since we were running on the micro servers, I am not sure why we hit the limits all of the sudden, however, upgrading to a small instance solved the issues.
I'm using currently Elastic Beanstalk to deploy and run an executable jar (Spring Boot app). I'm using a t2.small ec2 instance (2Go) which is sufficient for my need and for the size of the application.
I'm encountring now several problems :
Deploying new version of the app is tooooo slow. I have to wait more than 10 minutes to changes take effects.
60% of the time, the deployment does not work. A rollback is done by EB which takes another 5 or 10 minutes.
Nothing is reported to log files. So I have no idea why deployment does not work.
Below example of errors I get.
2018-05-04 16:45:25 UTC+0200 INFO requestEnvironmentInfo is starting.
2018-05-04 16:45:09 UTC+0200 INFO Pulled logs for environment instances.
2018-05-04 16:45:09 UTC+0200 INFO Command execution completed on all instances. Summary: [Successful: 0, TimedOut: 1].
2018-05-04 16:45:09 UTC+0200 WARN The following instances have not responded in the allowed command timeout time (they might still finish eventually on their own): [i-04cb070a376fbf077].
2018-05-04 16:31:06 UTC+0200 INFO requestEnvironmentInfo is starting.
2018-05-04 16:12:10 UTC+0200 WARN Environment health has transitioned from Info to Degraded. Incorrect application version found on all instances. Expected version "deepan-core-1.0.0-SNAPSHOT" (deployment 10). Application update is aborting (running for 15 minutes).
2018-05-04 16:11:33 UTC+0200 ERROR During an aborted deployment, some instances may have deployed the new application version. To ensure all instances are running the same version, re-deploy the appropriate application version.
2018-05-04 16:11:33 UTC+0200 ERROR Failed to deploy application.
2018-05-04 16:11:33 UTC+0200 ERROR Unsuccessful command execution on instance id(s) 'i-04cb070a376fbf077'. Aborting the operation.
Is there any reliable alternatives I can use on AWS to deploy a Spring Boot executable jar ?
Thank you for your help !
My ElasticBeanstalk stack includes an RDS instance, with the application using the RDS_* environment variables to connect.
I need to reset the RDS password, which is easy enough to do, but does anyone know if the EC2 servers will require restarting or similar in order to inherit the change?
EB does have to update the ec2 instances, but I don't think it's a full restart. I'm guessing you're wondering how long it will take; out of curiosity I just did a quick test in our dev environment, here is the event log:
2018-03-14 09:22:30 UTC-0700 INFO Environment update completed successfully.
2018-03-14 09:22:30 UTC-0700 INFO Successfully deployed new configuration to environment.
2018-03-14 09:21:28 UTC-0700 INFO Updating environment webtier's configuration settings.
2018-03-14 09:21:15 UTC-0700 INFO Environment update is starting.
Looks like a little over a minute in total.
The Elastic Beanstalk environment is running on t1.micro & db.t1.micro, where we tried to update the rds to db.m1.small instance. This configuration change has been made on the Elastic Beanstalk and it is on updating state for around 4 hours, we thought AWS stuck somewhere internally. In this meantime, the AWS has not updated the Trace information. In order to retain the environment back (rollback of configuration update operation), we have aborted the environment configuration update. This has ended up on invalid state with the health turned to grey. This because AWS will not allow creating t1 environment nowadays, however, the rollback is trying to bring back the older t1 setup with an error--> RDS does not support creating a DB instance with the following combination: DBInstanceClass=db.t1.micro, Engine=mysql, EngineVersion=5.6.37, LicenseModel=general-public-license. For supported combinations of instance class and database engine version.
Any suggestion to bring back the environment to green would be really helpful.
Yes, with the source of the snapshot we could able to create a new EBS instance. The only problem we faced during the snapshot to clone an instance is the application level configuration file change not done. BTW there is no proper root cause for the AWS DB upgrade failure from AWS.