Elastic Beanstalk application failing to deploy - existing application seemingly broken - amazon-web-services

We've been using EB for about a year, our main backend guy is not working with us at the moment. I've been deploying without any issues for the last 2 months, and today when I've tried to deploy I'm seeing this:
INFO: Environment update is starting. INFO: Deploying new version to
instance(s). ERROR: [Instance: i-dc773612,i-4073a981] Command failed
on instance. Return code: 1 Output:
[CMD-AppDeploy/AppDeployStage0/InfraWriteApp2] command failed with
error code 1: Error occurred during build: Command 01downloadVersion
failed. INFO: Command execution completed on all instances. Summary:
[Successful: 0, Failed: 2]. ERROR: [Instance: i-dc773612] Command
failed on instance. Return code: 1 Output:
[CMD-AppDeploy/AppDeployStage1/AppDeployEnactHook/01_flip.sh] command
failed with error code 1:
/opt/elasticbeanstalk/hooks/appdeploy/enact/01_flip.sh
++ /opt/elasticbeanstalk/bin/get-config container -k app_staging_dir
+ EB_APP_STAGING_DIR=/var/app/ondeck
++ /opt/elasticbeanstalk/bin/get-config container -k app_deploy_dir
+ EB_APP_DEPLOY_DIR=/var/app/current
+ '[' -d /var/app/current ']'
+ mv /var/app/current /var/app/current.old
+ mv /var/app/ondeck /var/app/current mv: cannot stat ‘/var/app/ondeck’: No such file or directory. INFO: Command execution
completed on 1 of 2 instances in environment. ERROR: [Instance:
i-4073a981] Command failed on instance. Return code: 1 Output:
[CMD-AppDeploy/AppDeployStage1/AppDeployEnactHook/01_flip.sh] command
failed with error code 1:
/opt/elasticbeanstalk/hooks/appdeploy/enact/01_flip.sh
++ /opt/elasticbeanstalk/bin/get-config container -k app_staging_dir
+ EB_APP_STAGING_DIR=/var/app/ondeck
++ /opt/elasticbeanstalk/bin/get-config container -k app_deploy_dir
+ EB_APP_DEPLOY_DIR=/var/app/current
+ '[' -d /var/app/current ']'
+ mv /var/app/current /var/app/current.old
+ mv /var/app/ondeck /var/app/current mv: cannot stat ‘/var/app/ondeck’: No such file or directory. INFO: Command execution
completed on 2 of 2 instances in environment. INFO: Command execution
completed on all instances. Summary: [Successful: 0, Failed: 2]. INFO:
New application version was deployed to running EC2 instances. ERROR:
Update environment operation is complete, but with errors. For more
information, see troubleshooting documentation. ERROR: Update
environment operation is complete, but with errors. For more
information, see troubleshooting documentation.
I have no idea why, as far as I can tell I've committed/deployed everything as I normally would.
I see the latest commit when I go into 'Application Versions' - and it would appear that is is deployed (it has the environment name next to the commit in the Deployed To column). However my mobile applications are no longer able to connect to the PHP files on the server, they keep getting 403 forbidden - even on files which were not changed in the last commit. It's as if the application is not there since the deploy failed.
Where should I go from here?
Edit: I've investigated a bit further, and the upload seemed to be successful as I can see the zip file is there. It's the deploy part itself which is failing. I've tried rebooting instances, as well as stopping instances and letting the load balancer create new ones. Still everything fails, although now I get a timeout. From start->timeout is taking less time that the timeout limit of 30 mins, the timeout happens in 5-10 mins.

Related

why eb deploy failed on first deploy?

I have created 2 NodeJs environments. It works fine.
Later created a new environment and eb deploy env_name give me the following error:
Creating application version archive "app-ceb7-200713_223016".
Uploading appname/app-ceb7-xxxxx.zip to S3. This may take a while.
Upload Complete.
2020-07-13 17:00:23 INFO Environment update is starting.
2020-07-13 17:00:44 INFO Deploying new version to instance(s).
2020-07-13 17:00:50 ERROR [Instance: i-066f599db7ee34] Command failed on instance.
An unexpected error has occurred [ErrorCode: 0000000001].
2020-07-13 17:00:50 INFO Command execution completed on all instances. Summary:
[Successful: 0, Failed: 1].
2020-07-13 17:00:50 ERROR Unsuccessful command execution on instance id(s) 'i-
06682f599db76ee34'. Aborting the operation.
2020-07-13 17:00:50 ERROR Failed to deploy application.
ERROR: ServiceError - Failed to deploy application.
Why this happen? Thanks in advance
Based on the comments.
To solve the issue, EB logs were inspected to identify the error message:
[app-deploy] - [Install customer specified node.js version]. Stop running the command. Error: unsupported node version >=6, please specify any of node versions in [v12.0.0 v12.1.0 v12.10.0 v12.11.0 v12.11.1 v12.12.0 v12.13.0 v12.13.1 v12.14.0 v12.14.1 v12.15.0 v12.16.0 v12.16.1 v12.16.2 v12.16.3 v12.17.0 v12.18.0 v12.18.1 v12.2.0 v12.3.0 v12.3.1 v12.4.0 v12.5.0 v12.6.0 v12.7.0 v12.8.0 v12.8.1 v12.9.0 v12.9.1]
Then, the correct version of the nodejs was specified in package.json:
"engines": { "node": "12.13.0"},

command terminated with exit code 1 when executing aws s3 cp in Jenkins Declarative Pipeline

I'm trying to upload my files to AWS S3 Bucket from my Kubernetes Pod thru Jenkins and got command terminated with exit code 1. Succeeding commands are terminated.
I've tried to manually run my command on the pod, and it works perfectly fine.
Here's my code on jenkins declarative pipeline:
sh (script: "kubectl exec -i ${MONGODB_POD_NAME} -n ${ENVIRONMENT} -- bash -c 'aws s3 sync /dump/${AUTOMATION_DUMP_DIRECTORY} s3://${S3_BUCKET}/${ENVIRONMENT}/${AUTOMATION_DUMP_DIRECTORY} --debug'")
Here's the log:
2019-04-29 09:55:23,373 - Thread-1 - awscli.customizations.s3.results - DEBUG - Shutdown request received in result processing thread,shutting down result thread.command terminated with exit code 1

AWS Elastic Beanstalk Issues Failed to deploy on all servers

Yesterday and today i tried to deploy on aws eb but I always get this error:
INFO: Environment update is starting.
INFO: Deploying new version to instance(s).
INFO: Environment health has transitioned from Ok to Info. Command is executing on all instances.
WARN: Environment health has transitioned from Info to Severe. None of the instances are sending data.
ERROR: Failed to run npm install. Snapshot logs for more details.
ERROR: [Instance: i-0632a672238b6ee0b] Command failed on instance. Return code: 1 Output: (TRUNCATED).../opt/elasticbeanstalk/containerfiles/ebnode.py", line 166, in npm_install
raise e
subprocess.CalledProcessError: Command '['/opt/elasticbeanstalk/node-install/node-v4.2.3-linux-x64/bin/npm', '--production', 'install']' returned non-zero exit status -9.
Hook /opt/elasticbeanstalk/hooks/appdeploy/pre/50npm.sh failed. For more detail, check /var/log/eb-activity.log using console or EB CLI.
INFO: Command execution completed on all instances. Summary: [Successful: 0, Failed: 1].
ERROR: Unsuccessful command execution on instance id(s) 'i-0632a672238b6ee0b'. Aborting the operation.
ERROR: Failed to deploy application.
ERROR: Failed to deploy application.
The built was successfull yesterday and i tried to push the old version and on another instance i just tried to deploy again. every instance is now in status severe or degraded.
Is this a aws issues, did i do something wrong? I did all the steps i do everyday and until now i never had any issues. Even after i uploaded the old applicationzip which was okay nothing changed.
I am having exactly the same issue. Build started to fail during the last weekend. Due to logs, the problem seems to be that while running npm install, the process runs out of memory. It doesn't matter whether I deploy directly or use a Docker container.
I am using a free tier micro instance with 1 GB of memory. I just can't understand why it worked before and now it doesn't work even if I deploy the same version that was working before...

AWS Elastic Beanstalk not deploying code fixes

I'm using aws EB to deploy my django app. I use the AWSEBCLI.
Everything works fine but if for some reason I mistakenly deploy broken code the ebcli will never deploy a new corrected version.
I'll have to SSH into the EC2 instance and fix it manually. Why? is there a way to force new code?
For example, I do some edits and I run git add . then git commit -m 'xx' then eb deploy
the result is:
TabError: inconsistent use of tabs and spaces in indentation.
container_command 01_migrate in .ebextensions/02_python.config failed.
For more detail, check /var/log/eb-activity.log using console or EB CLI.
INFO: Command execution completed on all instances. Summary: [Successful: 0, Failed: 1].
ERROR: Unsuccessful command execution on instance id(s) 'i-xxx'. Aborting the operation.
ERROR: Failed to deploy application.
At this point even if I edit the code, fix the indentation make the new git adds & commits eb will always show me this error until I SSH into the instance and I fix it manually.
Only then it will start to accept new commits. I really don't understand why and is rather annoying. Any ideas on how to skip the whole SSH part?

chef zero via AWS userdata fails

When running chef zero via AWS userdata, the run always fails. However, if I ssh onto the machine and manually execute the same commands, it works as expected. This is the output that I get:
Chef: 11.12.8
[2014-06-11T12:40:34+00:00] INFO: Auto-discovered chef repository at /opt/chef-zero
[2014-06-11T12:40:34+00:00] INFO: Starting chef-zero on port 8889 with repository at repository at /opt/chef-zero
One version per cookbook
[2014-06-11T12:40:34+00:00] INFO: Forking chef instance to converge...
[2014-06-11T12:40:35+00:00] DEBUG: Fork successful. Waiting for new chef pid: 1530
[2014-06-11T12:40:35+00:00] DEBUG: Forked instance now converging
[2014-06-11T12:40:35+00:00] ERROR: undefined method `[]' for nil:NilClass
[2014-06-11T12:40:35+00:00] FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)
The userdata that I set when launching the EC2 instance in AWS includes the following:
curl -L https://www.opscode.com/chef/install.sh | bash
mkdir /opt/chef-zero
cd /opt/chef-zero
wget http://myserver/chef-repo.tar.gz
tar zxf chef-repo
INSTANCE_ID=`curl http://169.254.169.254/latest/meta-data/instance-id`
cat <<EOF > /opt/chef-zero/solo.rb
ssl_verify_mode :verify_peer
node_name "$INSTANCE_ID"
EOF
/opt/chef/bin/chef-client -v >chef-zero.log 2>&1
/opt/chef/bin/chef-client -z -l debug -c solo.rb -o 'role[someRole]' -E BUILD >> chef-zero.log 2>&1
The AMI that I'm using is a custom one that was initially provisioned using knife + knife-ec2 (that bootstrapped chef 11.6.0 from an ubuntu 13.04 public ami). The omnibus installer from userdata (curl ... | bash) is upgrading chef to 11.12.8. The original knife run included chef-client::service in it's run, and the host is initially configured for use with chef-client + chef-server (i.e. there's a "validation.pem" and "client.rb" in /etc/chef - not sure if that makes a difference).
I am able to log onto the machine and execute chef-client -z -c solo.rb -o 'role[someRole]' -E BUILD as soon as the machine comes up (after waiting for files to be retrieved and the user-data chef-client to fail) and the chef run executes normally.
I have no idea why the userdata chef-client run fails with undefined method, any ideas what's causing it?
After some further investigation, and thanks to bit of chatting with the #chef guys on freenode, the problem was narrowed down to the environment.
When executing the script with userdata, the "HOME" variable is not set. shell.rb from the chef gem is littered with references to ENV["HOME"].
SSH:
# unset HOME
# chef-client -z -o 'role[test]'
ERROR: undefined method `[]' for nil:NilClass
# export HOME=/root
# chef-client -z -o 'role[test]'
Starting Chef Client, version ....
...
Chef Client finished, ...
If you need to execute chef-client via user data, you should manually export HOME before trying to execute chef.
Bug has been reported at https://tickets.opscode.com/browse/CHEF-5365
edit
Submitted a pull request which has since been merged into master. https://github.com/opscode/chef/pull/1494
This likely has nothing to do with chef-zero but indicates a problem in your recipe code (whatever's inside that chef-repo.tar.gz, or is driven by role[someRole]). It indicates an attempt to access a sub-element of a hash like
node['foo']['bar']
but when node['foo'] is nil (undefined)
Check the stacktrace that's generated by the chef client run to narrow it down.