I have a play application written in Scala that I deploy using elastic beanstalk. Up until now this has worked fine, but a few days ago new deployments started failing. The error message in eb-activity.log that I get is:
[2020-11-25T20:54:29.150Z] INFO [3127] - [Application deployment givinga-1.8.1-20201125b#152/AddonsBefore] : Starting activity...
[2020-11-25T20:54:29.150Z] INFO [3127] - [Application deployment givinga-1.8.1-20201125b#152/AddonsBefore/ConfigCWLAgent] : Starting activity...
[2020-11-25T20:54:29.150Z] INFO [3127] - [Application deployment givinga-1.8.1-20201125b#152/AddonsBefore/ConfigCWLAgent/10-config.sh] : Starting activity...
[2020-11-25T20:54:58.963Z] INFO [3127] - [Application deployment givinga-1.8.1-20201125b#152/AddonsBefore/ConfigCWLAgent/10-config.sh] : Activity execution failed, because: (ElasticBeanstalk::ExternalInvocationError)
caused by: (Executor::NonZeroExitStatus)
[2020-11-25T20:54:58.964Z] INFO [3127] - [Application deployment givinga-1.8.1-20201125b#152/AddonsBefore/ConfigCWLAgent/10-config.sh] : Activity failed.
[2020-11-25T20:54:58.964Z] INFO [3127] - [Application deployment givinga-1.8.1-20201125b#152/AddonsBefore/ConfigCWLAgent] : Activity failed.
Deploying to other test environments works, here is the relevant log line when it works:
[2020-11-25T23:19:51.549Z] INFO [3058] - [Application deployment givinga-1.8.1-20201126a#482/AddonsBefore/ConfigCWLAgent] : Starting activity...
[2020-11-25T23:19:51.549Z] INFO [3058] - [Application deployment givinga-1.8.1-20201126a#482/AddonsBefore/ConfigCWLAgent/10-config.sh] : Starting activity...
[2020-11-25T23:19:53.910Z] INFO [3058] - [Application deployment givinga-1.8.1-20201126a#482/AddonsBefore/ConfigCWLAgent/10-config.sh] : Completed activity. Result:
Starting awslogs: [ OK ]
Enabled log streaming.
[2020-11-25T23:19:53.910Z] INFO [3058] - [Application deployment givinga-1.8.1-20201126a#482/AddonsBefore/ConfigCWLAgent] : Completed activity. Result:
Successfully execute hooks in directory /opt/elasticbeanstalk/addons/logstreaming/hooks/config.
So my question is, what is the log streaming doing here? What could cause it to fail? There doesn't seem to be a way for me to delete this addon, or even to configure it.
Which AWS region are you using for your EB environments?
If that deployment worked yesterday and you didn't make changes. It is probably that us-east-1 had a failure today.
https://status.aws.amazon.com/
I am trying to run docker container for spring boot application however I am not able to access the application.
I am using docker on mac with the following version
Mac OS Mojave version
10.14.3 (18D109)
Docker version on mac:
anhs-MBP:spring-boot-with-fargate anhtrang$ docker --version
Docker version 18.09.2, build 6247962
I have checked the container logs and spring boot application started normally
Container logs:
019-08-12 01:08:42.885 INFO 6 --- [ main] o.s.j.e.a.AnnotationMBeanExporter : Registering beans for JMX exposure on startup
2019-08-12 01:08:43.026 INFO 6 --- [ main] s.b.c.e.t.TomcatEmbeddedServletContainer : Tomcat started on port(s): 8080 (http)
2019-08-12 01:08:43.045 INFO 6 --- [ main] c.e.s.SpringBootWithFargateApplication : Started SpringBootWithFargateApplication in 8.06 seconds (JVM running for 9.107)
Command to start docker container:
anhs-MBP:spring-boot-with-fargate anhtrang$ docker run -d -p 8080:8080 anhtrang/spring-boot-with-farget:0.0.1-SNAPSHOT
I also tried starting the spring boot application outside of container it's working perfectly. I just don't understand why its not able to expose the port
My GitHub repo:
https://github.com/anhtv08/spring-boot-with-fargate
Appreciate for any quick help
I've been trying to follow this thorough explanation on how to deploy a django app with celery worker to aws elastic beanstalk:
How to run a celery worker with Django app scalable by AWS Elastic Beanstalk?
I had some problems installing pycurl but solved it with the comment in:
Pip Requirements.txt --global-option causing installation errors with other packages. "option not recognized"
Then i got:
[2019-01-26T06:43:04.865Z] INFO [12249] - [Application update app-190126_134200#28/AppDeployStage0/EbExtensionPostBuild/Infra-EmbeddedPostBuild/postbuild_1_raiseflags/Command 05_celery_tasks_run] : Activity execution failed, because: /usr/bin/env: bash
: No such file or directory
(ElasticBeanstalk::ExternalInvocationError)
But also solved it: it turns out I had to convert "celery_configuration.txt" file to UNIX EOL (i'm using Windows, and Notepad++ automatically converted it to Windows EOL).
With all these modifications I can successfully deploy the project. But the problem is that the periodic tasks are not running.
I get:
2019-01-26 09:12:57,337 INFO exited: celeryd-beat (exit status 1; not expected)
2019-01-26 09:12:58,583 INFO spawned: 'celeryd-worker' with pid 25691
2019-01-26 09:12:59,453 INFO spawned: 'celeryd-beat' with pid 25695
2019-01-26 09:12:59,666 INFO exited: celeryd-worker (exit status 1; not expected)
2019-01-26 09:13:00,790 INFO spawned: 'celeryd-worker' with pid 25705
2019-01-26 09:13:00,791 INFO exited: celeryd-beat (exit status 1; not expected)
2019-01-26 09:13:01,915 INFO exited: celeryd-worker (exit status 1; not expected)
2019-01-26 09:13:03,919 INFO spawned: 'celeryd-worker' with pid 25728
2019-01-26 09:13:03,920 INFO spawned: 'celeryd-beat' with pid 25729
2019-01-26 09:13:05,985 INFO exited: celeryd-worker (exit status 1; not expected)
2019-01-26 09:13:06,091 INFO exited: celeryd-beat (exit status 1; not expected)
2019-01-26 09:13:07,092 INFO gave up: celeryd-beat entered FATAL state, too many start retries too quickly
2019-01-26 09:13:09,096 INFO spawned: 'celeryd-worker' with pid 25737
2019-01-26 09:13:10,084 INFO exited: celeryd-worker (exit status 1; not expected)
2019-01-26 09:13:11,085 INFO gave up: celeryd-worker entered FATAL state, too many start retries too quickly
I also have this part of the logs:
[2019-01-26T09:13:00.583Z] INFO [25247] - [Application update app-190126_161213#43/AppDeployStage1/AppDeployPostHook/run_supervised_celeryd.sh] : Completed activity. Result:
[program:celeryd-worker]
; Set full path to celery program if using virtualenv
command=/opt/python/run/venv/bin/celery worker -A raiseflags --loglevel=INFO
directory=/opt/python/current/app
user=nobody
numprocs=1
stdout_logfile=/var/log/celery-worker.log
stderr_logfile=/var/log/celery-worker.log
autostart=true
autorestart=true
startsecs=10
; Need to wait for currently executing tasks to finish at shutdown.
; Increase this if you have very long running tasks.
stopwaitsecs = 600
; When resorting to send SIGKILL to the program to terminate it
; send SIGKILL to its whole process group instead,
; taking care of its children as well.
killasgroup=true
; if rabbitmq is supervised, set its priority higher
; so it starts first
priority=998
environment=PYTHONPATH="/opt/python/current/app/:",PATH="/opt/python/run/venv/bin/:%%(ENV_PATH)s",RDS_PORT="5432",RDS_DB_NAME="ebdb",RDS_USERNAME="foobar",PYCURL_SSL_LIBRARY="nss",DJANGO_SETTINGS_MODULE="raiseflags.settings",RDS_PASSWORD="foobar",RDS_HOSTNAME="something.something.eu-west-1.rds.amazonaws.com"
[program:celeryd-beat]
; Set full path to celery program if using virtualenv
command=/opt/python/run/venv/bin/celery beat -A raiseflags --loglevel=INFO --workdir=/tmp -S django --pidfile /tmp/celerybeat.pid
directory=/opt/python/current/app
user=nobody
numprocs=1
stdout_logfile=/var/log/celery-beat.log
stderr_logfile=/var/log/celery-beat.log
autostart=true
autorestart=true
startsecs=10
; Need to wait for currently executing tasks to finish at shutdown.
; Increase this if you have very long running tasks.
stopwaitsecs = 600
; When resorting to send SIGKILL to the program to terminate it
; send SIGKILL to its whole process group instead,
; taking care of its children as well.
killasgroup=true
; if rabbitmq is supervised, set its priority higher
; so it starts first
priority=998
environment=PYTHONPATH="/opt/python/current/app/:",PATH="/opt/python/run/venv/bin/:%%(ENV_PATH)s",RDS_PORT="5432",RDS_DB_NAME="ebdb",RDS_USERNAME="puigdemontAWS",PYCURL_SSL_LIBRARY="nss",DJANGO_SETTINGS_MODULE="raiseflags.settings",RDS_PASSWORD="holahola",RDS_HOSTNAME="aa1m59206y4fljn.cdreg3t50bbl.eu-west-1.rds.amazonaws.com"
No config updates to processes
celeryd-beat: ERROR (not running)
celeryd-beat: ERROR (abnormal termination)
celeryd-worker: ERROR (not running)
celeryd-worker: ERROR (abnormal termination)
[2019-01-26T09:13:00.583Z] INFO [25247] - [Application update app-190126_161213#43/AppDeployStage1/AppDeployPostHook] : Completed activity. Result:
Successfully execute hooks in directory /opt/elasticbeanstalk/hooks/appdeploy/post.
[2019-01-26T09:13:00.583Z] INFO [25247] - [Application update app-190126_161213#43/AppDeployStage1] : Completed activity. Result:
Application version switch - Command CMD-AppDeploy stage 1 completed
[2019-01-26T09:13:00.583Z] INFO [25247] - [Application update app-190126_161213#43/AddonsAfter] : Starting activity...
[2019-01-26T09:13:00.583Z] INFO [25247] - [Application update app-190126_161213#43/AddonsAfter/ConfigLogRotation] : Starting activity...
[2019-01-26T09:13:00.583Z] INFO [25247] - [Application update app-190126_161213#43/AddonsAfter/ConfigLogRotation/10-config.sh] : Starting activity...
[2019-01-26T09:13:00.756Z] INFO [25247] - [Application update app-190126_161213#43/AddonsAfter/ConfigLogRotation/10-config.sh] : Completed activity. Result:
Disabled forced hourly log rotation.
[2019-01-26T09:13:00.756Z] INFO [25247] - [Application update app-190126_161213#43/AddonsAfter/ConfigLogRotation] : Completed activity. Result:
Successfully execute hooks in directory /opt/elasticbeanstalk/addons/logpublish/hooks/config.
I don't know if it has something to do with the error, but notice above the line [[ PATH="/opt/python/run/venv/bin/:%%(ENV_PATH)s" ]] --> shouldn't ENV_PATH be something else?:
environment=PYTHONPATH="/opt/python/current/app/:",PATH="/opt/python/run/venv/bin/:%%(ENV_PATH)s",RDS_PORT="5432",RDS_DB_NAME="ebdb",RDS_USERNAME="foobar",PYCURL_SSL_LIBRARY="nss",DJANGO_SETTINGS_MODULE="raiseflags.settings",RDS_PASSWORD="foobar",RDS_HOSTNAME="something.something.eu-west-1.rds.amazonaws.com"
I'ts my first time deploying an app with celery, and i'm really lost to be honest. I fought a lot to solve the first two errors (i'm really amateur), and now that i get this I don't even know where to start.
Also, i'm not sure if I'm using "celery_configuration.txt" the right way. The only thing I edited was the 2 places where it says "django_app", which I changed for "raiseflags" (the name of my django project). Is this correct?
Does anyone know how to solve it? I can paste my files if needed, but they are just like the ones provided in the first link. I'm using Windows.
Thank you very much!
Ok, the problem had nothing to do with the PATH line I was referring to. I just had to add 'django_celery_beat' and 'django_celery_results' in INSTALLED_APPS in my settings.py
The connection error I later referred to talking to Fran was because I needed to set BROKER_URL instead of CELERY_BROKER_URL, also in the settings.py file. I guess this had to do with me not specifying 'CELERY' as the namespace in the app.autodiscover_tasks() in celery.py file (although in the linked question they do it, i didn't do it because i was using a different version of celery).
Thanks to Fran for everything, specially for pointing out that i should review the celery error logs. I didn't know how to do it. If any other amateur is also struggling, know that you have to "eb ssh" to your instance and then "tail -n 40 /var/log/celery-worker.log" and ""tail -n 40 /var/log/celery-beat.log" (where "40" is the number of lines you want to read). I know this sounds obvious to a lot of people but, stupid me, I had no clue.
(btw, i'm still struggling with a problem with the celery worker, that can't find pycurl module, but this has nothing to do with this question).
Referring to the line you pointed out where appears
environment=PYTHONPATH="/opt/python/current/app/:",PATH="/opt/python/run/venv/bin/:%%(ENV_PATH)s",RDS_PORT="5432",RDS_DB_NAME="ebdb",RDS_USERNAME="foobar",PYCURL_SSL_LIBRARY="nss",DJANGO_SETTINGS_MODULE="raiseflags.settings",RDS_PASSWORD="foobar",RDS_HOSTNAME="something.something.eu-west-1.rds.amazonaws.com", do you copy this line from somewhere? Because I don't see it in the link you posted.
In the linked answer was environment=$celeryenv, where $celeryenv was defined as
celeryenv=`cat /opt/python/current/env | tr '\n' ',' | sed 's/export //g' | sed 's/$PATH/%(ENV_PATH)s/g' | sed 's/$PYTHONPATH//g' | sed 's/$LD_LIBRARY_PATH//g' | sed 's/%/%%/g'`
celeryenv=${celeryenv%?}```
I've been struggling AWS-Elastic Beanstalk problems.
The other day, I appended some gems into Gemfile and then used eb deploy.
(Maybe the gem is whenever? or bcrypt? Sorry not sure)
It didn't work correctly on my deployment. The results are below.
ERROR: [Instance: i-452520da] Command failed on instance. Return code: 10 Output: /opt/elasticbeanstalk/hooks/appdeploy/post/10_reload_cron.sh: line 3: cd: HOME not set
Could not locate Gemfile or .bundle/ directory.
Hook /opt/elasticbeanstalk/hooks/appdeploy/post/10_reload_cron.sh failed. For more detail, check /var/log/eb-activity.log using console or EB CLI.
INFO: Command execution completed on all instances. Summary: [Successful: 0, Failed: 1].
ERROR: Unsuccessful command execution on instance id(s) 'i-452520da'. Aborting the operation.
ERROR: Failed to deploy application.
Here's eb-activity.log.
[2016-08-28T01:51:16.844Z] INFO [20749] - [Application update app-ed0b6-160828_104745#147/AppDeployStage1/AppDeployEnactHook] : Completed activity. Result:
Successfully execute hooks in directory /opt/elasticbeanstalk/hooks/appdeploy/enact.
[2016-08-28T01:51:16.844Z] INFO [20749] - [Application update app-ed0b6-160828_104745#147/AppDeployStage1/AppDeployPostHook] : Starting activity...
[2016-08-28T01:51:16.844Z] INFO [20749] - [Application update app-ed0b6-160828_104745#147/AppDeployStage1/AppDeployPostHook/01_create_pids.sh] : Starting activity...
[2016-08-28T01:51:17.044Z] INFO [20749] - [Application update app-ed0b6-160828_104745#147/AppDeployStage1/AppDeployPostHook/01_create_pids.sh] : Completed activity.
[2016-08-28T01:51:17.044Z] INFO [20749] - [Application update app-ed0b6-160828_104745#147/AppDeployStage1/AppDeployPostHook/10_reload_cron.sh] : Starting activity...
[2016-08-28T01:51:17.242Z] INFO [20749] - [Application update app-ed0b6-160828_104745#147/AppDeployStage1/AppDeployPostHook/10_reload_cron.sh] : Activity execution failed, because: /opt/elasticbeanstalk/hooks/appdeploy/post/10_reload_cron.sh: line 3: cd: HOME not set
Could not locate Gemfile or .bundle/ directory (ElasticBeanstalk::ExternalInvocationError)
caused by: /opt/elasticbeanstalk/hooks/appdeploy/post/10_reload_cron.sh: line 3: cd: HOME not set
Could not locate Gemfile or .bundle/ directory (Executor::NonZeroExitStatus)
[2016-08-28T01:51:17.242Z] INFO [20749] - [Application update app-ed0b6-160828_104745#147/AppDeployStage1/AppDeployPostHook/10_reload_cron.sh] : Activity failed.
[2016-08-28T01:51:17.243Z] INFO [20749] - [Application update app-ed0b6-160828_104745#147/AppDeployStage1/AppDeployPostHook] : Activity failed.
[2016-08-28T01:51:17.243Z] INFO [20749] - [Application update app-ed0b6-160828_104745#147/AppDeployStage1] : Activity failed.
[2016-08-28T01:51:17.243Z] INFO [20749] - [Application update app-ed0b6-160828_104745#147] : Completed activity. Result:
Application update - Command CMD-AppDeploy failed
[2016-08-28T01:52:38.630Z] INFO [2486] - [CMD-TailLogs] : Starting activity...
[2016-08-28T01:52:38.630Z] INFO [2486] - [CMD-TailLogs/AddonsBefore] : Starting activity...
How could I eliminate this problem? Thanks in advance.
I can work out this problem.
The thing is "/opt/elasticbeanstalk/hooks/appdeploy/post/10_reload_cron.sh".
So I deleted this file by using .ebextensions/some.sh
I'm trying to install GhostScript 9.10 on Elastic Beanstalk because currently only Ghostscript 8.70 is available via yum packages.
The installation is working via SSH on the EC2 instance but the configuration file is always failing and I don't understand whats the reason.
Here is my .ebextensions configuration file:
commands:
01_admin_rights:
command: "sudo su"
02_get_gs:
command: "curl -O http://downloads.ghostscript.com/public/old-gs-releases/ghostscript-9.10.tar.gz"
03_extract_gs:
command: "tar -xzf ghostscript-9.10.tar.gz"
04_cd_gs:
command: "cd ghostscript-9.10"
05_configure_gs:
command: "bash configure"
06_install_gs:
command: "make install"
07_so_gs:
command: "make so"
08_reboot:
command: "reboot"
And here goes the elastic beanstalk error log part:
[2016-06-21T12:22:52.720Z] INFO [24703] - [Application update Come on #15#25/AppDeployStage0/EbExtensionPreBuild/Infra-EmbeddedPreBuild/prebuild_2__Staging/Command 01_admin_rights] : Starting activity...
[2016-06-21T12:22:52.757Z] INFO [24703] - [Application update Come on #15#25/AppDeployStage0/EbExtensionPreBuild/Infra-EmbeddedPreBuild/prebuild_2__Staging/Command 01_admin_rights] : Completed activity.
[2016-06-21T12:22:52.757Z] INFO [24703] - [Application update Come on #15#25/AppDeployStage0/EbExtensionPreBuild/Infra-EmbeddedPreBuild/prebuild_2__Staging/Command 02_get_gs] : Starting activity...
[2016-06-21T12:22:53.524Z] INFO [24703] - [Application update Come on #15#25/AppDeployStage0/EbExtensionPreBuild/Infra-EmbeddedPreBuild/prebuild_2__Staging/Command 02_get_gs] : Completed activity. Result:
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
100 33.6M 100 33.6M 0 0 49.2M 0 --:--:-- --:--:-- --:--:-- 49.2M
[2016-06-21T12:22:53.524Z] INFO [24703] - [Application update Come on #15#25/AppDeployStage0/EbExtensionPreBuild/Infra-EmbeddedPreBuild/prebuild_2__Staging/Command 03_extract_gs] : Starting activity...
[2016-06-21T12:22:55.066Z] INFO [24703] - [Application update Come on #15#25/AppDeployStage0/EbExtensionPreBuild/Infra-EmbeddedPreBuild/prebuild_2__Staging/Command 03_extract_gs] : Completed activity.
[2016-06-21T12:22:55.066Z] INFO [24703] - [Application update Come on #15#25/AppDeployStage0/EbExtensionPreBuild/Infra-EmbeddedPreBuild/prebuild_2__Staging/Command 04_cd_gs] : Starting activity...
[2016-06-21T12:22:55.069Z] INFO [24703] - [Application update Come on #15#25/AppDeployStage0/EbExtensionPreBuild/Infra-EmbeddedPreBuild/prebuild_2__Staging/Command 04_cd_gs] : Completed activity.
[2016-06-21T12:22:55.070Z] INFO [24703] - [Application update Come on #15#25/AppDeployStage0/EbExtensionPreBuild/Infra-EmbeddedPreBuild/prebuild_2__Staging/Command 05_configure_gs] : Starting activity...
[2016-06-21T12:22:55.073Z] INFO [24703] - [Application update Come on #15#25/AppDeployStage0/EbExtensionPreBuild/Infra-EmbeddedPreBuild/prebuild_2__Staging/Command 05_configure_gs] : Activity execution failed, because: bash: configure: No such file or directory
(ElasticBeanstalk::ExternalInvocationError)
[2016-06-21T12:22:55.073Z] INFO [24703] - [Application update Come on #15#25/AppDeployStage0/EbExtensionPreBuild/Infra-EmbeddedPreBuild/prebuild_2__Staging/Command 05_configure_gs] : Activity failed.
I understand that the command 5 is failing because the file doesn't exist. However when I'm doing the steps manually via SSH the file exists and all commands can be executed in this order.
What do I miss?
EDIT:
I played around with the configure argument and tried:
/bin/bash: ./configure
./configure
bash ./configure
All arguments fail with the same error "No such file or directory."
If I connect via SSH and enter one of the configure commands then it works without any issues.
Anybody knows whats going on here?
I think this could be because separate commands don't keep the environment or the current directory consistent between them. One of your commands is cd, but I don't think the changed directory persists to the next command. Try combining all commands into one, like this:
command: |
cd directory
./configure
Also, ebextensions run as root, so you may not need the sudo su in the beginning.