Aws Beanstalk nginx killed when tried to add new environment - amazon-web-services

Question

I receive this error every time that I need to add a new environment variable from AWS EBS panel:
AWS Beanstalk events:
2018-02-16 14:49:21 UTC-0200 INFO The environment was reverted to the previous configuration setting.
2018-02-16 14:48:49 UTC-0200 ERROR During an aborted deployment, some instances may have deployed the new application version. To ensure all instances are running the same version, re-deploy the appropriate application version.
2018-02-16 14:48:49 UTC-0200 ERROR Failed to deploy configuration.
2018-02-16 14:48:49 UTC-0200 ERROR Unsuccessful command execution on instance id(s) 'i-xxxxxxxxxxxxxx'. Aborting the operation.
2018-02-16 14:48:49 UTC-0200 INFO Command execution completed on all instances. Summary: [Successful: 0, Failed: 1].
eb-activity.log:
Successfully execute hooks in directory /opt/elasticbeanstalk/hooks/configdeploy/enact.
[2018-02-16T16:21:18.921Z] INFO [8550] – [Configuration update app-0_0_10-180216_141535#104/ConfigDeployStage1/ConfigDeployPostHook] : Starting activity…
[2018-02-16T16:21:18.921Z] INFO [8550] – [Configuration update app-0_0_10-180216_141535#104/ConfigDeployStage1/ConfigDeployPostHook/99_kill_default_nginx.sh] : Starting activity…
[2018-02-16T16:21:19.164Z] INFO [8550] – [Configuration update app-0_0_10-180216_141535#104/ConfigDeployStage1/ConfigDeployPostHook/99_kill_default_nginx.sh] : Activity execution failed, because: + rm -f /etc/nginx/conf.d/00_elastic_beanstalk_proxy.conf
+ service nginx stop
Stopping nginx: /sbin/service: line 66: 8986 Killed env -i PATH=”$PATH” TERM=”$TERM” “${SERVICEDIR}/${SERVICE}” ${OPTIONS} (ElasticBeanstalk::ExternalInvocationError)
caused by: + rm -f /etc/nginx/conf.d/00_elastic_beanstalk_proxy.conf
+ service nginx stop
Stopping nginx: /sbin/service: line 66: 8986 Killed env -i PATH=”$PATH” TERM=”$TERM” “${SERVICEDIR}/${SERVICE}” ${OPTIONS} (Executor::NonZeroExitStatus)
[2018-02-16T16:21:19.164Z] INFO [8550] – [Configuration update app-0_0_10-180216_141535#104/ConfigDeployStage1/ConfigDeployPostHook/99_kill_default_nginx.sh] : Activity failed.
[2018-02-16T16:21:19.165Z] INFO [8550] – [Configuration update app-0_0_10-180216_141535#104/ConfigDeployStage1/ConfigDeployPostHook] : Activity failed.
[2018-02-16T16:21:19.165Z] INFO [8550] – [Configuration update app-0_0_10-180216_141535#104/ConfigDeployStage1] : Activity failed.
[2018-02-16T16:21:19.165Z] INFO [8550] – [Configuration update app-0_0_10-180216_141535#104] : Completed activity. Result:
Configuration update – Command CMD-ConfigDeploy failed
Edit: Added stack-https.config file
eb-activity.log:
Command 01_copy_conf_file] : Activity execution failed, because: (ElasticBeanstalk::ExternalInvocationError
Starting activity...
[2018-02-16T20:38:30.476Z] INFO [2536] - [Application deployment app-0_0_10-1-gb633-180216_175029#124/StartupStage0/EbExtensionPostBuild/Infra-EmbeddedPostBuild/postbuild_0_paneladm_api_stack_SampleApplication_W4FJ8W83X64B] : Starting activity...
[2018-02-16T20:38:32.456Z] INFO [2536] - [Application deployment app-0_0_10-1-gb633-180216_175029#124/StartupStage0/EbExtensionPostBuild/Infra-EmbeddedPostBuild/postbuild_0_paneladm_api_stack_SampleApplication_W4FJ8W83X64B/Command 00_removeconfig] : Starting activity...
[2018-02-16T20:38:32.463Z] INFO [2536] - [Application deployment app-0_0_10-1-gb633-180216_175029#124/StartupStage0/EbExtensionPostBuild/Infra-EmbeddedPostBuild/postbuild_0_paneladm_api_stack_SampleApplication_W4FJ8W83X64B/Command 00_removeconfig] : Completed activity.
[2018-02-16T20:38:34.493Z] INFO [2536] - [Application deployment app-0_0_10-1-gb633-180216_175029#124/StartupStage0/EbExtensionPostBuild/Infra-EmbeddedPostBuild/postbuild_0_paneladm_api_stack_SampleApplication_W4FJ8W83X64B/Command 01_copy_conf_file] : Starting activity...
[2018-02-16T20:38:34.538Z] INFO [2536] - [Application deployment app-0_0_10-1-gb633-180216_175029#124/StartupStage0/EbExtensionPostBuild/Infra-EmbeddedPostBuild/postbuild_0_paneladm_api_stack_SampleApplication_W4FJ8W83X64B/Command 01_copy_conf_file] : Activity execution failed, because: (ElasticBeanstalk::ExternalInvocationError)
I don't know if the problem is because I previous removed the default elastic_beanstalk_proxy.conf file with my commands as below:
Resources:
sslSecurityGroupIngress:
Type: AWS::EC2::SecurityGroupIngress
Properties:
GroupId: {"Fn::GetAtt" : ["AWSEBSecurityGroup", "GroupId"]}
IpProtocol: tcp
ToPort: 443
FromPort: 443
CidrIp: 0.0.0.0/0
files:
/etc/letsencrypt/configs/http_proxy.pre:
mode: "000644"
owner: root
group: root
content: |
# Elastic Beanstalk Managed
upstream nodejs {
server 127.0.0.1:8081;
keepalive 256;
}
server {
listen 8080;
access_log /var/log/nginx/access.log main;
location /.well-known {
allow all;
root /usr/share/nginx/html;
}
# Redirect non-https traffic to https.
location / {
if ($scheme != "https") {
return 301 https://$host$request_uri;
} # managed by Certbot
}
}
# The Nginx config forces https, and is meant as an example only.
/etc/letsencrypt/configs/https_custom.pos:
mode: "000644"
owner: root
group: root
content: |
# HTTPS server
server {
listen 443 default ssl;
server_name localhost;
error_page 497 https://$host$request_uri;
ssl_certificate /etc/letsencrypt/live/ebcert/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/ebcert/privkey.pem;
ssl_session_timeout 5m;
ssl_protocols TLSv1.1 TLSv1.2;
ssl_ciphers "EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH";
ssl_prefer_server_ciphers on;
if ($ssl_protocol = "") {
rewrite ^ https://$host$request_uri? permanent;
}
location / {
proxy_pass http://nodejs;
proxy_set_header Connection "";
proxy_http_version 1.1;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
}
gzip on;
gzip_comp_level 4;
gzip_types text/plain text/css application/json application/javascript application/x-javascript text/xml application/xml application/xml+rss text/javascript;
}
/etc/letsencrypt/configs/generate-cert.sh:
mode: "000664"
owner: root
group: root
content: |
#!/bin/sh
_EMAIL=
_DOMAIN=
while getopts ":e:d:" OPTION;
do
case "${OPTION}" in
"e") _EMAIL="${OPTARG}";;
"d") _DOMAIN="${OPTARG}";;
esac
done
if [ -z "${_EMAIL}" ]; then
echo "Param email isn't specified!"
fi
if [ -z "${_DOMAIN}" ]; then
echo "Param domain isn't specified!"
fi
if [ -n "$_EMAIL" ] && [ -n "$_DOMAIN" ]; then
cd /opt/certbot/
./certbot-auto certonly \
--debug --non-interactive --email ${_EMAIL} \
--webroot -w /usr/share/nginx/html --agree-tos -d ${_DOMAIN} --keep-until-expiring
fi
if [ $? -ne 0 ]
then
ERRORLOG="/var/log/letsencrypt/letsencrypt.log"
echo "The Let's Encrypt cert has not been renewed!\n" >> $ERRORLOG
else
/etc/init.d/nginx reload
fi
exit 0
/opt/elasticbeanstalk/hooks/configdeploy/post/99_kill_default_nginx.sh:
mode: "000755"
owner: root
group: root
content: |
#!/bin/bash -xe
rm -f /etc/nginx/conf.d/00_elastic_beanstalk_proxy.conf
service nginx stop
service nginx start
packages:
yum:
epel-release: []
container_commands:
00_removeconfig:
command: "rm -f /tmp/deployment/config/#etc#nginx#conf.d#00_elastic_beanstalk_proxy.conf /etc/nginx/conf.d/00_elastic_beanstalk_proxy.conf"
01_copy_conf_file:
command: "cp /etc/letsencrypt/configs/http_proxy.pre /etc/nginx/conf.d/http_proxy.conf; /etc/init.d/nginx reload"
02_createdir:
command: "mkdir /opt/certbot || true"
03_installcertbot:
command: "wget https://dl.eff.org/certbot-auto -O /opt/certbot/certbot-auto"
04_permission:
command: "chmod a+x /opt/certbot/certbot-auto"
05_getcert:
command: "sudo sh /etc/letsencrypt/configs/generate-cert.sh -e ${CERT_EMAIL} -d ${CERT_DOMAIN}"
06_link:
command: "ln -sf /etc/letsencrypt/live/${CERT_DOMAIN} /etc/letsencrypt/live/ebcert"
07_copy_ssl_conf_file:
command: "cp /etc/letsencrypt/configs/https_custom.pos /etc/nginx/conf.d/https_custom.conf; /etc/init.d/nginx reload"
08_cronjob_renew:
command: "sudo sh /etc/letsencrypt/configs/generate-cert.sh -e ${CERT_EMAIL} -d ${CERT_DOMAIN}"
I'm doing this because I replace this file to my own proxy.conf file.
Please I need your help.
References:
awslabs/elastic-beanstalk-sampes/https-redirect-nodejs.config
AWS EBS - Environment Properties and Other Software Settings

Answer 1

I had this problem as well and Amazon acknowledged the error in the documentation. This is a working restart script that you can use in your .ebextensions config file.
/opt/elasticbeanstalk/hooks/configdeploy/post/99_kill_default_nginx.sh:
mode: "000755"
owner: root
group: root
content: |
#!/bin/bash -xe
rm -f /etc/nginx/conf.d/00_elastic_beanstalk_proxy.conf
status=`/sbin/status nginx`
if [[ $status = *"start/running"* ]]; then
echo "stopping nginx..."
stop nginx
echo "starting nginx..."
start nginx
else
echo "nginx is not running... starting it..."
start nginx
fi