Starting NginX with my modified nginx.conf on ECS - amazon-web-services

I have an environment in AWS with an ECS cluster, an EFS source and some services running on this cluster.
One of my services is the NginX web server which I use to serve our site and our services. As a solution to keep some sensitive and static configuration files we have chosen the EFS service. So, each service creates a volume from this EFS and mount it every time a container starts.
The problem is with NginX. I want to store my nginx.conf file into an EFS folder and after the NginX service starts, we want the container to copy this file at /etc/nginx/ folder in order for my NginX server to start with my configuration.
I've tried to build my own image including my configuration with success but this is not what we want.That means that we should build a new image every time we want to change a line on nginx.conf.
I've tried to create a script to run every time the container starts and copy my configuration but i didn't manage to make it play on ECS. Either the NginX failed to reload, either the syntax is wrong, either the file is not available.
#!/bin/bash
cp /efs/nginx.conf /etc/nginx/
nginx -s reload
Ι considered to find out how to create a cron job to run every X minutes and copy my nginx.conf to etc/nginx but this seems to be a stupid approach.
I made like 60 different task definitions revisions in order to find out how this CMD Environment option works on ECS. Of course the most of them has to do with the syntax and i get bach errors like "invalid option: bash" or "invalid option: /tmp/1.sh" etc
Samples:
1.Command ["cp","/efs/nginx.conf /etc/nginx/"]
2.Entry point ["nginx","-g","daemon off;"]
Command ["cp /efs/nginx.conf /etc/nginx/"]
Entry point: ["nginx","-g","daemon off"]
Command: ["/bin/sh","cp","/efs/nginx.conf/","/etc/nginx/"]
Command ["[\"cp\"","\"/efs/nginx.conf\"","\"/etc/nginx/\"]","[\"nginx\"","\"-g\"","\"daemon off;\"]"]
Command ["cp /efs/nginx.conf /etc/nginx/","nginx -g daemon off;"]
Command ["cp","/efs/nginx.conf /etc/nginx/","nginx -g daemon off;"]
-
Does anyone knows or does anyone already implement this solution on ECS?
To replace /etc/nginx/nginx.conf with a modified one from a binded volume?
Thanks in advance

SOLUTION:
As I mention at my question above, I'd like to use a static nginx.conf file, which will be into an EFS folder, into my nginx service container.
My task definition is simple like this
FROM nginx
EXPOSE 80
RUN mkdir /etc/nginx/html
Through ECS task definition I create a volume and then a mounting point which is an easy process and works fine. The problem was in the entrypoint field which supposed to include my script's directory and to my script itself.
At ECS task definition Environment entrypoint field i putted
sh,-c,/efs/docker-cmd-nginx.sh
and my script is just the following
#!/bin/dash
cp /efs/nginx.conf /etc/nginx/ &&
nginx -g "daemon off;"
PS: The problem probably was at:
my script which I didn't use double quotes at the daemon off; part but I was using double quotes on the whole line nginx -g daemon off;
my script was trying to reload nginx which was not even running yet.
my attempt to put the commands seperately at my task's entrypoint was wrong, syntax-wise for sure and maybe strategy-wise as well.

Related

Elastic BeanStalk app deploy post hook not executing my command

I recently was able to get my Laravel app deployed using codepipeline on Elastic Beanstalk but ran into a problem. I noticed that my routes where failing because of php.conf Nginx configuration. I had to add a few lines of code to EB's nginx php.conf file to get it to work.
My problem now was that after every deployment, the instance of the application I modified the php.conf file was destroyed and recreated fresh. I wanted a way to dynamically update the file after every successful deployment. I had a version of the file I wanted versioned with my application and so wanted to create a symlink to that file after deployment.
After loads of research, I stumbled on appDeploy Hooks on Elastic Beanstalk that runs post scripts after deployment so did this
files:
"/opt/elasticbeanstalk/hooks/appdeploy/post/91_post_deploy_script.sh":
mode: "000755"
owner: root
group: root
content: |
#!/usr/bin/env bash
sudo mkdir /var/testing1
sudo ln -sfn /var/www/html/php.conf.example /etc/nginx/conf.d/elasticbeanstalk/php.conf
sudo mkdir /var/testing
sudo nginx -s reload
And this for some reason does not work. The symlink is not created so my routes are still not working..
I even added some mkdir so am sure the commands in that script runs, none of those commands ran because none of those directories where created.
Please note that if I ssh into the ec2 instance and run the commands there it works. That bash script also exists in the post directory and if I manually run in on the server it works too.
Any pointers to how I could fix this would be helpful. Maybe I am doing something wrong too.
Now I have gotten my scripts to run by following this. However, the script is not running. I am getting an error
2020/06/28 08:22:13.653339 [INFO] Following platform hooks will be executed in order: [01_myconf.config]
2020/06/28 08:22:13.653344 [INFO] Running platform hook: .platform/hooks/postdeploy/01_myconf.config
2020/06/28 08:22:13.653516 [ERROR] An error occurred during execution of command [app-deploy] - [RunPostDeployHooks]. Stop running the command. Error: Command .platform/hooks/postdeploy/01_myconf.config failed with error fork/exec .platform/hooks/postdeploy/01_myconf.config: permission denied
I tried to follow this forum post here to make my file executable by adding to my container command a new command like so:
01_chmod1:
command: "chmod +x .platform/hooks/postdeploy/91_post_deploy_script.sh"
I am still running into the same issue. Permission denied
Sadly, the hooks you are describing (i.e. /opt/elasticbeanstalk/hooks/appdeploy) are for Amazon Linux 1.
Since you are using Amazon Linux 2, as clarified in the comments, the hooks you are trying to use do not apply. Thus they are not being executed.
In Amazon Linux 2, there are new hooks as described here and they are:
prebuild – Files here run after the Elastic Beanstalk platform engine downloads and extracts the application source bundle, and before it sets up and configures the application and web server.
predeploy – Files here run after the Elastic Beanstalk platform engine sets up and configures the application and web server, and before it deploys them to their final runtime location.
postdeploy – Files here run after the Elastic Beanstalk platform engine deploys the application and proxy server.
The use of these new hooks is different than in Amazon Linux 1. Thus you have to either move back to Amazon Linux 1 or migrate your application to Amazon Linux 2.
General migration steps from Amazon Linux 1 to Amazon Linux 2 in EB are described here
Create a folder called .platform in your project root folder and create a file with name 00_myconf.config inside the .platform folder.
.platform/
00_myconf.config
Open 00_myconf.config and add the scripts
files:
"/opt/elasticbeanstalk/hooks/appdeploy/post/91_post_deploy_script.sh":
mode: "000755"
owner: root
group: root
content: |
#!/usr/bin/env bash
sudo mkdir /var/testing1
sudo ln -sfn /var/www/html/php.conf.example /etc/nginx/conf.d/elasticbeanstalk/php.conf
sudo mkdir /var/testing
sudo nginx -s reload
Commit your changes or reupload the project. This .platform folder will be considered in each new instance creation and your application will deploy properly in all the new instances Amazon Elastic beanstalk creates.
If you access the documentation here and scroll to the section with the title "Application example with extensions" you can see an example of the folder structure of your .platform folder so it adds your custom configuration to NGINX conf on every deploy.
You can either replace the entire nginx.conf file with your file or add additional configuration files to the conf.d directory
Replace conf file with your file on app deploy:
.platform/nginx/nginx.conf
Add configuration files to nginx.conf:
.platform/nginx/conf.d/custom.conf

Can i keep a task definition active, even after my Java job is completed when using docker

I have an application that i run through docker file. Is it possible for me to keep the task definition active even if the application in it gets killed.
Docker File sample
FROM <domain>:<port>/<package_name>/<image_nmae>
ADD target/<app_jar>.jar /app.jar
ADD lib/* /lib/
ADD log4j2.xml /lib/
ADD <some_script>.sh /setup.sh
CMD ["/bin/bash", "/setup.sh"]
yep task definition is like the docker run command with options so it's only a config. you only get billed for ec2 or fargate usage and first 500mb of ECR free

How to copy data from docker container to ECS on startup (AWS)?

I have two containers, one is web-server based on Node.JS with assets directory. Another container is nginx which proxify page requests to web-server and getting statics from assets directory.
I created AWS cluster, EC2 instance, built and pushed docker images to registry, made tasks to deploy my applications, but I can't share with assets directory to nginx because directory is not part of this container.
So to solve my problem I figured out to create EFS and attach the volume, add permissions to ec2-user and makes directory available by path /var/html/assets.
Cool and how to copy assets content from my web-server docker container to /var/html/assets?
I want to make it public / shared because soon I will make additional servers which should also place assets to this common directory.
The process should be automized and work on each deployment, guys, any suggestions? Thanks!
To copy assets content from your web-server docker container to your host machine,
say you want to save your assets content from container to /var/html/assets on host machine, use this command to run your container:
docker run --name=nginx -d -v ~/var/html/assets:[Your Container path] -p 5000:80 nginx
-v ~/var/html/assets:[Your Container path] Sets up a bindmount volume that links [Your Container path] directory from inside the Nginx container to the ~/var/html/assets directory on the host machine. Docker uses a : to split the host's path from the container path, and the host path always comes first.
Hope it will help!
I solved problem by making host directory accessible for writing chmod 777 /var/html/assets, then added a volume which is looking to host directory and applied it to web and nginx containers. When running the web container, it invokes cp instruction to copy assets to mount directory (host directory). Nginx will see populated directory and can use it.
Note: It's a temporary / workaround solution, giving xrw access to directory is not a good way because of security.

Elastic Beanstalk Procfile for go

I'm trying to deploy my go restful server program to EC2 Linux using Elastic Beanstalk. The document says that I need to create a Procfile at the root. So I did. Here are the steps:
Build my go program myapp.go to using
$ go build -o myapp -i myapp.go
Create a Procfile with exact name at the root with
web: myapp
Zip up the Procfile and the myapp image to a myapp.zip file.
Upload to the server via Elastic Beanstalk console. But I keep getting Degraded health and warning with
WARN Process termination taking longer than 10 seconds.
Any suggestions. By the way, I tried to use the same procfile procedure on the simple application.go zip file came from the Elastic Beanstalk example library. It didn't work either.
I was finally able to get a Go application to deploy with Elastic Beanstalk using the eb client. There are a few things that EB requires:
The name of your main file should be application.go.
Make sure your app is listening on port 5000.
You'll need a Procfile in the main root with
web: bin/application
You'll need a Buildfile with
make: ./build.sh
And finally you'll need a build.sh file with
#!/usr/bin/env bash
# Stops the process if something fails
set -xe
# All of the dependencies needed/fetched for your project.
# FOR EXAMPLE:
go get "github.com/gin-gonic/gin"
# create the application binary that eb uses
GOOS=linux GOARCH=amd64 go build -o bin/application -ldflags="-s -w"
Then if you run eb deploy (after creating your initial eb repository), it should work. I wrote a whole tutorial for deploying a Gin application on EB here. The section specifically on deploying with Elastic Beanstalk is here.

Deployment on AWS Elastic Beanstalk with Docker fails

I'm developing a web application with Play framework and I'm running it on AWS Elastic Beanstalk using a single docker container and a load balancer. Normally, everything is running fine, but when I rebuild the whole environment I get the following error:
Command failed on instance. Return code: 6 Output: (TRUNCATED)... in /etc/nginx/sites-enabled/elasticbeanstalk-nginx-docker-proxy.conf:11 nginx: [emerg] host not found in upstream "docker" in /etc/nginx/sites-enabled/elasticbeanstalk-nginx-docker-proxy.conf:24 nginx: configuration file /etc/nginx/nginx.conf test failed.
When I log into the EC2 I can see that no docker image is running and therefore the Nginx server cannot start. I cannot see any other error in the logs (or maybe I don't know where to look). The strange thing is that the same version worked fine before rebuilding the environment.
I'm using the following Dockerfile for the deployment:
FROM java
COPY <app_folder> /opt/<app_name>
WORKDIR /opt/<app_name>
CMD [ "/opt/<app_name>/bin/<app_name>", "-mem", "512", "-J-server" ]
EXPOSE 9000
Any ideas what the problem could be or where to check for more details?
I had this same problem. elasticbeanstalk-nginx-docker-proxy.conf is referring to proxy_pass http://docker but the definition of that is missing. You need to add something like
# List of application servers
upstream docker {
server 127.0.0.1:8080; # your app
}
(Make sure it's outside of the server directive.)
I have just been working through the same challenge (deploying an updated Docker image to Elastic Beanstalk). And it depends on what you want to do exactly, but what I found out is that (once you have the eb cli setup) you can just use the eb deploy command to push out your code changes without worrying about the image at all.
Granted you'd still want to push your image up to your repo for sharing purposes (with other developers), OR if you actually need to change the environment configuration for some reason... but if you're just looking to push code look into eb deploy
As far as the specifics of your error unfortunately I can't be of much help there. Good luck!