How to customize the docker run command on Elastic Beanstalk? - amazon-web-services

Here's the thing, I need to tell Docker to not containerize the container’s networking, because it needs to connect to a MongoDB that is inside a VPN (enterprise private DB).
There is a Docker command that let's me do exactly that: --net=host. Reference here.
So, for example, when running the container on my local machine, I will do something like:
docker run --rm -it --net=host [image-name]:[version] bash -il
And that command will do the trick. Thanks to that, I can connect to the "private" MongoDB.
So, my question is: Is there a way customize the docker run command of a Single Docker Environment on Elastic Beanstalk so I can add the --net=host?
I have tried using the container_commands into the config.yml file to add that instruction there, but I don't think that does what I need, here is a snippet:
container_commands:
00-test_command:
command: bundle exec thin --net=host
01-networking-fix:
command: "docker run --rm -it --net=host [image-name]:[version] bash -il"

I ended up fixing it with two container commands
container_commands:
00_fix_networking:
command: sed -i 's/docker run -d/docker run --net=host -d/' /opt/elasticbeanstalk/hooks/appdeploy/enact/00run.sh
01_fix_docker_ip:
command: sed -i 's/server $EB_CONFIG_NGINX_UPSTREAM_IP/server localhost/' /opt/elasticbeanstalk/hooks/appdeploy/enact/01flip.sh
Update:
I also had to fix the Upstart script. Unfortunately, I didn't write down what I did because I didn't end up needing to alter the docker run command. You would do a files directive for (I think) /etc/init/docker. AWS edits the Nginx configuration in the same manner as in 01flip.sh in that file as well.
Explanation:
In the 64bit Amazon Linux 2015.03 v2.0.2 running Docker 1.7.1 platform version, the file you need to edit is /opt/elasticbeanstalk/hooks/appdeploy/enact/00run.sh. This file is now far more complex than Samar's version so I didn't want to put the actual contents in there. However, the change is basically the same. There's the line that starts with
docker run -d
I fixed it with a container command:
container_commands:
00_fix_networking:
command: sed -i 's/docker run -d/docker run --net=host -d/' /opt/elasticbeanstalk/hooks/appdeploy/enact/00run.sh
This successfully adds the --net=host argument but now there's another problem. The system ends up with an invalid Nginx directive. Using --net=host means that when you run docker inspect <container id> there is no IP address in the NetworkSettings. AWS uses this to create the server directive for Nginx and ends up generating server :<some port you chose> (before adding --net=host it would look like server <ip>:<port>). I needed to patch that file, too. It's generated in /opt/elasticbeanstalk/hooks/appdeploy/enact/01flip.sh.
01_fix_docker_ip:
command: sed -i 's/server $EB_CONFIG_NGINX_UPSTREAM_IP/server localhost/' /opt/elasticbeanstalk/hooks/appdeploy/enact/01flip.sh

While elastic beanstalk is generally well suited for applications that work with standard set of configurations, its difficult to customize and keep things updated along with the updates AWS provides to EB stacks. Having said that, I've done something like below which is a bit hacky but works fine.
files:
"/opt/elasticbeanstalk/hooks/appdeploy/pre/04run.sh":
mode: "000755"
owner: root
group: root
encoding: plain
content: |
#script content of original 04run.sh along with modification on docker run cmd
# eg. I injected multi-ports here
docker run -d \
"${EB_CONFIG_DOCKER_ENV_ARGS[#]}" \
"${EB_CONFIG_DOCKER_VOLUME_MOUNTS[#]}" \
"${EB_CONFIG_DOCKER_ENTRYPOINT_ARGS[#]}" \
"${PORT_ARGS[#]}" \
$EB_CONFIG_DOCKER_IMAGE_STAGING \
"${EB_CONFIG_DOCKER_COMMAND_ARGS[#]}" 2>&1 | tee /tmp/docker_run.log | tee $EB_CONFIG_DOCKER_STAGING_APP_FILE
This is not very neat, at least I have to make sure that it does not break with updates on elastic beanstalk. The above one is for docker 1.5 stack but you can do something similar with the version you're running.

Note that the latest version of the AWS stack (with Docker 1.7.1) has a slightly different pre-deploy setup. You'll need to update the file at the location: /opt/elasticbeanstalk/hooks/appdeploy/enact/00run.sh
commands:
00001_add_privileged:
cwd: /tmp
command: 'sed -i "s/docker run -d/docker run --privileged -d/" /opt/elasticbeanstalk/hooks/appdeploy/enact/00run.sh'
or, for example, if you want to pass args to your Docker image:
commands:
00001_modify_docker_run:
cwd: /tmp
command: 'sed -i "s/\$EB_CONFIG_DOCKER_IMAGE_STAGING/\$EB_CONFIG_DOCKER_IMAGE_STAGING -gzip -enable-url-source/" /opt/elasticbeanstalk/hooks/appdeploy/enact/00run.sh'

Related

Accessing S3 object using docker run command through Boto3, throws ProfileNotFound exceptions but works well when run through python

I am working on a project, where I need to read Text File from S3 bucket using Boto3. Then need to dockerize my application. I Implemented my code using Boto3, its running Perfectly fine(Note: Taking two arguments through argparser with switches -p and -n). But when I try the same using Docker run
PS **my Working Directory** > docker run --rm Image_name -p Argument1 -n Argument2
<class 'botocore.exceptions.ProfileNotFound'> Code.py 92
I searched lot of things on it, my understanding is AWS container directory is unable to locate my Credential File and config file stored in my home directory/.aws folder.
What I Tried:
1. Path mounting as below:
PS **my Working Directory** > docker run --rm -it -v %userprofile%\.aws:/root/.aws
amazon/aws-cli
docker: Error response from daemon: %!u(string=is not a valid Windows path)serprofile%!\
(MISSING).aws.
See 'docker run --help'.
I completely don't understand what's wrong with the syntax. I tried manually inputting my user profile directory in the %userprofile% as C:/Users/Deepak
Then Strangely WSL2(backend) popup comes saying using passing containers on windows may poorly work.
I am not sure what it means. Does it have any effect on Docker containers build on windows environment?
2. I moved my Credential and Config file in my working directory as well and tried below code:
PS my Working Directory > docker run --rm -it -v ${PWD}:/root/.aws amazon/aws-cli Image_name -p
Argument1 -n Argument2

Docker expose not working elastic beanstalk

I have a docker container of tinyproxy that I have put on EB. The container is running fine, Docker ps shows ports 8888/tcp, and logging into the container and doing netstat I see it waiting 0.0.0.0:8888 & :::8888.
But on my EB host I don't see any connection. Below is my dockerfile, I don't have a Dockerrun.aws.json for my single container as it seems not be be mandatory. Any ideas?
FROM alpine:3.7
MAINTAINER Daniel Middleton <monokal.io>
RUN apk add --no-cache \
bash \
tinyproxy
COPY run.sh /opt/docker-tinyproxy/run.sh
RUN chmod +x /opt/docker-tinyproxy/run.sh
EXPOSE 8888
ENTRYPOINT ["/opt/docker-tinyproxy/run.sh","docker run -d --name='tinyproxy' -p 8888:8888 dannydirect/tinyproxy:latest ANY"]
Edit from comments below:
docker ps -a output is:
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
cd352352ddb ffe23df0902 "/opt/docker-tinypro…" 2 hours ago Up 2 hours 8888/tcp zealous_volhard
and when accessing url from allowed ip in security group following output is given:
Not Implemented
Unknown method or unsupported protocol.
Generated by tinyproxy version 1.8.4.
Any ideas?
Judging from your output, you are reaching tinyproxy so your ports are properly exposed and utilitized.
Now, what is strange is your entrypoint... "/opt/docker-tinyproxy/run.sh" seems like regular tinyproxy start, but... "docker run -d --name='tinyproxy' -p 8888:8888 dannydirect/tinyproxy:latest ANY" seems completely out of place. As far as I can tell from inspecting tinyproxy container, following should be set as entrypoint and cmd:
"Cmd": [
"ANY"
],
"Entrypoint": [
"/opt/docker-tinyproxy/run.sh"
]
So you might want to adjust your entrypoint accordingly, since it basically expects /opt/docker-tinyproxy/run.sh ANY as starting point and gets something completely different.

docker restart container failed: "already in use", but there's no more docker image

I first got my nginx docker image:
docker pull nginx
Then I started it:
docker run -d -p 80:80 --name webserver nginx
Then I stopped it:
docker stop webserver
Then I tried to restart it:
$docker run -d -p 80:80 --name webserver nginx
docker: Error response from daemon: Conflict. The container name "/webserver" is already in use by container 036a0bcd196c5b23431dcd9876cac62082063bf62a492145dd8a55141f4dfd74. You have to remove (or rename) that container to be able to reuse that name..
See 'docker run --help'.
Well, it's an error. But in fact there's nothing in container list now:
docker container list
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
Why I restart nginx image failed? How to fix it?
It is because
you have used --name switch.
container is stopped and not removed
You find it stopped
docker ps -a
You can simply start it using below command:
docker start webserver
EDIT: Alternatives
If you want to start the container with below command each time,
docker run -d -p 80:80 --name webserver nginx
then use one of the following:
method 1: use --rm switch i.e., container gets destroyed automatically as soon as it is stopped
docker run -d -p 80:80 --rm --name webserver nginx
method 2: remove it explicitly after stopping the container before starting the command that you are currently using.
docker stop <container name>
docker rm <container name>
As the error says.
You have to remove (or rename) that container to be able to reuse that name
This leaves you two options.
You may delete the container that is using the name "webserver" using the command
docker rm 036a0bcd196c5b23431dcd9876cac62082063bf62a492145dd8a55141f4dfd74
and retry.
Or you may use a different name during the run command. This is not recommended, as you no longer need that docker.
It's better to remove the unwanted docker and reuse the name.
While the great answers are correct, they didn't actually solve the problem I was facing.
How To:
Safely automate starting of named docker container regardless of its prior state
The solution is to wrap the docker run command with an additional check and either do a run or a stop + run (effectively restart with the new image) based on the result.
This achieves both of my goals:
Avoids the error
Allows me to periodically update the image (say new build) and restart safely
#!/bin/bash
# Adapt the following 3 parameters to your specific case
NAME=myname
IMAGE=myimage
RUN_OPTIONS='-d -p 8080:80'
ContainerID="$(docker ps --filter name="$NAME" -q)"
if [[ ! -z "$ContainerID" ]]; then
echo "$NAME already running as container $ContainerID: stopping ..."
docker stop "$ContainerID"
fi
echo "Starting $NAME ..."
exec docker run --rm --name "$NAME" $RUN_OPTIONS "$IMAGE"
Now I can run (or stop + start if already running) the $NAME docker container in a idempotent way, without worrying about this possible failure.

Where does dockerized jetty store its logs?

I'm packaging a project into a docker jetty image and I'm trying to access the logs, but no logs.
Dockerfile
FROM jetty:9.2.10
MAINTAINER Me "me#me.com"
ADD ./target/abc-1.0.0 /var/lib/jetty/webapps/ROOT
EXPOSE 8080
Bash script to start docker image:
docker pull me/abc
docker stop abc
docker rm abc
docker run --name='abc' -d -p 10908:8080 -v /var/log/abc:/var/log/jetty me/abc:latest
The image is running, but I'm not seeing any jetty logs in /var/log.
I've tried a docker run -it jetty bash, but not seeing any jetty logs in /var/log either.
Am I missing a parameter to make jetty output logs or does it output it somewhere other than /var/log/jetty?
Why you aren't seeing logs
2 things to note:
Running docker run -it jetty bash will start a new container instead of connecting you to your existing daemonized container.
And it would invoke bash instead of starting jetty in that container, so it won't help you to get logs from either container.
So this interactive container won't help you in any case.
But also...
JettyLogs are disabled anyways
Also, you won't see the logs in the standard location (say, if you tried to use docker exec to read the logs, or to get them in a volume), quite simply because the Jetty Docker file is aptly disabling logging entirely.
If you look at the jetty:9.2.10 Dockerfile, you will see this line:
&& sed -i '/jetty-logging/d' etc/jetty.conf \
Which nicely removes the entire line referencing the jetty-logging.xml default logging configuration.
What to do then?
Reading logs with docker logs
Docker gives you access to the container's standard output.
After you did this:
docker run --name='abc' -d -p 10908:8080 -v /var/log/abc:/var/log/jetty me/abc:latest
You can simply do this:
docker logs abc
And be greeted with somethig similar to this:
Running Jetty:
2015-05-15 13:33:00.729:INFO::main: Logging initialized #2295ms
2015-05-15 13:33:02.035:INFO:oejs.SetUIDListener:main: Setting umask=02
2015-05-15 13:33:02.102:INFO:oejs.SetUIDListener:main: Opened ServerConnector#73ec519{HTTP/1.1}{0.0.0.0:8080}
2015-05-15 13:33:02.102:INFO:oejs.SetUIDListener:main: Setting GID=999
2015-05-15 13:33:02.106:INFO:oejs.SetUIDListener:main: Setting UID=999
2015-05-15 13:33:02.133:INFO:oejs.Server:main: jetty-9.2.10.v20150310
2015-05-15 13:33:02.170:INFO:oejdp.ScanningAppProvider:main: Deployment monitor [file:/var/lib/jetty/webapps/] at interval 1
2015-05-15 13:33:02.218:INFO:oejs.ServerConnector:main: Started ServerConnector#73ec519{HTTP/1.1}{0.0.0.0:8080}
2015-05-15 13:33:02.219:INFO:oejs.Server:main: Started #3785ms
Use docker help logs for more details.
Customize
Obviously your other option is to revert what the default Dockerfile for jetty is doing, or to create your own dockerized Jetty.

Amazon Elastic Beanstalk - Change Timezone

I´m running an EC2 instance through AWS Elastic Beanstalk. Unfortunately it has the incorrect timezone - it´s 2 hours earlier than it should be, because timezone is set to UTC. What I need is GMT+1.
Is there a way to set up the .ebextensions configuration, in order to force the EC2 instance to use the right timezone?
Yes, you can.
Just create a file /.ebextensions/00-set-timezone.config with following content
commands:
set_time_zone:
command: ln -f -s /usr/share/zoneinfo/Australia/Sydney /etc/localtime
This is assuming your are using default Amazon Linux AMI image. If you use some other Linux distribution, just change the command to whatever it requires to set timezone in that Linux.
This is a response from the aws Support Business and this works!
---- Original message ----
How can I change the timezone of an enviroment or rather to the instances of the enviroment in Elastic Beasntalk to UTC/GMT -3 hours (Buenos Aires, Argentina)?
I´m currently using Amazon Linux 2016.03. Thanks in advance for your help.
Regards.
---------- Response ----------
Hello,Thank you for contacting AWS support regarding modifying your Elastic Beanstalk instances time zone to use UTC/GMT -3 hours (Buenos Aires, Argentina), please see below on steps on how to perform this modification.
The below example shows how to modify timezone for Elastic Beanstalk environment using .ebextensions for Amazon Linux OS:
Create .ebextensions folder in the root of your application
Create a .config file for example 00-set-timezone.config file and add the below content in yaml formatting.
container_commands:
01changePHP:
command: sed -i '/PHP_DATE_TIMEZONE/ s/UTC/America\/Argentina\/Buenos_Aires/' /etc/php.d/environment.ini
01achangePHP:
command: sed -i '/aws.php_date_timezone/ s/UTC/America\/Argentina\/Buenos_Aires/' /etc/php.d/environment.ini
02change_AWS_PHP:
command: sed -i '/PHP_DATE_TIMEZONE/ s/UTC/America\/Argentina\/Buenos_Aires/' /etc/httpd/conf.d/aws_env.conf
03php_ini_set:
command: sed -i '/date.timezone/ s/UTC/America\/Argentina\/Buenos_Aires/' /etc/php.ini
commands:
01remove_local:
command: "rm -rf /etc/localtime"
02link_Buenos_Aires:
command: "ln -s /usr/share/zoneinfo/America/Argentina/Buenos_Aires /etc/localtime"
03restart_http:
command: sudo service httpd restart
Deploy application to Elastic Beanstalk including the .ebextensions and the timezone will change as per the above.
I hope that helps
Regards!
If you are running windows in your eb environment...
.
create a folder named .ebextensions in the root of your project..
inside that folder create a file named timezone.config
in that file add the following :
commands:
set_time_zone:
command: tzutil /s "Central Standard Time"
set the time zone as needed
screenshot
I'm using custom .ini file in php.d folder along with regular recommendations from http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/set-time.html#change_time_zone:
The sed command inserts (rewrites) only the first line of /etc/sysconfig/clock, since the second line (UTC=true) should be left alone, per the above AWS documentation.
# .ebextensions/02-timezone.config
files:
/etc/php.d/webapp.ini:
mode: "000644"
owner: root
group: root
content: |
date.timezone="Europe/Amsterdam"
commands:
01_set_ams_timezone:
command:
- sed -i '1 s/UTC/Europe\/Amsterdam/g' /etc/sysconfig/clock
- ln -sf /usr/share/zoneinfo/Europe/Amsterdam /etc/localtime
Changing the time zone of EC2 with Elastic Beanstalk is simple:
Create a .ebextensions folder in the root
Add a file with filename end with .config (timezone.config)
Inside the file
container_commands:
time_zone:
command: ln -f -s /usr/share/zoneinfo/America/Argentina/Buenos_Aires /etc/localtime
Then you have done.
Note that the container_commands is different from commands, from the document it states:
commands run before the application and web server are set up and
the application version file is extracted.
That's the reason of your time zone command doesn't work because the server hasn't started yet.
container_commands run after the application and web server have been
set up and the application version file has been extracted, but before
the application version is deployed.
If you are runing a java/Tomcat container, just put the JVM Option on the configuration.
-Duser.timezone=America/Sao_Paulo
Possibles values: timezones
Moving to AWS Linux 2 was challenging. It took me a while to work out how to do this easily in .ebextensions.
I wrote the simple solution in another stackoverflow question .. but for anyone needing instant gratification .. add the following commands into the file .ebextensions/xxyyzz.config:
container_commands:
01_set_bne:
command: "sudo timedatectl set-timezone Australia/Brisbane"
command: "sudo systemctl restart crond.service"
These workarounds only fixes the timezone for applications. But when you have any system services like a cron run it looks at the /etc/sysconfig/clock and that is always UTC. If you tail the cron logs or aws-sqsd logs would will notice timestamps are still 2hrs behind - in my case. And a change to the clock setting would need a reboot into order to take effect - which is not an option to consider should you have autoscaling in place or should you want to use ebextensions to change the system clock's config.
Amazon is aware of this issue and I dont think they have resolved it yet.
If your EB application is using the Java/Tomcat container, you can add the JVM timezone Option to the Procfile configuration. Example:
web: java -Duser.timezone=Europe/Berlin -jar application.jar
Make sure to add all configuration options before the -jar option, otherwise they are ignored.
in the .ebextensions added below for PHP
container_commands:
00_changePHP:
command: sed -i '/;date.timezone =/c\date.timezone = \"Australia/Sydney\"' /etc/php.ini
01_changePHP:
command: sed -i '/date.timezone = UTC/c\date.timezone = \"Australia/Sydney\"' /etc/php.d/aws.ini
02_set_tz_AEST:
command: "sudo timedatectl set-timezone Australia/Sydney"
command: "sudo systemctl restart crond.service"
commands:
01remove_local:
command: "rm -rf /etc/localtime"
02change_clock:
command: sed -i 's/\"UTC\"/\"Australia\/Sydney\"/g' /etc/sysconfig/clock
03link_Australia_Sydney:
command: "ln -f -s /usr/share/zoneinfo/Australia/Sydney /etc/localtime"
cwd: /etc
Connect AMI(amazon linux instance) via putty or ssh and execute the commands below;
sudo rm /etc/localtime
sudo ln -sf /usr/share/zoneinfo/Europe/Istanbul /etc/localtime
sudo reboot
Explanation of the procedure above is simply;
remove localtime,
update the timezone,
reboot
Please notify that I've changed my timezone to Turkey's localtime, you can find your timezone by listing zoneinfo directory with the command below;
ls /usr/share/zoneinfo
or just check timezone abbrevetaions via wikipedia;
http://en.wikipedia.org/wiki/Category:Tz_database
You can also check out the related Amazon AWS documentation;
http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/set-time.html
Note: I'm not sure that if this is the best practice or not (probably not), however I've applied the procedure I've written above and it's working for me.