I am trying to run a django application inside of a docker container (ECS - Fargate)
However, I am having trouble figuring out how to run multiple commands in the Command section of a task definition, currently it is configured like this
Howevery my containers keep on STOPPING and I cant even see the logs in CloudWatch
How do I get it to execute properly?, any help is highly appreciated
In your case I would do this by using /bin/sh -c despite the entry point:
/bin/sh -c "python manage.py ... <and the rest>"
This is also how it is done in the offical AWS ECS tutorial:
"entryPoint": [
"sh",
"-c"
],
"command": [
"/bin/sh -c \"echo '<html> <head> <title>Amazon ECS Sample App</title> <style>body {margin-top: 40px; background-color: #333;} </style> </head><body> <div style=color:white;text-align:center> <h1>Amazon ECS Sample App</h1> <h2>Congratulations!</h2> <p>Your application is now running on a container in Amazon ECS.</p> </div></body></html>' > /usr/local/apache2/htdocs/index.html && httpd-foreground\""
]
This worked for me using a aws ecs run-task command.
{
"command": [
"bin/sh", "-c", "echo 'Hello' && echo ' alien ' && echo 'World'"
]
}
The command only has to be split for bin/sh and -c and the rest of the commands can be joined together with &&.
Made it with && replacing spaces by commas, just like that:
touch,/usr/app/log/test1.txt,&&,touch,/usr/app/log/test2.txt
The issue was with one of the services rk-nginx
This was the previous nginx.conf
upstream gunicorn {
# docker will automatically resolve this to the correct address
# because we use the same name as the service: "django"
server django:8000;
}
However the django keyword gave me issues ( I think it was part of how ecs handles networks), I simply changed it to this
upstream gunicorn {
server localhost:8000;
}
I have the following yaml in my AWS::ECS::TaskDefinition that works
Command:
- /bin/sh
- -c
- >-
echo 'before: showmigrations --list'
&& python manage.py showmigrations --list
&& echo 'before: showmigrations --plan'
&& python manage.py showmigrations --plan
&& echo 'migrate'
&& python manage.py migrate
&& echo 'after: showmigrations --list'
&& python manage.py showmigrations --list
&& echo 'after: showmigrations --plan'
&& python manage.py showmigrations --plan
Related
I'm looking to create sql views after the django tables are built as the views rely on tables created by Django models.
The problem is that, when trying to run a python script via a Dockerfile CMD calling entrypoint.sh
I get the following issue with the hostname when trying to connect to the postgresql database from the create_views.py
I've tried the following hostnames options: localhost, db, 0.0.0.0, 127.0.0.1 to no avail.
e.g.
psycopg2.OperationalError: connection to server at "0.0.0.0", port 5432 failed: Connection refused
could not translate host name "db" to address: Temporary failure in name resolution
connection to server at "localhost" (127.0.0.1), port 5432 failed: Connection refused
I can't use the containers IP address as everytime you start up docker-compose up you get different IP's for the containers...
docker-compose.yml
services:
app:
container_name: django-mhb-0.3.1
build:
context: .
ports:
- "8000:8000"
volumes:
- ./myproject/:/app/
environment:
- DB_HOST=db
- DB_NAME=${POSTGRES_DB}
- DB_USER=${POSTGRES_USER}
- DB_PWD=${POSTGRES_PASSWORD}
depends_on:
- "postgres"
postgres:
container_name: postgres-mhb-0.1.1
image: postgres:14
volumes:
- postgres_data:/var/lib/postgresql/data/
# The following works. However, it runs before the Dockerfile entrypoint script.
# So in this case its trying to create views before the tables exist.
#- ./myproject/sql/:/docker-entrypoint-initdb.d/
ports:
- "5432:5432"
environment:
- POSTGRES_DB=${POSTGRES_DB}
- POSTGRES_USER=${POSTGRES_USER}
- POSTGRES_PASSWORD=${POSTGRES_PASSWORD}
volumes:
postgres_data:
Docker environment variables are in a .env file in the same directory as the Dockerfile and docker-compose.yml
Django secrets are in secrets.json file in django project directory
Dockerfile
### Dockerfile for Django Applications ###
# Pull Base Image
FROM python:3.9-slim-buster AS myapp
# set work directory
WORKDIR /app
# set environment variables
ENV PYTHONUNBUFFERED 1
ENV PYTHONDONTWRITEBYTECODE 1
# Compiler and OS libraries
RUN apt-get update\
&& apt-get install -y --no-install-recommends build-essential curl libpq-dev \
&& rm -rf /var/lib/apt/lists/* /usr/share/doc /usr/share/man \
&& apt-get clean \
&& useradd --create-home python
# install dependencies
COPY --chown=python:python ./requirements.txt /tmp/requirements.txt
COPY --chown=python:python ./scripts /scripts
ENV PATH="/home/python/.local/bin:$PATH"
RUN pip install --upgrade pip \
&& pip install -r /tmp/requirements.txt \
&& rm -rf /tmp/requirements.txt
USER python
# Section 5- Code and User Setup
ENV PATH="/scripts:$PATH"
CMD ["entrypoint.sh"]
entrypoint.sh
#!/bin/sh
echo "start of entrypoint"
set -e
whoami
pwd
#ls -l
#cd ../app/
#ls -l
python manage.py wait_for_db
python manage.py makemigrations
python manage.py migrate
python manage.py djstripe_sync_models product plan
python manage.py shell < docs/build-sample-data.py
## issue arises running this script ##
python manage.py shell < docs/create_views.py
python manage.py runserver 0.0.0.0:8000
create_views.py
#!/usr/bin/env python
import psycopg2 as db_connect
def get_connection():
try:
return db_connect.connect(
database="devdb",
user="devuser",
password="devpassword",
host="0.0.0.0",
port=5432,
)
except (db_connect.Error, db_connect.OperationalError) as e:
#t_msg = "Database connection error: " + e + "/n SQL: " + s
print('t_msg ',e)
return False
try:
conn = get_connection()
...
I've removed the rest of the script as it's unnecessary
When I run the Django/postgresql outside of docker on local machine localhost works fine as you would expect.
Hoping someone can help, it's doing my head in and I've spent a few days looking for a possible answwer.
Thanks
Thanks to hints from Erik, solved by the following
python manage.py makemigrations --empty yourappname
Then added the following (note cut down for space)
from django.db import migrations
def get_all_items_view(s=None):
s = ""
s += "create or replace view v_all_items_report"
s += " as"
s += " SELECT project.id, project.slug, project.name as p_name,"
...
return s
def get_full_summary_view(s=None):
s = ""
s += "CREATE or replace VIEW v_project_summary"
s += " AS"
s += " SELECT project.id, project.slug, project.name as p_name,
...
return s
class Migration(migrations.Migration):
dependencies = [
('payment', '0002_payment_user'),
]
operations = [
migrations.RunSQL(get_all_items_view()),
migrations.RunSQL(get_full_summary_view()),
migrations.RunSQL(get_invoice_view()),
migrations.RunSQL(get_payment_view()),
]
Note to ensure you list the last table that needs to be created in your dependencies before the views get created. In my case django defaulted all other migration to be dependant on one another in a chain order.
In my dockerfile entrypoint.sh script is where I had the commands to makemigrations, migrate and build some sample data
I am trying to deploy a docker container to Elastic Beanstalk in AWS. I'm repeatedly getting error while doing so and each time the error is related to the ENTRYPOINT that I specified in the dockerrun.aws.json. What am I doing wrong here ?
The webapp uses Django, python3 and keras.
This is my Dockerfile content:
# reference: https://hub.docker.com/_/ubuntu/
FROM ubuntu:18.04
RUN apt-get update && apt-get install \
-y --no-install-recommends python3 python3-virtualenv
# Adds metadata to the image as a key value pair example LABEL
version="1.0"
LABEL maintainer="Amir Ashraff <amir.ashraff#gmail.com>"
##Set environment variables
ENV VIRTUAL_ENV=/opt/venv
RUN python3 -m virtualenv --python=/usr/bin/python3 $VIRTUAL_ENV
ENV PATH="$VIRTUAL_ENV/bin:$PATH"
# Install dependencies:
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
# Open Ports for Web App
EXPOSE 8000
WORKDIR /manage.py
COPY . /manage.py
RUN chmod +x /manage.py
ENTRYPOINT [ "python3" ]
CMD [ "python3", "manage.py runserver 0.0.0.0:8000" ]
And this is the dockerrun.aws.json content:
{
"AWSEBDockerrunVersion": "1",
"Ports": [
{
"ContainerPort": ""
}
],
"Volumes": [
{
"HostDirectory": "/~/aptos",
"ContainerDirectory": "/aptos/diabetes_retinopathy_recognition"
}
],
"Logging": "/aptos/diabetes_recognition",
"Entrypoint": "/opt/venv/bin/python3",
"Command": "python3 manage.py runserver 0.0.0.0:8000"
}
And this is the error from AWS logs:
Docker container quit unexpectedly on Tue Aug 20 13:03:47 UTC 2019:
/opt/venv/bin/python3: can't open file 'python3': [Errno 2] No such file
or directory.
My configuration worked up until yesterday. I have added the nginx NodeJS https redirect extension from AWS. Now, when I try to add a new Environment Variable through the Elastic Beanstalk configuration, I get this error:
[Instance: i-0364b59cca36774a0] Command failed on instance. Return code: 137 Output: + rm -f /etc/nginx/conf.d/00_elastic_beanstalk_proxy.conf + service nginx stop Stopping nginx: /sbin/service: line 66: 27395 Killed env -i PATH="$PATH" TERM="$TERM" "${SERVICEDIR}/${SERVICE}" ${OPTIONS}. Hook /opt/elasticbeanstalk/hooks/configdeploy/post/99_kill_default_nginx.sh failed. For more detail, check /var/log/eb-activity.log using console or EB CLI.
When I look at the eb-activity.log, I see this error:
[2018-02-18T17:24:58.762Z] INFO [13848] - [Configuration update 1.0.61#112/ConfigDeployStage1/ConfigDeployPostHook/99_kill_default_nginx.sh] : Starting activity...
[2018-02-18T17:24:58.939Z] INFO [13848] - [Configuration update 1.0.61#112/ConfigDeployStage1/ConfigDeployPostHook/99_kill_default_nginx.sh] : Activity execution failed, because: + rm -f /etc/nginx/conf.d/00_elastic_beanstalk_proxy.conf
+ service nginx stop
Stopping nginx: /sbin/service: line 66: 14258 Killed env -i PATH="$PATH" TERM="$TERM" "${SERVICEDIR}/${SERVICE}" ${OPTIONS} (ElasticBeanstalk::ExternalInvocationError)
caused by: + rm -f /etc/nginx/conf.d/00_elastic_beanstalk_proxy.conf
+ service nginx stop
Stopping nginx: /sbin/service: line 66: 14258 Killed env -i PATH="$PATH" TERM="$TERM" "${SERVICEDIR}/${SERVICE}" ${OPTIONS} (Executor::NonZeroExitStatus)
What am I doing wrong? And what has changed recently since this worked fine when I changed an Environment Variable a couple months ago.
I had this problem as well and Amazon acknowledged the error in the documentation. This is a working restart script that you can use in your .ebextensions config file.
/opt/elasticbeanstalk/hooks/configdeploy/post/99_kill_default_nginx.sh:
mode: "000755"
owner: root
group: root
content: |
#!/bin/bash -xe
rm -f /etc/nginx/conf.d/00_elastic_beanstalk_proxy.conf
status=`/sbin/status nginx`
if [[ $status = *"start/running"* ]]; then
echo "stopping nginx..."
stop nginx
echo "starting nginx..."
start nginx
else
echo "nginx is not running... starting it..."
start nginx
fi
service nginx stop exits with status 137 (Killed).
Your script starts with: #!/bin/bash -xe
The parameter -e makes the script exit immediately whenever something exits with a non-zero status.
If you want to continue the execution, you need to catch the exit status (137).
/opt/elasticbeanstalk/hooks/configdeploy/post/99_kill_default_nginx.sh:
mode: "000755"
owner: root
group: root
content: |
#!/bin/bash -xe
rm -f /etc/nginx/conf.d/00_elastic_beanstalk_proxy.conf
status=`/sbin/status nginx`
if [[ $status = *"start/running"* ]]; then
set +e
service nginx stop
exitStatus = $?
if [ $exitStatus -ne 0 ] && [ $exitStatus -ne 137 ]
then
exit $exitStatus
fi
set -e
fi
service nginx start
The order of events looks like this to me:
Create a post-deploy hook to delete /etc/nginx/conf.d/00_elastic_beanstalk_proxy.conf
Run a container command to delete /etc/nginx/conf.d/00_elastic_beanstalk_proxy.conf
Run the post-deploy hook, which tries to delete /etc/nginx/conf.d/00_elastic_beanstalk_proxy.conf
So it doesn't seem surprising to me that the post-deploy script fails as the file you are trying to delete probably doesn't exist.
I would try one of two things:
Move the deletion of the temporary conf file from the container command to the 99_kill_default_nginx.sh script, then remove the whole container command section.
Remove the line rm -f /etc/nginx/conf.d/00_elastic_beanstalk_proxy.conf from the 99_kill_default_nginx.sh script.
/sbin/status nginx seems not to work anymore. I updated the script to use service nginx status:
/opt/elasticbeanstalk/hooks/configdeploy/post/99_kill_default_nginx.sh:
mode: "000755"
owner: root
group: root
content: |
#!/bin/bash -xe
rm -f /etc/nginx/conf.d/00_elastic_beanstalk_proxy.conf
status=$(service nginx status)
if [[ "$status" =~ "running" ]]; then
echo "stopping nginx..."
stop nginx
echo "starting nginx..."
start nginx
else
echo "nginx is not running... starting it..."
start nginx
fi
And the faulty script is STILL in amazon's docs... I wonder when they are going to fix it. It's been enough time already
I try to install the latest version of nginx (>= 1.9.5) on a fresh amazon linux to make use of http2. I followed the instructions that are described here -> http://nginx.org/en/linux_packages.html
I created a repo file /etc/yum.repos.d/nginx.repowith this content:
[nginx]
name=nginx repo
baseurl=http://nginx.org/packages/mainline/centos/7/$basearch/
gpgcheck=0
enabled=1
If I run yum update and yum install nginx I get this:
nginx x86_64 1:1.8.1-1.26.amzn1 amzn-main 557 k
It seems that it fetches still from the amzn-main repo. How do I install a newer version of nginx?
-- edit --
I added "priority=10" to the nginx.repo file and now I can install 1.9.15 with yum install nginx with this result:
Loaded plugins: priorities, update-motd, upgrade-helper
Resolving Dependencies
--> Running transaction check
---> Package nginx.x86_64 1:1.9.15-1.el7.ngx will be installed
--> Processing Dependency: systemd for package: 1:nginx-1.9.15-1.el7.ngx.x86_64
--> Processing Dependency: libpcre.so.1()(64bit) for package: 1:nginx-1.9.15-1.el7.ngx.x86_64
--> Finished Dependency Resolution
Error: Package: 1:nginx-1.9.15-1.el7.ngx.x86_64 (nginx)
Requires: libpcre.so.1()(64bit)
Error: Package: 1:nginx-1.9.15-1.el7.ngx.x86_64 (nginx)
Requires: systemd
You could try using --skip-broken to work around the problem
You could try running: rpm -Va --nofiles --nodigest
If you're using AWS Linux2, you have to install nginx from the AWS "Extras Repository". To see a list of the packages available:
# View list of packages to install
amazon-linux-extras list
You'll see a list similar to:
0 ansible2 disabled [ =2.4.2 ]
1 emacs disabled [ =25.3 ]
2 memcached1.5 disabled [ =1.5.1 ]
3 nginx1.12 disabled [ =1.12.2 ]
4 postgresql9.6 disabled [ =9.6.6 ]
5 python3 disabled [ =3.6.2 ]
6 redis4.0 disabled [ =4.0.5 ]
7 R3.4 disabled [ =3.4.3 ]
8 rust1 disabled [ =1.22.1 ]
9 vim disabled [ =8.0 ]
10 golang1.9 disabled [ =1.9.2 ]
11 ruby2.4 disabled [ =2.4.2 ]
12 nano disabled [ =2.9.1 ]
13 php7.2 disabled [ =7.2.0 ]
14 lamp-mariadb10.2-php7.2 disabled [ =10.2.10_7.2.0 ]
Use the amazon-linux-extras install command to install it, like:
sudo amazon-linux-extras install nginx1.12
More details are here: https://aws.amazon.com/amazon-linux-2/faqs/.
At the time of writing, the latest version of nginx available from the AWS yum repo is 1.8.
The best thing to do for now is to build any newer version from source.
The AWS Linux AMI already has the necessary build tools.
For example, based on the Nginx 1.10 (I've assumed you're logged in as the regular ec2-user. Anything needing superuser rights is preceded with sudo)
cd /tmp #so we can clean-up easily
wget http://nginx.org/download/nginx-1.10.0.tar.gz
tar zxvf nginx-1.10.0.tar.gz && rm -f nginx-1.10.0.tar.gz
cd nginx-1.10.0
sudo yum install pcre-devel openssl-devel #required libs, not installed by default
./configure \
--prefix=/etc/nginx \
--conf-path=/etc/nginx/nginx.conf \
--pid-path=/var/run/nginx.pid \
--lock-path=/var/run/nginx.lock \
--with-http_ssl_module \
--with-http_v2_module \
--user=nginx \
--group=nginx
make
sudo make install
sudo groupadd nginx
sudo useradd -M -G nginx nginx
rm -rf nginx-1.10.0
You'll then want a service file, so that you can start/stop nginx, and load it on boot.
Here's one that matches the above config. Put it in /etc/rc.d/init.d/nginx:
#!/bin/sh
#
# nginx - this script starts and stops the nginx daemon
#
# chkconfig: - 85 15
# description: NGINX is an HTTP(S) server, HTTP(S) reverse \
# proxy and IMAP/POP3 proxy server
# processname: nginx
# config: /etc/nginx/nginx.conf
# config: /etc/sysconfig/nginx
# pidfile: /var/run/nginx.pid
# Source function library.
. /etc/rc.d/init.d/functions
# Source networking configuration.
. /etc/sysconfig/network
# Check that networking is up.
[ "$NETWORKING" = "no" ] && exit 0
nginx="/etc/nginx/sbin/nginx"
prog=$(basename $nginx)
NGINX_CONF_FILE="/etc/nginx/nginx.conf"
[ -f /etc/sysconfig/nginx ] && . /etc/sysconfig/nginx
lockfile=/var/run/nginx.lock
make_dirs() {
# make required directories
user=`$nginx -V 2>&1 | grep "configure arguments:" | sed 's/[^*]*--user=\([^ ]*\).*/\1/g' -`
if [ -z "`grep $user /etc/passwd`" ]; then
useradd -M -s /bin/nologin $user
fi
options=`$nginx -V 2>&1 | grep 'configure arguments:'`
for opt in $options; do
if [ `echo $opt | grep '.*-temp-path'` ]; then
value=`echo $opt | cut -d "=" -f 2`
if [ ! -d "$value" ]; then
# echo "creating" $value
mkdir -p $value && chown -R $user $value
fi
fi
done
}
start() {
[ -x $nginx ] || exit 5
[ -f $NGINX_CONF_FILE ] || exit 6
make_dirs
echo -n $"Starting $prog: "
daemon $nginx -c $NGINX_CONF_FILE
retval=$?
echo
[ $retval -eq 0 ] && touch $lockfile
return $retval
}
stop() {
echo -n $"Stopping $prog: "
killproc $prog -QUIT
retval=$?
echo
[ $retval -eq 0 ] && rm -f $lockfile
return $retval
}
restart() {
configtest || return $?
stop
sleep 1
start
}
reload() {
configtest || return $?
echo -n $"Reloading $prog: "
killproc $nginx -HUP
RETVAL=$?
echo
}
force_reload() {
restart
}
configtest() {
$nginx -t -c $NGINX_CONF_FILE
}
rh_status() {
status $prog
}
rh_status_q() {
rh_status >/dev/null 2>&1
}
case "$1" in
start)
rh_status_q && exit 0
$1
;;
stop)
rh_status_q || exit 0
$1
;;
restart|configtest)
$1
;;
reload)
rh_status_q || exit 7
$1
;;
force-reload)
force_reload
;;
status)
rh_status
;;
condrestart|try-restart)
rh_status_q || exit 0
;;
*)
echo $"Usage: $0 {start|stop|status|restart|condrestart|try-restart|reload|force-reload|configtest}"
exit 2
esac
Set the service file to be executable:
sudo chmod 755 /etc/rc.d/init.d/nginx
Now you can start it with:
sudo service nginx start
To load it automatically on boot:
sudo chkconfig nginx on
Finally, don't forget to edit /etc/nginx/nginx.conf to match your requirements and run sudo service nginx reload to refresh the changes.
Note, there is no 1.10 where you're looking. You can see the list here
http://nginx.org/packages/mainline/centos/7/x86_64/RPMS/
After you yum update use yum search nginx to see the different versions you have and choose a specific one:
yum search nginx
on centos 6 gives
nginx.x86_64 : A high performance web server and reverse proxy server
nginx16.x86_64 : A high performance web server and reverse proxy server
nginx18.x86_64 : A high performance web server and reverse proxy server
I have two versions to choose from, 1.6 and 1.8.
You're getting error because those nginx RPMs are built for RHEL7, not Amazon Linux. Amazon Linux is a weird hybrid of RHEL6, RHEL7, and Fedora. You should contact Amazon and ask them to create a proper nginx19 RPM specifically built for their distro.
I have VirtualBox VM that runs a server that can be accessed via localhost and forwarded ports.
I need to run some shells scripts and implement some business logic based on the results.
I tried following command as example:
VBoxManage guestcontrol <UUID> exec --image /bin/sh --username <su username> --password <su password> --wait-exit --wait-stdout --wait-stderr -- "[ -d /<server_folder>/ ] && echo "OK" || echo "Server is not installed""
but I'm getting the error:
/bin/sh: [ -d <server_folder> ] && echo : No such file or directory
What is wrong with the syntax above?
First make sure that VBoxManage.exe is in your path!
Secondly you have to be carefull with your quotations. You used:
"[ -d /<server_folder>/ ] && echo "OK" || echo "Server is not installed""
you have to use singel quotaions for the outermost quotation:
'[ -d /<server_folder>/ ] && echo "OK" || echo "Server is not installed"'
Finally you have to add a -c in front of your arguments (to call /bin/sh -c '...').
The complete command:
VBoxManage guestcontrol <UUID> exec --image /bin/sh --username <su username> --password <su password> --wait-exit --wait-stdout --wait-stderr -- -c '[ -d /<server_folder>/ ] && echo "OK" || echo "Server is not installed"'