Docker env variables not set while log via shell - django

In my settings file i am getting env variables like this
'NAME': os.environ['PG_DBNAME'], # Database
I am setting in docker file like this
-e PG_DBNAME= "mapp"
Now
The web app work fine
If i log into shell via docker exec ... bash then env variables are also set
But if i log in via ipaddress and port number from ssh client then i am able to login but env variables are not set

As commented in issue 2569:
This is expected. SSH wipes out the environment as part of the login process.
One way to work around it is to dump the environment variables in /etc/environment (e.g. env | grep _ >> /etc/environment) before starting Supervisor.
Further "login processes" should source this file, and tada! There is your environment.
That env | grep _ >> /etc/environment could be part of a default run script associated (through ENTRYPOINT or CMD) to your image.
Daniel A.A. Pelsmaeker suggests jenkinsci/docker-ssh-agent issue 33 for an approach that selects and sets all environment variables excluding a specific denylist:
For my own uses I changed that line to the following:
env | egrep -v "^(HOME=|USER=|MAIL=|LC_ALL=|LS_COLORS=|LANG=|HOSTNAME=|PWD=|TERM=|SHLVL=|LANGUAGE=|_=)" >> /etc/environment
This takes all environment variables, except those listed, and appends then to /etc/environment, overriding any previously defined there.

I also had the exact same problem. I found the example on docs.docker.com appending variables by echo'ing to /etc/profile not the nicest way to do that. So here is my solution:
Dockerbuild:
I execute the docker build by the following command which also fetches the http_proxy, https_proxy and no_proxy variables from the
current shell session. The variables are passed as agruments with the --build-arg option.
[root#localhost dock-centOS]# docker build
--build-arg http_proxy="{{ lookup('env', 'http_proxy')}}"
--build-arg https_proxy="{{ lookup('env', 'https_proxy')}}"
--build-arg no_proxy="{{ lookup('env', 'no_proxy')}}"
-t my_pv_repo:centOS-with-sshd .
Dockerfile:
I use the following dockerfile snippet for setting the enviroment variables for all users. The ARG command is used instead of
ENV because i don't want docker to persist my variables in the image. The ARG variable is only available during the docker build.
The RUN command creates a bash script which is placed in the /etc/profile.d directory. During start-up of the container
/etc/profile script is run and sources all readable files in the /etc/profile.d directory.
FROM centos:7.3.1611
ARG http_proxy=$http_proxy
ARG https_proxy=$https_proxy
ARG no_proxy=$no_proxy
ARG JAVA_HOME=/usr/lib/jvm/jdk1.6.0_45
ARG DOMAIN_HOME=/home/oracle/w001/D1/app/user_projects/domains/fancy_app_domain
ARG PATH=$PATH:/usr/lib/jvm/jdk1.6.0_45/bin
ARG XAUTHORITY=~/.Xauthority
RUN shebang='#!/usr/bin/env bash'; \
env_vars="export http_proxy=${http_proxy} https_proxy=${https_proxy} no_proxy=${no_proxy}"; \
env_vars+=' JAVA_HOME=/usr/lib/jvm/jdk1.6.0_45 DOMAIN_HOME=/home/oracle/w001/D1/app/user_projects/domains/fancy_app_domain'; \
env_vars+=" PATH=${PATH}:/usr/lib/jvm/jdk1.6.0_45/bin XAUTHORITY=${XAUTHORITY}"; \
echo $shebang$'\n'$env_vars > /etc/profile.d/env_vars.sh
Test result: Well lets hit the cli to check if our environment variables are available during a ssh session.
[root#localhost vagrant]# docker exec -u root -it centOS-container bash
[root#33e7efab489c /]#
[root#33e7efab489c /]#
[root#33e7efab489c /]# cat /etc/profile.d/env_vars.sh
#!/usr/bin/env bash
export http_proxy=http://10.0.2.2:3128 https_proxy=http://10.0.2.2:3128 no_proxy=localhost,127.0.0.1 JAVA_HOME=/usr/lib/jvm/jdk1.6.0_45 DOMAIN_HOME=/home/oracle/w001/D1/app/user_projects/domains/fancy_app_domain PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/lib/jvm/jdk1.6.0_45/bin XAUTHORITY=~/.Xauthority
[root#33e7efab489c /]#
[root#33e7efab489c /]#
[root#33e7efab489c /]# printenv
HOSTNAME=33e7efab489c
TERM=xterm
http_proxy=http://10.0.2.2:3128
LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=01;05;37;41:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.axv=01;35:*.anx=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=01;36:*.au=01;36:*.flac=01;36:*.mid=01;36:*.midi=01;36:*.mka=01;36:*.mp3=01;36:*.mpc=01;36:*.ogg=01;36:*.ra=01;36:*.wav=01;36:*.axa=01;36:*.oga=01;36:*.spx=01;36:*.xspf=01;36:
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/lib/jvm/jdk1.6.0_45/bin
DOMAIN_HOME=/home/oracle/w001/D1/app/user_projects/domains/fancy_app_domain
PWD=/
JAVA_HOME=/usr/lib/jvm/jdk1.6.0_45
LANG=en_US.UTF-8
https_proxy=http://10.0.2.2:3128
SHLVL=1
HOME=/root
no_proxy=localhost,127.0.0.1
XAUTHORITY=/root/.Xauthority
_=/usr/bin/printenv
[root#33e7efab489c /]#
[root#33e7efab489c /]#
[root#33e7efab489c /]# exit
[root#localhost vagrant]# exit
[vagrant#localhost ~]$ logout
Connection to 127.0.0.1 closed.
me#my-mac$ ssh -X root#localhost -p 7022 -o UserKnownHostsFile=/dev/null -o IdentityFile=/development/workspace/supercalifragilisticexpialidocious-app/.vagrant/machines/default/virtualbox/private_key
The authenticity of host '[localhost]:7022 ([127.0.0.1]:7022)' can't be established.
ECDSA key fingerprint is SHA256:dTd/vsmPTbrA3kPeIfArZMFEgfdlgjGHwMgE3Z5BgBc.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '[localhost]:7022' (ECDSA) to the list of known hosts.
/usr/bin/xauth: file /root/.Xauthority does not exist
[root#33e7efab489c ~]# su - oracle
bash-4.2$
bash-4.2$
bash-4.2$ printenv
HOSTNAME=33e7efab489c
SHELL=/bin/bash
TERM=xterm-256color
HISTSIZE=1000
http_proxy=http://10.0.2.2:3128
USER=oracle
LS_COLORS=rs=0:di=38;5;27:ln=38;5;51:mh=44;38;5;15:pi=40;38;5;11:so=38;5;13:do=38;5;5:bd=48;5;232;38;5;11:cd=48;5;232;38;5;3:or=48;5;232;38;5;9:mi=05;48;5;232;38;5;15:su=48;5;196;38;5;15:sg=48;5;11;38;5;16:ca=48;5;196;38;5;226:tw=48;5;10;38;5;16:ow=48;5;10;38;5;21:st=48;5;21;38;5;15:ex=38;5;34:*.tar=38;5;9:*.tgz=38;5;9:*.arc=38;5;9:*.arj=38;5;9:*.taz=38;5;9:*.lha=38;5;9:*.lz4=38;5;9:*.lzh=38;5;9:*.lzma=38;5;9:*.tlz=38;5;9:*.txz=38;5;9:*.tzo=38;5;9:*.t7z=38;5;9:*.zip=38;5;9:*.z=38;5;9:*.Z=38;5;9:*.dz=38;5;9:*.gz=38;5;9:*.lrz=38;5;9:*.lz=38;5;9:*.lzo=38;5;9:*.xz=38;5;9:*.bz2=38;5;9:*.bz=38;5;9:*.tbz=38;5;9:*.tbz2=38;5;9:*.tz=38;5;9:*.deb=38;5;9:*.rpm=38;5;9:*.jar=38;5;9:*.war=38;5;9:*.ear=38;5;9:*.sar=38;5;9:*.rar=38;5;9:*.alz=38;5;9:*.ace=38;5;9:*.zoo=38;5;9:*.cpio=38;5;9:*.7z=38;5;9:*.rz=38;5;9:*.cab=38;5;9:*.jpg=38;5;13:*.jpeg=38;5;13:*.gif=38;5;13:*.bmp=38;5;13:*.pbm=38;5;13:*.pgm=38;5;13:*.ppm=38;5;13:*.tga=38;5;13:*.xbm=38;5;13:*.xpm=38;5;13:*.tif=38;5;13:*.tiff=38;5;13:*.png=38;5;13:*.svg=38;5;13:*.svgz=38;5;13:*.mng=38;5;13:*.pcx=38;5;13:*.mov=38;5;13:*.mpg=38;5;13:*.mpeg=38;5;13:*.m2v=38;5;13:*.mkv=38;5;13:*.webm=38;5;13:*.ogm=38;5;13:*.mp4=38;5;13:*.m4v=38;5;13:*.mp4v=38;5;13:*.vob=38;5;13:*.qt=38;5;13:*.nuv=38;5;13:*.wmv=38;5;13:*.asf=38;5;13:*.rm=38;5;13:*.rmvb=38;5;13:*.flc=38;5;13:*.avi=38;5;13:*.fli=38;5;13:*.flv=38;5;13:*.gl=38;5;13:*.dl=38;5;13:*.xcf=38;5;13:*.xwd=38;5;13:*.yuv=38;5;13:*.cgm=38;5;13:*.emf=38;5;13:*.axv=38;5;13:*.anx=38;5;13:*.ogv=38;5;13:*.ogx=38;5;13:*.aac=38;5;45:*.au=38;5;45:*.flac=38;5;45:*.mid=38;5;45:*.midi=38;5;45:*.mka=38;5;45:*.mp3=38;5;45:*.mpc=38;5;45:*.ogg=38;5;45:*.ra=38;5;45:*.wav=38;5;45:*.axa=38;5;45:*.oga=38;5;45:*.spx=38;5;45:*.xspf=38;5;45:
MAIL=/var/spool/mail/oracle
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/lib/jvm/jdk1.6.0_45/bin
DOMAIN_HOME=/home/oracle/w001/D1/app/user_projects/domains/fancy_app_domain
PWD=/home/oracle
JAVA_HOME=/usr/lib/jvm/jdk1.6.0_45
LANG=en_US.UTF-8
https_proxy=http://10.0.2.2:3128
HISTCONTROL=ignoredups
SHLVL=1
HOME=/home/oracle
no_proxy=localhost,127.0.0.1
LOGNAME=oracle
XAUTHORITY=/home/oracle/.Xauthority
_=/usr/bin/printenv

Related

Django channels elastic beanstalk (linux 2) hooks postdeploy: -b command not found

For deployment of django channels on elastic beanstalk (linux 2 AMI) I tried implementing this blog ,it required creation of ./platform/hooks/postdeploy/ and in it two files 01_set_env.sh and 02_run_supervisor_daemon.sh, on eb deploy it fails and on checking the eb-engine.logs this error shows up.
2021/09/28 05:05:44.382229 [ERROR] An error occurred during execution of command [app-deploy] - [RunAppDeployPostDeployHooks]. Stop running the command. Error: Command .platform/hooks/postdeploy/02_run_supervisor_daemon.sh failed with error exit status 2. Stderr:.platform/hooks/postdeploy/02_run_supervisor_daemon.sh: line 13: -b: command not found
02_run_supervisor_daemon.sh
#!/bin/bash
# Get system environment variables
systemenv=`cat /opt/elasticbeanstalk/deployment/custom_env_var | tr '
' ',' | sed 's/%/%%/g' | sed 's/export //g' | sed 's/$PATH/%(ENV_PATH)s/g' | sed 's/:$PYTHONPATH//g' | sed 's/$LD_LIBRARY_PATH//g'`
systemenv=${systemenv%?}
systemenv=`echo $systemenv | sed 's/,/",/g' | sed 's/=/="/g'`
systemenv="$systemenv""
# Create daemon configuration script
daemonconf="[program:daphne]
command=daphne -b :: -p 5000 backend.asgi:application
directory=/var/app
user=ec2-user
numprocs=1
stdout_logfile=/var/log/stdout_daphne.log
stderr_logfile=/var/log/stderr_daphne.log
autostart=true
autorestart=true
startsecs=10
; Need to wait for currently executing tasks to finish at shutdown.
; Increase this if you have very long running tasks.
stopwaitsecs = 600
; When resorting to send SIGKILL to the program to terminate it
; send SIGKILL to its whole process group instead,
; taking care of its children as well.
killasgroup=true
environment=$systemenv"
# Create the Supervisor conf script
echo "$daemonconf" | sudo tee /etc/supervisor/conf.d/daemon.conf
# Reread the Supervisor config
supervisorctl reread
# Update Supervisor in cache without restarting all services
supervisorctl update
# Start/restart processes through Supervisor
supervisorctl restart daphne
01_set_env.sh
#!/bin/bash
#Create a copy of the environment variable file.
cp /opt/elasticbeanstalk/deployment/env /opt/elasticbeanstalk/deployment/custom_env_var
#Set permissions to the custom_env_var file so this file can be accessed by any user on the instance. You can restrict permissions as per your requirements.
chmod 644 /opt/elasticbeanstalk/deployment/custom_env_var
#Remove duplicate files upon deployment.
rm -f /opt/elasticbeanstalk/deployment/*.bak
Would be great if you could suggest some resources for the deployment of Django channels on ebs(linux 2 as most blogs are for linux1) as well.

ssh tunnel script hangs forever on beanstalk deployment

I'm attempting to create a ssh tunnel, when deploying an application to aws beanstalk. I want to put the tunnel as a background process, that is always connected on application deploy. The script is hanging forever on the deployment and I can't see why.
"/home/ec2-user/eclair-ssh-tunnel.sh":
mode: "000500" # u+rx
owner: root
group: root
content: |
cd /root
eval $(ssh-agent -s)
DISPLAY=":0.0" SSH_ASKPASS="./askpass_script" ssh-add eclair-test-key </dev/null
# we want this command to keep running in the backgriund
# so we add & at then end
nohup ssh -L 48682:localhost:8080 ubuntu#[host...] -N &
and here is the output I'm getting from /var/log/eb-activity.log:
[2019-06-14T14:53:23.268Z] INFO [15615] - [Application update suredbits-api-root-0.37.0-testnet-ssh-tunnel-fix-port-9#30/AppDeployStage1/AppDeployPostHook/01_eclair-ssh-tunnel.sh] : Starting activity...
The ssh tunnel is spawned, and I can find it by doing:
[ec2-user#ip-172-31-25-154 ~]$ ps aux | grep 48682
root 16047 0.0 0.0 175560 6704 ? S 14:53 0:00 ssh -L 48682:localhost:8080 ubuntu#ec2-34-221-186-19.us-west-2.compute.amazonaws.com -N
If I kill that process, the deployment continues as expected, which indicates that the bug is in the tunnel script. I can't seem to find out where though.
You need to add -n option to ssh when run it in background to avoid reading from stdin.

"Illegal variable name" error when running "docker stop $(docker ps -a -q)"

I am trying to issue command docker stop $(docker ps -a -q) over ssh, using Python's paramiko package
stdin,stdout,stderr = ssh_client.exec_command('docker stop $(docker ps -a -q)');
print stderr.readlines()
I get an error Illegal variable name. I tried putting in double quote (""), escape the $, (, ) ... but running out of ideas
Sounds like you're using csh as the login shell on the ssh server. Csh does not support $(...) so try
ssh_client.exec_command('docker stop `docker ps -a -q` ');
Csh example:
# echo $(echo foo)
Illegal variable name.
#

Unable to set ASG instance hostnames using Troposphere/CloudFormation UserData

Here's the relevant part of my Troposphere file:
LaunchConfiguration = t.add_resource(LaunchConfiguration(
"LaunchConfigA",
ImageId=UBUNTU_IMG,
SecurityGroups=[Ref(SecurityGroup)],
InstanceType="m3.medium",
UserData=Base64(Join('', [
"#cloud-boothook\n",
"#!/bin/bash\n",
"sudo hostname test\n",
"sudo sh -c 'echo test > /etc/hostname'\n",
"sudo sh -c 'echo 127.0.0.1 test >> /etc/hosts'\n",
"sudo touch /var/log/TESTING\n"
])),
))
AutoScalingGroupA = t.add_resource(AutoScalingGroup(
"GroupA",
AvailabilityZones=GetAZs(Ref(AWS_REGION)),
LaunchConfigurationName=Ref(LaunchConfiguration),
MinSize="1",
DesiredCapacity="2",
MaxSize="2",
))
When I create a brand new CloudFormation stack from this template, the hostnames on the instances look like ip-172-XXX-XXX-XXX, the default.
I am certain that the script is running, because of my TESTING file:
atrose#ip-172-31-32-40:~$ ls -la /var/log/TESTING
-rw-r--r-- 1 root root 0 Jul 14 20:10 /var/log/TESTING
If I run the script manually, the hostname is properly set. Like so:
atrose#ip-172-31-32-40:~$ hostname
ip-172-31-32-40
atrose#ip-172-31-32-40:~$ sudo cat /var/lib/cloud/instance/user-data.txt
#cloud-boothook
#!/bin/bash
sudo hostname test
sudo sh -c 'echo test > /etc/hostname'
sudo sh -c 'echo 127.0.0.1 test >> /etc/hosts'
atrose#ip-172-31-32-40:~$ sudo bash /var/lib/cloud/instance/user-data.txt
atrose#ip-172-31-32-40:~$ hostname
test
How can I set hostnames on instances when they first boot into an ASG?
It looks like you're using an Ubuntu AMI, which means CloudInit should have a hostname parameter built into it, and you shouldn't need a shell script to do what you want. I'm going to guess that cloudinit itself is colliding with your script. You should check this out:
http://bazaar.launchpad.net/~cloud-init-dev/cloud-init/trunk/view/head:/doc/examples/cloud-config.txt#L540
Let me know if you have any questions about how to use that. Thanks!

sudo apache2 stop doesn't work

not sure why but when I run:
ubuntu#ip-10-46-206-16:/etc/init.d$ sudo apache2 stop
Usage: apache2 [-D name] [-d directory] [-f file]
[-C "directive"] [-c "directive"]
[-k start|restart|graceful|graceful-stop|stop]
[-v] [-V] [-h] [-l] [-L] [-t] [-S] [-X]
Options:
-D name : define a name for use in <IfDefine name> directives
-d directory : specify an alternate initial ServerRoot
-f file : specify an alternate ServerConfigFile
-C "directive" : process directive before reading config files
-c "directive" : process directive after reading config files
-e level : show startup errors of level (see LogLevel)
-E file : log startup errors to file
-v : show version number
-V : show compile settings
-h : list available command line options (this page)
-l : list compiled in modules
-L : list available configuration directives
-t -D DUMP_VHOSTS : show parsed settings (currently only vhost settings)
-S : a synonym for -t -D DUMP_VHOSTS
-t -D DUMP_MODULES : show all loaded modules
-M : a synonym for -t -D DUMP_MODULES
-t : run syntax check for config files
-X : debug mode (only one worker, do not detach)
it doesn't seem to stop the server. I still trying to ping the ip and it's returning the default page. Is there a reason why?
Try:
sudo service apache2 stop
See here:
https://help.ubuntu.com/community/ApacheMySQLPHP
The older way would be:
sudo /etc/init.d/apache2 stop
Note that when you do sudo apache2 stop, you are running apache2 from you PATH, not from the current folder (usually, . is not in the PATH). Try sudo ./apache2 stop for that.
See here:
http://www.cyberciti.biz/faq/ubuntu-linux-start-restart-stop-apache-web-server/
This is not a programming question. I guess you have to look for answers here http://serverfault.com
You have to type full path to apache2 init script. Example:
sudo /etc/init.d/apache2 stop
check ubuntu#ip-10-46-206-16:/etc/init.d$ ./apache2 stop
this will slove your problem
One thing is missing in the command, I encountered the same issue:
sudo apache2 stop
Should be
sudo service apache2 stop
Hope no one search too far for missing service part of the command :)