Gunicorn can't access lets encrypt files - django

(venv) ubuntu#ip-172-31-6-77:~/redrebelgames_python$ gunicorn redrebelgames_python.wsgi:application
[2021-11-25 20:01:09 +0000] [3758] [INFO] Starting gunicorn 20.1.0
Traceback (most recent call last):
File "/home/ubuntu/redrebelgames_python/venv/bin/gunicorn", line 8, in <module>
sys.exit(run())
File "/home/ubuntu/redrebelgames_python/venv/lib/python3.8/site-packages/gunicorn/app/wsgiapp.py", line 67, in run
WSGIApplication("%(prog)s [OPTIONS] [APP_MODULE]").run()
File "/home/ubuntu/redrebelgames_python/venv/lib/python3.8/site-packages/gunicorn/app/base.py", line 231, in run
super().run()
File "/home/ubuntu/redrebelgames_python/venv/lib/python3.8/site-packages/gunicorn/app/base.py", line 72, in run
Arbiter(self).run()
File "/home/ubuntu/redrebelgames_python/venv/lib/python3.8/site-packages/gunicorn/arbiter.py", line 198, in run
self.start()
File "/home/ubuntu/redrebelgames_python/venv/lib/python3.8/site-packages/gunicorn/arbiter.py", line 155, in start
self.LISTENERS = sock.create_sockets(self.cfg, self.log, fds)
File "/home/ubuntu/redrebelgames_python/venv/lib/python3.8/site-packages/gunicorn/sock.py", line 162, in create_sockets
raise ValueError('certfile "%s" does not exist' % conf.certfile)
ValueError: certfile "/etc/letsencrypt/live/api.redrebelgames.com/cert.pem" does not exist
How do I allow gunicorn to access these files? For some reason it's not working and simply changing the chmod permissions won't work because certbot will eventually change them back.

The certbot files are owned by one identity (typically root). You are running Gunicorn under a different identity. The key is to grant permission to the Gunicorn identity to read the Let's Encrypt files. Typically you can add the Gunicorn username to the Let's Encrypt identity group name and make the files readable by the group.
Example command:
sudo usermod -a -G groupname username
The identity username must re-login after changing group membership. It is simpler to just restart the system.
Another method (not recommended) is to run Gunicorn as a privileged process. That has security risks.

Related

How can I send E-Mails through python django frequently with a cronjob?

I try to run a function in django every 5 minutes. For this I user the django-crontab package. The function which should run, checks for some conditions in the database and if they are met, sends an e-mail to the user of the app.
I have django 4 running on my linux ubuntu 20.04 server. I added the cronjob via python3 manage.py crontab add (in activated virtual environment). But then I wondered why the cronjob is not running. I tried to execute the job by hand and it worked.
I think the problem boils down to this: When I'm in the activated virtual environment and run the crontab with "python3 manage.py crontab run " it works. But when I run it outside of the virtual environment I get the following error:
Failed to complete cronjob at ('*/5 * * * *', 'evaluation_tool.scripts.cron.send_mail_if_classeval_ended')
Traceback (most recent call last):
File "/var/www/amadeus/lib/python3.10/site-packages/django_crontab/crontab.py", line 145, in run_job
func(*job_args, **job_kwargs)
File "/var/www/amadeus/evaluation_tool/scripts/cron.py", line 12, in send_mail_if_classeval_ended
send_mail_time_over_class_evaluation(class_evaluation=class_evaluation.pk,
File "/var/www/amadeus/evaluation_tool/scripts/email_handler.py", line 122, in send_mail_time_over_class_evaluation
send_falko_mail("AMADEUS Evaluation abgeschlossen", message, to_email_address)
File "/var/www/amadeus/evaluation_tool/scripts/email_handler.py", line 34, in send_falko_mail
msg.send(fail_silently=False)
File "/var/www/amadeus/lib/python3.10/site-packages/django/core/mail/message.py", line 298, in send
return self.get_connection(fail_silently).send_messages([self])
File "/var/www/amadeus/lib/python3.10/site-packages/django/core/mail/backends/smtp.py", line 124, in send_messages
new_conn_created = self.open()
File "/var/www/amadeus/lib/python3.10/site-packages/django/core/mail/backends/smtp.py", line 87, in open
self.connection.starttls(
File "/usr/lib/python3.10/smtplib.py", line 769, in starttls
self.ehlo_or_helo_if_needed()
File "/usr/lib/python3.10/smtplib.py", line 611, in ehlo_or_helo_if_needed
if not (200 <= self.ehlo()[0] <= 299):
File "/usr/lib/python3.10/smtplib.py", line 451, in ehlo
self.putcmd(self.ehlo_msg, name or self.local_hostname)
File "/usr/lib/python3.10/smtplib.py", line 378, in putcmd
self.send(f'{s}{CRLF}')
File "/usr/lib/python3.10/smtplib.py", line 365, in send
raise SMTPServerDisconnected('please run connect() first')
smtplib.SMTPServerDisconnected: please run connect() first
Why is the connection to the SMTP Server working in the activated virtual environment but not outside, which is the desired way using a cronjob?
I found the problem.
I set certain environment variables for SMTP Host, User and Password. But crontab cannot access them. So there are two possibilities:
Add them directly to the head of cron file via crontab -e
EMAIL_HOST=XXX
EMAIL_HOST_USER=XXX
EMAIL_PASS=XXX
Add them via a django-crontab argument in settings.py:
CRONTAB_COMMAND_PREFIX = 'EMAIL_HOST=XXX\nEMAIL_HOST_USER=XXX\nEMAIL_PASS=XXX'

Postgres to Ubuntu Docker container linking not working

[EDIT]: I'm able to reproduce this without any of my own custom code. I just created a fresh Linode image running Ubuntu 14.04 and installed Docker according to the steps on Docker's website.
I then ran:
docker run -d --name db postgres
and can see it running:
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
9d335e8fc70b postgres "/docker-entrypoint. 7 minutes ago Up 7 minutes 5432/tcp db
Then I start an interactive Ubuntu container, linked to db, and try to test the link via nc and curl:
$docker run -it --link db ubuntu /bin/bash
root#eb02f4e7b89e:/# apt-get install curl
...
root#eb02f4e7b89e:/# curl http://$DB_PORT_5432_TCP_ADDR:$DB_PORT_5432_TCP_PORT/
curl: (7) Failed to connect to 172.17.0.2 port 5432: Connection timed out
What am I missing?
I'm trying to link a postgres container with an app container running Django but it doesn't seem to be linking properly.
The commands to start the containers were:
$ docker run -d --name db postgres
$ docker run -d --name web --link db -p 8000:80 test_image
Both containers appear to be running fine:
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
5838047eb14c test_image "/test/.docker/st 40 minutes ago Up 40 minutes 0.0.0.0:8000->80/tcp web
d2d6754430a2 postgres "/docker-entrypoint. 44 minutes ago Up 44 minutes 5432/tcp db
And seem to be correctly linked:
$ docker inspect -f "{{ .HostConfig.Links }}" web
[/db:/web/db]
However, when I try to run "python manage.py migrate" in the web container it doesn't seem to be able to connect to the postgres container:
# python manage.py migrate
Traceback (most recent call last):
File "manage.py", line 10, in <module>
execute_from_command_line(sys.argv)
File "/usr/local/lib/python2.7/dist-packages/django/core/management/__init__.py", line 338, in execute_from_command_line
utility.execute()
File "/usr/local/lib/python2.7/dist-packages/django/core/management/__init__.py", line 330, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/usr/local/lib/python2.7/dist-packages/django/core/management/base.py", line 390, in run_from_argv
self.execute(*args, **cmd_options)
File "/usr/local/lib/python2.7/dist-packages/django/core/management/base.py", line 441, in execute
output = self.handle(*args, **options)
File "/usr/local/lib/python2.7/dist-packages/django/core/management/commands/migrate.py", line 93, in handle
executor = MigrationExecutor(connection, self.migration_progress_callback)
File "/usr/local/lib/python2.7/dist-packages/django/db/migrations/executor.py", line 19, in __init__
self.loader = MigrationLoader(self.connection)
File "/usr/local/lib/python2.7/dist-packages/django/db/migrations/loader.py", line 47, in __init__
self.build_graph()
File "/usr/local/lib/python2.7/dist-packages/django/db/migrations/loader.py", line 180, in build_graph
self.applied_migrations = recorder.applied_migrations()
File "/usr/local/lib/python2.7/dist-packages/django/db/migrations/recorder.py", line 59, in applied_migrations
self.ensure_schema()
File "/usr/local/lib/python2.7/dist-packages/django/db/migrations/recorder.py", line 49, in ensure_schema
if self.Migration._meta.db_table in self.connection.introspection.table_names(self.connection.cursor()):
File "/usr/local/lib/python2.7/dist-packages/django/db/backends/base/base.py", line 164, in cursor
cursor = self.make_cursor(self._cursor())
File "/usr/local/lib/python2.7/dist-packages/django/db/backends/base/base.py", line 135, in _cursor
self.ensure_connection()
File "/usr/local/lib/python2.7/dist-packages/django/db/backends/base/base.py", line 130, in ensure_connection
self.connect()
File "/usr/local/lib/python2.7/dist-packages/django/db/utils.py", line 97, in __exit__
six.reraise(dj_exc_type, dj_exc_value, traceback)
File "/usr/local/lib/python2.7/dist-packages/django/db/backends/base/base.py", line 130, in ensure_connection
self.connect()
File "/usr/local/lib/python2.7/dist-packages/django/db/backends/base/base.py", line 119, in connect
self.connection = self.get_new_connection(conn_params)
File "/usr/local/lib/python2.7/dist-packages/django/db/backends/postgresql_psycopg2/base.py", line 172, in get_new_connection
connection = Database.connect(**conn_params)
File "/usr/lib/python2.7/dist-packages/psycopg2/__init__.py", line 179, in connect
connection_factory=connection_factory, async=async)
django.db.utils.OperationalError: could not connect to server: Connection timed out
Is the server running on host "db" (172.17.0.42) and accepting
TCP/IP connections on port 5432?
I'm able to connect to the postgres container directly:
$ docker exec -it db bash
root#d2d6754430a2:/# cat /etc/hosts
172.17.0.42 d2d6754430a2
127.0.0.1 localhost
::1 localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
root#d2d6754430a2:/# su postgres
$ psql
psql (9.4.4)
Type "help" for help.
postgres=#
But not from inside the web container:
# curl http://$DB_PORT_5432_TCP_ADDR:$DB_PORT_5432_TCP_PORT/
curl: (7) Failed to connect to 172.17.0.42 port 5432: Connection timed out
My host machine is running Ubuntu 14.04.
Any ideas what I'm missing?
Check that your postgresql.conf has listen_addresses = '*'
The default is listen_addresses = 'localhost', which would behave in the manner you describe.
Edit:
This works for me, does it work for you?
$ docker pull postgres
$ docker pull django
$ docker run -d --name db -d postgres
$ docker run -it --link db:db django bash
root#11c767bd3d09:/# psql -h db -U postgres
psql (9.4.3, server 9.4.4)
Type "help" for help.
postgres=#
Edit (Docker 1.7.1 output)
$ docker rm $(docker ps -a -q)
$ docker images
REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE
django latest 29755dd6751b 3 days ago 434.5 MB
postgres latest f33438ff9aef 3 days ago 265.5 MB
$ docker -v
Docker version 1.7.1, build 786b29d
$ docker run -d --name db -d postgres
$ docker run -it --link db:db django bash
root#11c767bd3d09:/# psql -h db -U postgres
psql (9.4.3, server 9.4.4)
Type "help" for help.
postgres=#

Apache WSGI permission error using django logging on production server

Here is some software information
Django 1.8.1
Apache2
Fedora 21
error_log output
mod_wsgi (pid=8272): Target WSGI script '/var/www/anime/anime/wsgi.py' cannot be loaded as Python module.
mod_wsgi (pid=8272): Exception occurred processing WSGI script '/var/www/anime/anime/wsgi.py'.
Traceback (most recent call last):
File "/usr/lib64/python3.4/logging/config.py", line 557, in configure
handler = self.configure_handler(handlers[name])
File "/usr/lib64/python3.4/logging/config.py", line 725, in configure_handler
result = factory(**kwargs)
File "/usr/lib64/python3.4/logging/__init__.py", line 999, in __init__
StreamHandler.__init__(self, self._open())
File "/usr/lib64/python3.4/logging/__init__.py", line 1023, in _open
return open(self.baseFilename, self.mode, encoding=self.encoding)
PermissionError: [Errno 13] Permission denied: '/var/www/anime/log/info.log'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/www/anime/anime/wsgi.py", line 16, in <module>
application = get_wsgi_application()
File "/opt/virtualenvs/django_project/lib64/python3.4/site-packages/django/core/wsgi.py", line 14, in get_wsgi_application
django.setup()
File "/opt/virtualenvs/django_project/lib64/python3.4/site-packages/django/__init__.py", line 17, in setup
configure_logging(settings.LOGGING_CONFIG, settings.LOGGING)
File "/opt/virtualenvs/django_project/lib64/python3.4/site-packages/django/utils/log.py", line 86, in configure_logging
logging_config_func(logging_settings)
File "/usr/lib64/python3.4/logging/config.py", line 789, in dictConfig
dictConfigClass(config).configure()
File "/usr/lib64/python3.4/logging/config.py", line 565, in configure
'%r: %s' % (name, e))
ValueError: Unable to configure handler 'file': [Errno 13] Permission denied: '/var/www/anime/log/info.log'
Here is the permission mask on the file
drwxrwxrwx. 2 apache apache 21 May 28 15:22 .
drwxr-xr-x. 6 apache apache 90 May 28 14:53 ..
-rwxrwxrwx. 1 apache apache 0 May 28 15:22 info.log
I have already searched SOF on all the possible solutions and none of them works. Therefore I suspect that it has something to do with SELinux setting? If it is, can someone tell me which flag do i need to set to true?
After reading up on SELinux, I have figured out the solution for this permission error. And I hope that it will help the others who have encountered similar situation when deploying on production server under RHEL linux.
Basically running the command ls -Z shows the following
drwxr-xr-x. apache apache unconfined_u:object_r:httpd_sys_content_t:s0 log
the folder is labelled with httpd_sys_content_t which does not allow httpd to have write access the the folder. Therefore we need to change this label to httpd_sys_rw_content_t
Firstly, we need to add an entry to the fcontext to inform SELinux what is the default label for files, that will be created, in this folder.
sudo semanage fcontext -a -t httpd_sys_rw_content_t "/path/to/directory(/.*)?"
This will add an entry to the fcontext file (/etc/selinux/targeted/contexts/files/file_contexts.local)
Next we need to update all the labels of the files in the folder using restorecon.
sudo restorecon -R -v /path/to/directory
And now the permission error related to the django logging will be gone from the httpd error_log =)

"password authentication failed for user" after push to Heroku

I followed the heroku documentation to install a django app and at first it worked fine. After a day I pushed some changes to the server. After that, I was not able to access the app at all: FATAL: password authentication failed for user "drjstoymyqyarj"
I can not even sync the db anymore:
$ heroku run python manage.py syncdb
Running `python manage.py syncdb` attached to terminal... up, run.1
Traceback (most recent call last):
File "manage.py", line 10, in <module>
execute_from_command_line(sys.argv)
File "/app/.heroku/venv/lib/python2.7/site-packages/django/core/management/__init__.py", line 443, in execute_from_command_line
utility.execute()
File "/app/.heroku/venv/lib/python2.7/site-packages/django/core/management/__init__.py", line 382, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/app/.heroku/venv/lib/python2.7/site-packages/django/core/management/base.py", line 196, in run_from_argv
self.execute(*args, **options.__dict__)
File "/app/.heroku/venv/lib/python2.7/site-packages/django/core/management/base.py", line 232, in execute
output = self.handle(*args, **options)
File "/app/.heroku/venv/lib/python2.7/site-packages/django/core/management/base.py", line 371, in handle
return self.handle_noargs(**options)
File "/app/.heroku/venv/lib/python2.7/site-packages/django/core/management/commands/syncdb.py", line 57, in handle_noargs
cursor = connection.cursor()
File "/app/.heroku/venv/lib/python2.7/site-packages/django/db/backends/__init__.py", line 306, in cursor
cursor = self.make_debug_cursor(self._cursor())
File "/app/.heroku/venv/lib/python2.7/site-packages/django/db/backends/postgresql_psycopg2/base.py", line 177, in _cursor
self.connection = Database.connect(**conn_params)
File "/app/.heroku/venv/lib/python2.7/site-packages/psycopg2/__init__.py", line 179, in connect
connection_factory=connection_factory, async=async)
psycopg2.OperationalError: FATAL: password authentication failed for user "drjstoymyqyarj"
FATAL: password authentication failed for user "drjstoymyqyarj"
I have used the database settings recommended in the heroku doc:
import dj_database_url
DATABASES = {'default': dj_database_url.config(default='postgres://localhost')}
When I check the logs after pushing code to the server, there is a suspicious Process exited with status 143 that I did not notice before. Maybe that has something to do with it?
$ heroku logs
heroku[web.1]: State changed from up to starting
heroku[web.1]: Stopping all processes with SIGTERM
heroku[web.1]: Starting process with command `python ./manage.py runserver 0.0.0.0:41048 --noreload`
app[web.1]: Validating models...
app[web.1]:
app[web.1]: 0 errors found
app[web.1]: Django version 1.4, using settings 'ClosetList.settings'
app[web.1]: Development server is running at http://0.0.0.0:41048/
app[web.1]: Quit the server with CONTROL-C.
heroku[web.1]: Process exited with status 143
heroku[web.1]: State changed from starting to up
[Edit]
Same error msg with heroku pg:psql. I am however able to open a Django shell with heroku run python manage.py shell, but I can not access any data from within it (same error of course).
[/Edit]
any help with this is appreciated.
Run
heroku config
and see if there are more than 2 heroku dbs configured and also see if DATABASE_URL point to your configured DB. If not, then you can promote your db to default DATABASE_URL by running:
heroku pg:promote HEROKU_POSTGRESQL_GREEN
Where HEROKU_POSTGRESQL_GREEN is your database name, you will see HEROKU_POSTGRESQL_GREEN_URL in your config.
Once your configured database is promoted to default database, you are ready to go.

fabric deploy problem

I'm trying to deploy a django app with fabric and get the following
error:
Alexs-MacBook:fabric alex$ fab config:instance=peergw deploy -H <ip> -
u <username> -p <password>
[192.168.2.93] run: cat /etc/issue
Traceback (most recent call last):
File "build/bdist.macosx-10.6-universal/egg/fabric/main.py", line
419, in main
File "/Users/alex/Rabota/server/mx30/scripts/fabric/fab/
commands.py", line 37, in deploy
checkup()
File "/Users/alex/Rabota/server/mx30/scripts/fabric/fab/
commands.py", line 140, in checkup
if not 'Ubuntu' in run('cat /etc/issue'):
File "build/bdist.macosx-10.6-universal/egg/fabric/network.py", line
382, in host_prompting_wrapper
File "build/bdist.macosx-10.6-universal/egg/fabric/operations.py",
line 414, in run
File "build/bdist.macosx-10.6-universal/egg/fabric/network.py", line
65, in __getitem__
File "build/bdist.macosx-10.6-universal/egg/fabric/network.py", line
140, in connect
File "build/bdist.macosx-10.6-universal/egg/paramiko/client.py",
line 149, in load_system_host_keys
File "build/bdist.macosx-10.6-universal/egg/paramiko/hostkeys.py",
line 154, in load
File "build/bdist.macosx-10.6-universal/egg/paramiko/hostkeys.py",
line 66, in from_line
File "build/bdist.macosx-10.6-universal/egg/paramiko/rsakey.py",
line 61, in __init__
paramiko.SSHException: Invalid key
Alexs-MacBook:fabric alex$
I can't connect to the server via ssh. What can be my problem?
Regards, Arshavski Alexander.
Going out on a limb here, I'd say your SSH key is incorrect:
paramiko.SSHException: Invalid key
What does your servers say when you try to ssh into it, using the username and password you were providing to fabric?
On second thought: as you are providing fabric with a password, that might suggest your SSH host key has changed and / or has not yet been added to ~/.ssh/known_hosts.
Yeah, I'd say that the host key on the machine you're connecting to has changed. (Or you are connecting from a machine that never went through the "xxx is an unknown host, do you want to add it to the list of known hosts?" dialogue.)
If you are not concerned about man-in-the-middle attacks or have changed the key yourself a few days ago add the following line somewhere in your env.variables:
env.disable_known_hosts = True
that should take care of it!