Sudden 502 messages on Gunicorn - django

Question

My server has been working great for about 18 months. My setup is Nginx, Gunicorn, Django on a Ubuntu server. I will share this next piece of information just for the sake of full disclosure. One of my hard drives died on this same server. It's not the hard drive with any of the data that I'm referring to, but this hard disk did end up getting a new uuid. I feel compelled to share this information because this was when I started having trouble. Could be coincidence, could be related, I dunno. This was last Friday and I've been struggling with it ever since.
I get a 502 error. When I drill down a little bit that shows up as a Gunicorn error. I'll attach all of the information that I can think of below about my setup. But what gets me is that I can start Gunicorn from the command line, but not from the computer boot.
This works from the command line, which tells me that I have something right:
sudo service gunicorn stop
sudo /media/Storage/sales_portal/venv/bin/gunicorn --chdir=/media/Storage/sales_portal --timeout 600 --access-logfile /var/log/gunicorn/access.log --error-logfile /var/log/gunicorn/error.log --log-level debug --workers 3 --bind unix:/run/gunicorn.sock sales_portal.wsgi:application
My gunicorn.service file is:
[Unit]
Description=gunicorn daemon
Requires=gunicorn.socket
After=network.target
[Service]
# Type=notify
# the specific user that our service will run as
User=administrator
Group=administrator
# another option for an even more restricted service is
# DynamicUser=yes
# see http://0pointer.net/blog/dynamic-users-with-systemd.html
# RuntimeDirectory=gunicorn
WorkingDirectory=/media/Storage/sales_portal
ExecStart=/media/Storage/sales_portal/venv/bin/gunicorn --chdir=/media/Storage/sales_portal --timeout 600 --access-logfile /var/log/gunicorn/gunicorn.log --error-logfile /var/log/gunicorn/error.log --workers 3 --bind unix:/run/gunicorn.sock sales_portal.wsgi:application
# ExecReload=/bin/kill -s HUP $MAINPID
# KillMode=mixed
# TimeoutStopSec=5
# PrivateTmp=true
[Install]
WantedBy=multi-user.target
my gunicorn.socket information is:
[Unit]
Description=gunicorn socket
[Socket]
ListenStream=/run/gunicorn.sock
# Our service won't need permissions for the socket, since it
# inherits the file descriptor by socket activation
# only the nginx daemon will need access to the socket
# SocketUser=www-data
# Optionally restrict the socket permissions even more.
# SocketMode=600
[Install]
WantedBy=sockets.target
Trying
sudo service gunicorn start
gives me
A dependency job for gunicorn.service failed. See 'journalctl -xe' for details.
the output of this (or a portion of it, anyway) is:
Nov 20 08:21:11 server systemd[1172]: tracker-miner-fs.service: Failed with result 'exit-code'.
-- Subject: Unit failed
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- The unit UNIT has entered the 'failed' state with result 'exit-code'.
Nov 20 08:21:11 server systemd[1172]: Failed to start Tracker file system data miner.
-- Subject: A start job for unit UNIT has failed
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- A start job for unit UNIT has finished with a failure.
--
-- The job identifier is 2343 and the job result is failed.
Nov 20 08:21:11 server systemd[1172]: tracker-store.service: Scheduled restart job, restart counter is at 1.
-- Subject: Automatic restarting of a unit has been scheduled
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- Automatic restarting of the unit UNIT has been scheduled, as the result for
-- the configured Restart= setting for the unit.
Nov 20 08:21:11 server systemd[1172]: tracker-miner-fs.service: Scheduled restart job, restart counter is at 54.
-- Subject: Automatic restarting of a unit has been scheduled
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- Automatic restarting of the unit UNIT has been scheduled, as the result for
-- the configured Restart= setting for the unit.
Nov 20 08:21:11 server systemd[1172]: Stopped Tracker file system data miner.
-- Subject: A stop job for unit UNIT has finished
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- A stop job for unit UNIT has finished.
--
-- The job identifier is 2388 and the job result is done.
Nov 20 08:21:11 server systemd[1172]: Starting Tracker file system data miner...
-- Subject: A start job for unit UNIT has begun execution
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- A start job for unit UNIT has begun execution.
--
-- The job identifier is 2388.
Nov 20 08:21:11 server systemd[1172]: Stopped Tracker metadata database store and lookup manager.
-- Subject: A stop job for unit UNIT has finished
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- A stop job for unit UNIT has finished.
--
-- The job identifier is 2373 and the job result is done.
Nov 20 08:21:11 server systemd[1172]: Starting Tracker metadata database store and lookup manager...
-- Subject: A start job for unit UNIT has begun execution
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- A start job for unit UNIT has begun execution.
--
-- The job identifier is 2373.
Nov 20 08:21:11 server tracker-miner-f[167517]: Set scheduler policy to SCHED_IDLE
Nov 20 08:21:11 server tracker-miner-f[167517]: Setting priority nice level to 19
Nov 20 08:21:11 server systemd[1172]: Started Tracker metadata database store and lookup manager.
-- Subject: A start job for unit UNIT has finished successfully
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- A start job for unit UNIT has finished successfully.
--
-- The job identifier is 2373.
Nov 20 08:21:11 server tracker-miner-f[167517]: Unable to get XDG user directory path for special directory &DOCUMENTS. Ignoring this location.
Nov 20 08:21:11 server tracker-miner-f[167517]: Unable to get XDG user directory path for special directory &MUSIC. Ignoring this location.
Nov 20 08:21:11 server tracker-miner-f[167517]: Unable to get XDG user directory path for special directory &PICTURES. Ignoring this location.
Nov 20 08:21:11 server tracker-miner-f[167517]: Unable to get XDG user directory path for special directory &VIDEOS. Ignoring this location.
Nov 20 08:21:11 server tracker-miner-f[167517]: Unable to get XDG user directory path for special directory &DOWNLOAD. Ignoring this location.
Nov 20 08:21:11 server tracker-miner-f[167517]: Unable to get XDG user directory path for special directory &DOCUMENTS. Ignoring this location.
Nov 20 08:21:11 server tracker-miner-f[167517]: Unable to get XDG user directory path for special directory &MUSIC. Ignoring this location.
Nov 20 08:21:11 server tracker-miner-f[167517]: Unable to get XDG user directory path for special directory &PICTURES. Ignoring this location.
Nov 20 08:21:11 server tracker-miner-f[167517]: Unable to get XDG user directory path for special directory &VIDEOS. Ignoring this location.
Nov 20 08:21:11 server unknown[167518]: Cannot initialize database: tracker data directory does not exist and could not be created: Permission denied
Nov 20 08:21:11 server systemd[1172]: tracker-store.service: Main process exited, code=exited, status=1/FAILURE
-- Subject: Unit process exited
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- An ExecStart= process belonging to unit UNIT has exited.
--
-- The process' exit code is 'exited' and its exit status is 1.
Nov 20 08:21:11 server systemd[1172]: tracker-store.service: Failed with result 'exit-code'.
-- Subject: Unit failed
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- The unit UNIT has entered the 'failed' state with result 'exit-code'.
Nov 20 08:21:11 server dbus-daemon[1215]: [session uid=1000 pid=1215] Activating via systemd: service name='org.freedesktop.Tracker1' unit='tracker-store.service' requested by ':1.114' (uid=1000 pid=167517 comm="/usr/libexec/tracker-miner-fs " label="unconfined")
Nov 20 08:21:11 server systemd[1172]: tracker-store.service: Scheduled restart job, restart counter is at 2.
-- Subject: Automatic restarting of a unit has been scheduled
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- Automatic restarting of the unit UNIT has been scheduled, as the result for
-- the configured Restart= setting for the unit.
Nov 20 08:21:11 server systemd[1172]: Stopped Tracker metadata database store and lookup manager.
-- Subject: A stop job for unit UNIT has finished
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- A stop job for unit UNIT has finished.
--
-- The job identifier is 2403 and the job result is done.
Nov 20 08:21:11 server systemd[1172]: Starting Tracker metadata database store and lookup manager...
-- Subject: A start job for unit UNIT has begun execution
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- A start job for unit UNIT has begun execution.
--
-- The job identifier is 2403.
Nov 20 08:21:11 server dbus-daemon[1215]: [session uid=1000 pid=1215] Successfully activated service 'org.freedesktop.Tracker1'
Nov 20 08:21:11 server systemd[1172]: Started Tracker metadata database store and lookup manager.
-- Subject: A start job for unit UNIT has finished successfully
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- A start job for unit UNIT has finished successfully.
--
-- The job identifier is 2403.
Nov 20 08:21:11 server tracker-store[167527]: Cannot initialize database: tracker data directory does not exist and could not be created: Permission denied
Nov 20 08:21:11 server systemd[1172]: tracker-store.service: Main process exited, code=exited, status=1/FAILURE
-- Subject: Unit process exited
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- An ExecStart= process belonging to unit UNIT has exited.
--
-- The process' exit code is 'exited' and its exit status is 1.
Nov 20 08:21:11 server systemd[1172]: tracker-store.service: Failed with result 'exit-code'.
-- Subject: Unit failed
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- The unit UNIT has entered the 'failed' state with result 'exit-code'.
Nov 20 08:21:11 server tracker-miner-f[167517]: Couldn't create new Files miner: 'Failed to load SPARQL backend: GDBus.Error:org.freedesktop.DBus.Error.NoReply: Message recipient disconnected from message bus without replying'
Nov 20 08:21:11 server systemd[1172]: tracker-miner-fs.service: Main process exited, code=exited, status=1/FAILURE
-- Subject: Unit process exited
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- An ExecStart= process belonging to unit UNIT has exited.
--
-- The process' exit code is 'exited' and its exit status is 1.
Nov 20 08:21:11 server systemd[1172]: tracker-miner-fs.service: Failed with result 'exit-code'.
-- Subject: Unit failed
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- The unit UNIT has entered the 'failed' state with result 'exit-code'.
Nov 20 08:21:11 server systemd[1172]: Failed to start Tracker file system data miner.
-- Subject: A start job for unit UNIT has failed
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- A start job for unit UNIT has finished with a failure.
--
-- The job identifier is 2388 and the job result is failed.
Nov 20 08:21:11 server systemd[1172]: tracker-store.service: Scheduled restart job, restart counter is at 3.
-- Subject: Automatic restarting of a unit has been scheduled
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- Automatic restarting of the unit UNIT has been scheduled, as the result for
-- the configured Restart= setting for the unit.
Nov 20 08:21:11 server systemd[1172]: tracker-miner-fs.service: Scheduled restart job, restart counter is at 55.
-- Subject: Automatic restarting of a unit has been scheduled
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- Automatic restarting of the unit UNIT has been scheduled, as the result for
-- the configured Restart= setting for the unit.
Nov 20 08:21:11 server systemd[1172]: Stopped Tracker file system data miner.
-- Subject: A stop job for unit UNIT has finished
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- A stop job for unit UNIT has finished.
--
-- The job identifier is 2448 and the job result is done.
Nov 20 08:21:11 server systemd[1172]: Starting Tracker file system data miner...
-- Subject: A start job for unit UNIT has begun execution
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- A start job for unit UNIT has begun execution.
--
-- The job identifier is 2448.
Nov 20 08:21:11 server systemd[1172]: Stopped Tracker metadata database store and lookup manager.
-- Subject: A stop job for unit UNIT has finished
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- A stop job for unit UNIT has finished.
--
-- The job identifier is 2433 and the job result is done.
Nov 20 08:21:11 server systemd[1172]: Starting Tracker metadata database store and lookup manager...
-- Subject: A start job for unit UNIT has begun execution
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- A start job for unit UNIT has begun execution.
--
-- The job identifier is 2433.
Nov 20 08:21:11 server tracker-miner-f[167532]: Set scheduler policy to SCHED_IDLE
Nov 20 08:21:11 server tracker-miner-f[167532]: Setting priority nice level to 19
Nov 20 08:21:11 server systemd[1172]: Started Tracker metadata database store and lookup manager.
-- Subject: A start job for unit UNIT has finished successfully
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- A start job for unit UNIT has finished successfully.
--
-- The job identifier is 2433.
Nov 20 08:21:11 server tracker-miner-f[167532]: Unable to get XDG user directory path for special directory &DOCUMENTS. Ignoring this location.
Nov 20 08:21:11 server tracker-miner-f[167532]: Unable to get XDG user directory path for special directory &MUSIC. Ignoring this location.
Nov 20 08:21:11 server tracker-miner-f[167532]: Unable to get XDG user directory path for special directory &PICTURES. Ignoring this location.
Nov 20 08:21:11 server tracker-store[167533]: Cannot initialize database: tracker data directory does not exist and could not be created: Permission denied
Nov 20 08:21:11 server tracker-miner-f[167532]: Unable to get XDG user directory path for special directory &VIDEOS. Ignoring this location.
Nov 20 08:21:11 server tracker-miner-f[167532]: Unable to get XDG user directory path for special directory &DOWNLOAD. Ignoring this location.
Nov 20 08:21:11 server tracker-miner-f[167532]: Unable to get XDG user directory path for special directory &DOCUMENTS. Ignoring this location.
Nov 20 08:21:11 server systemd[1172]: tracker-store.service: Main process exited, code=exited, status=1/FAILURE
-- Subject: Unit process exited
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- An ExecStart= process belonging to unit UNIT has exited.
--
-- The process' exit code is 'exited' and its exit status is 1.
Nov 20 08:21:11 server tracker-miner-f[167532]: Unable to get XDG user directory path for special directory &MUSIC. Ignoring this location.
Nov 20 08:21:11 server tracker-miner-f[167532]: Unable to get XDG user directory path for special directory &PICTURES. Ignoring this location.
Nov 20 08:21:11 server tracker-miner-f[167532]: Unable to get XDG user directory path for special directory &VIDEOS. Ignoring this location.
Nov 20 08:21:11 server systemd[1172]: tracker-store.service: Failed with result 'exit-code'.
-- Subject: Unit failed
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- The unit UNIT has entered the 'failed' state with result 'exit-code'.
Nov 20 08:21:11 server dbus-daemon[1215]: [session uid=1000 pid=1215] Activating via systemd: service name='org.freedesktop.Tracker1' unit='tracker-store.service' requested by ':1.117' (uid=1000 pid=167532 comm="/usr/libexec/tracker-miner-fs " label="unconfined")
Nov 20 08:21:12 server systemd[1172]: tracker-store.service: Scheduled restart job, restart counter is at 4.
-- Subject: Automatic restarting of a unit has been scheduled
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- Automatic restarting of the unit UNIT has been scheduled, as the result for
-- the configured Restart= setting for the unit.
Nov 20 08:21:12 server systemd[1172]: Stopped Tracker metadata database store and lookup manager.
-- Subject: A stop job for unit UNIT has finished
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- A stop job for unit UNIT has finished.
--
-- The job identifier is 2463 and the job result is done.
Nov 20 08:21:12 server systemd[1172]: Starting Tracker metadata database store and lookup manager...
-- Subject: A start job for unit UNIT has begun execution
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- A start job for unit UNIT has begun execution.
--
-- The job identifier is 2463.
Nov 20 08:21:12 server dbus-daemon[1215]: [session uid=1000 pid=1215] Successfully activated service 'org.freedesktop.Tracker1'
Nov 20 08:21:12 server systemd[1172]: Started Tracker metadata database store and lookup manager.
-- Subject: A start job for unit UNIT has finished successfully
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- A start job for unit UNIT has finished successfully.
--
-- The job identifier is 2463.
Nov 20 08:21:12 server tracker-store[167542]: Cannot initialize database: tracker data directory does not exist and could not be created: Permission denied
Nov 20 08:21:12 server systemd[1172]: tracker-store.service: Main process exited, code=exited, status=1/FAILURE
-- Subject: Unit process exited
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- An ExecStart= process belonging to unit UNIT has exited.
--
-- The process' exit code is 'exited' and its exit status is 1.
Nov 20 08:21:12 server systemd[1172]: tracker-store.service: Failed with result 'exit-code'.
-- Subject: Unit failed
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- The unit UNIT has entered the 'failed' state with result 'exit-code'.
Nov 20 08:21:12 server tracker-miner-f[167532]: Couldn't create new Files miner: 'Failed to load SPARQL backend: GDBus.Error:org.freedesktop.DBus.Error.NoReply: Message recipient disconnected from message bus without replying'
Nov 20 08:21:12 server systemd[1172]: tracker-miner-fs.service: Main process exited, code=exited, status=1/FAILURE
-- Subject: Unit process exited
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- An ExecStart= process belonging to unit UNIT has exited.
--
-- The process' exit code is 'exited' and its exit status is 1.
Nov 20 08:21:12 server systemd[1172]: tracker-miner-fs.service: Failed with result 'exit-code'.
-- Subject: Unit failed
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- The unit UNIT has entered the 'failed' state with result 'exit-code'.
Nov 20 08:21:12 server systemd[1172]: Failed to start Tracker file system data miner.
-- Subject: A start job for unit UNIT has failed
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- A start job for unit UNIT has finished with a failure.
--
-- The job identifier is 2448 and the job result is failed.
Nov 20 08:21:12 server systemd[1172]: tracker-store.service: Scheduled restart job, restart counter is at 5.
-- Subject: Automatic restarting of a unit has been scheduled
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- Automatic restarting of the unit UNIT has been scheduled, as the result for
-- the configured Restart= setting for the unit.
Nov 20 08:21:12 server systemd[1172]: tracker-miner-fs.service: Scheduled restart job, restart counter is at 56.
-- Subject: Automatic restarting of a unit has been scheduled
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- Automatic restarting of the unit UNIT has been scheduled, as the result for
-- the configured Restart= setting for the unit.
Nov 20 08:21:12 server systemd[1172]: Stopped Tracker file system data miner.
-- Subject: A stop job for unit UNIT has finished
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- A stop job for unit UNIT has finished.
--
-- The job identifier is 2508 and the job result is done.
Nov 20 08:21:12 server systemd[1172]: Starting Tracker file system data miner...
-- Subject: A start job for unit UNIT has begun execution
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- A start job for unit UNIT has begun execution.
--
-- The job identifier is 2508.
Nov 20 08:21:12 server systemd[1172]: Stopped Tracker metadata database store and lookup manager.
-- Subject: A stop job for unit UNIT has finished
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- A stop job for unit UNIT has finished.
--
-- The job identifier is 2493 and the job result is done.
Nov 20 08:21:12 server systemd[1172]: tracker-store.service: Start request repeated too quickly.
Nov 20 08:21:12 server systemd[1172]: tracker-store.service: Failed with result 'exit-code'.
-- Subject: Unit failed
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- The unit UNIT has entered the 'failed' state with result 'exit-code'.
Nov 20 08:21:12 server systemd[1172]: Failed to start Tracker metadata database store and lookup manager.
-- Subject: A start job for unit UNIT has failed
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- A start job for unit UNIT has finished with a failure.
--
-- The job identifier is 2493 and the job result is failed.
Nov 20 08:21:12 server tracker-miner-f[167547]: Set scheduler policy to SCHED_IDLE
Nov 20 08:21:12 server tracker-miner-f[167547]: Setting priority nice level to 19
Nov 20 08:21:12 server tracker-miner-f[167547]: Unable to get XDG user directory path for special directory &DOCUMENTS. Ignoring this location.
Nov 20 08:21:12 server tracker-miner-f[167547]: Unable to get XDG user directory path for special directory &MUSIC. Ignoring this location.
Nov 20 08:21:12 server tracker-miner-f[167547]: Unable to get XDG user directory path for special directory &PICTURES. Ignoring this location.
Nov 20 08:21:12 server tracker-miner-f[167547]: Unable to get XDG user directory path for special directory &VIDEOS. Ignoring this location.
Nov 20 08:21:12 server tracker-miner-f[167547]: Unable to get XDG user directory path for special directory &DOWNLOAD. Ignoring this location.
Nov 20 08:21:12 server tracker-miner-f[167547]: Unable to get XDG user directory path for special directory &DOCUMENTS. Ignoring this location.
Nov 20 08:21:12 server tracker-miner-f[167547]: Unable to get XDG user directory path for special directory &MUSIC. Ignoring this location.
Nov 20 08:21:12 server tracker-miner-f[167547]: Unable to get XDG user directory path for special directory &PICTURES. Ignoring this location.
Nov 20 08:21:12 server tracker-miner-f[167547]: Unable to get XDG user directory path for special directory &VIDEOS. Ignoring this location.
Nov 20 08:21:12 server dbus-daemon[1215]: [session uid=1000 pid=1215] Activating via systemd: service name='org.freedesktop.Tracker1' unit='tracker-store.service' requested by ':1.119' (uid=1000 pid=167547 comm="/usr/libexec/tracker-miner-fs " label="unconfined")
Nov 20 08:21:12 server systemd[1172]: tracker-store.service: Start request repeated too quickly.
Nov 20 08:21:12 server systemd[1172]: tracker-store.service: Failed with result 'exit-code'.
-- Subject: Unit failed
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- The unit UNIT has entered the 'failed' state with result 'exit-code'.
Nov 20 08:21:12 server systemd[1172]: Failed to start Tracker metadata database store and lookup manager.
-- Subject: A start job for unit UNIT has failed
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- A start job for unit UNIT has finished with a failure.
--
-- The job identifier is 2523 and the job result is failed.
Nov 20 08:21:48 server sudo[167559]: administrator : TTY=pts/0 ; PWD=/home/administrator ; USER=root ; COMMAND=/bin/journalctl -xe
Nov 20 08:21:48 server sudo[167559]: pam_unix(sudo:session): session opened for user root by administrator(uid=0)
the error log that this gives me is
[2020-11-20 08:23:30 -0600] [167639] [DEBUG] Current configuration:
config: None
bind: ['unix:/run/gunicorn.sock']
backlog: 2048
workers: 3
worker_class: sync
threads: 1
worker_connections: 1000
max_requests: 0
max_requests_jitter: 0
timeout: 600
graceful_timeout: 30
keepalive: 2
limit_request_line: 4094
limit_request_fields: 100
limit_request_field_size: 8190
reload: False
reload_engine: auto
reload_extra_files: []
spew: False
check_config: False
preload_app: False
sendfile: None
reuse_port: False
chdir: /media/Storage/sales_portal
daemon: False
raw_env: []
pidfile: None
worker_tmp_dir: None
user: 0
group: 0
umask: 0
initgroups: False
tmp_upload_dir: None
secure_scheme_headers: {'X-FORWARDED-PROTOCOL': 'ssl', 'X-FORWARDED-PROTO': 'https', 'X-FORWARDED-SSL': 'on'}
forwarded_allow_ips: ['127.0.0.1']
accesslog: /var/log/gunicorn/access.log
disable_redirect_access_to_syslog: False
access_log_format: %(h)s %(l)s %(u)s %(t)s "%(r)s" %(s)s %(b)s "%(f)s" "%(a)s"
errorlog: /var/log/gunicorn/error.log
loglevel: debug
capture_output: False
logger_class: gunicorn.glogging.Logger
logconfig: None
logconfig_dict: {}
syslog_addr: udp://localhost:514
syslog: False
syslog_prefix: None
syslog_facility: user
enable_stdio_inheritance: False
statsd_host: None
dogstatsd_tags:
statsd_prefix:
proc_name: None
default_proc_name: sales_portal.wsgi:application
pythonpath: None
paste: None
on_starting: <function OnStarting.on_starting at 0x7fd62efa18b0>
on_reload: <function OnReload.on_reload at 0x7fd62efa19d0>
when_ready: <function WhenReady.when_ready at 0x7fd62efa1af0>
pre_fork: <function Prefork.pre_fork at 0x7fd62efa1c10>
post_fork: <function Postfork.post_fork at 0x7fd62efa1d30>
post_worker_init: <function PostWorkerInit.post_worker_init at 0x7fd62efa1e50>
worker_int: <function WorkerInt.worker_int at 0x7fd62efa1f70>
worker_abort: <function WorkerAbort.worker_abort at 0x7fd62efb20d0>
pre_exec: <function PreExec.pre_exec at 0x7fd62efb21f0>
pre_request: <function PreRequest.pre_request at 0x7fd62efb2310>
post_request: <function PostRequest.post_request at 0x7fd62efb23a0>
child_exit: <function ChildExit.child_exit at 0x7fd62efb24c0>
worker_exit: <function WorkerExit.worker_exit at 0x7fd62efb25e0>
nworkers_changed: <function NumWorkersChanged.nworkers_changed at 0x7fd62efb2700>
on_exit: <function OnExit.on_exit at 0x7fd62efb2820>
proxy_protocol: False
proxy_allow_ips: ['127.0.0.1']
keyfile: None
certfile: None
ssl_version: 2
cert_reqs: 0
ca_certs: None
suppress_ragged_eofs: True
do_handshake_on_connect: False
ciphers: None
raw_paste_global_conf: []
strip_header_spaces: False
[2020-11-20 08:23:30 -0600] [167639] [INFO] Starting gunicorn 20.0.4
[2020-11-20 08:23:30 -0600] [167639] [DEBUG] Arbiter booted
[2020-11-20 08:23:30 -0600] [167639] [INFO] Listening at: unix:/run/gunicorn.sock (167639)
[2020-11-20 08:23:30 -0600] [167639] [INFO] Using worker: sync
[2020-11-20 08:23:30 -0600] [167641] [INFO] Booting worker with pid: 167641
[2020-11-20 08:23:30 -0600] [167642] [INFO] Booting worker with pid: 167642
[2020-11-20 08:23:30 -0600] [167643] [INFO] Booting worker with pid: 167643
[2020-11-20 08:23:30 -0600] [167639] [DEBUG] 3 workers
[2020-11-20 14:23:38 +0000] [167643] [DEBUG] GET /opportunities/products/
That's all the information that I can think to include. If you think anything else would be helpful, just let me know. Just to reiterate, this exact setup was working until last Friday. I didn't change anything last Friday to make it stop working, that I know of. I changed a uuid in my fstab file for an unrelated hard disk, in my opinion, that should not have resulted in this behavior. Any help is appreciated. Hopefully I'm overlooking something simple.
Thanks,
Jonathan