Uwsgi disables django.request logging - django

If I start application using uwsgi I don't see logs related to django.requests.
But If I start the same code on the same machine using
manage.py runserver 8080
it works perfectly.
Any ideas why it might happen?
I run uwsgi by this command
/home/gs/python-env/bin/uwsgi --ini /etc/uwsgi.d/uwsgi.ini --static-map /static=/home/gs/api/static/
uwsgi.ini
[uwsgi]
http-socket=:8080
home=/home/gs/python-env
chdir=/home/gs/api
module=server.wsgi
env=server.settings
processes=1
enable-threads=true
My logging configuration from settings.py
LOGGING = {
'version': 1,
'disable_existing_loggers': True,
'formatters': {
'verbose': {
'format': '%(levelname)s %(asctime)s %(process)d %(threadName)s %(module)s %(funcName)s %(message)s'
}
},
'handlers': {
'console': {
'class': 'logging.StreamHandler',
},
'file': {
'level': 'DEBUG',
'class': 'logging.handlers.RotatingFileHandler',
'filename': '/var/log/gs/api.log',
'formatter': 'verbose',
'maxBytes': 1024 * 1024 * 16, # 16Mb
},
'elasticsearch': {
'level': 'DEBUG',
'class': 'api.common.elasticsearch_log_handler.ElasticSearchHandler',
'hosts': [{'host': cluster.ES_HOST, 'port': 443}],
'es_index_name': 'logstash',
'es_additional_fields': {'type': 'api', 'cluser': cluster.CLUSTER_NAME},
'auth_type': ElasticSearchHandler.AuthType.NO_AUTH,
'use_ssl': True,
}
},
'loggers': {
'django': {
'handlers': ['file', 'elasticsearch', 'console'],
'level': 'INFO',
'propagate': True
},
'django.request': {
'handlers': ['file', 'elasticsearch', 'console'],
'level': 'DEBUG',
'propagate':False
}
}
}
If I change info to debug for 'django' I will see my logs from django logger but not from django.request.
UPD: If I write my own middleware I can log requests. But I want to know why django.request doesn't work with uwsgi.

Django's runserver provides the log messages that show up under django.server. When not running under runserver there are still some messages that can be logged to django.request (mostly error messages) but the informational log message for each request only exists in runserver. I verified this by looking at the uWSGI and the Django source.
If you want a similar log message you can use django-request-logging.

Related

How to include a Django logging.FileHanler with Celery on ElasticBeanstalk

If I do not include logging I have an Django application with Celery working fine on ElasticBeanstalk using AWS SQS.
When I include logging with a 'logging.FileHandler' celery gets permission denied error because it doesn't have permission rights for my log files. This is my error
ValueError: Unable to configure handler 'celery_file': [Errno 13]
Permission denied: '/opt/python/log/django.log'
This is my logging setup
LOGGING = {
'version': 1,
'disable_existing_loggers': False,
'filters': {
'require_debug_false': {
'()': 'django.utils.log.RequireDebugFalse'
}
},
'handlers': {
'console': {
'level': 'INFO',
'class': 'logging.StreamHandler'
},
'file': {
'level': log_level,
'class': 'logging.FileHandler',
#'filters': ['require_debug_false'],
'filename': os.environ.get('LOG_FILE_PATH', LOG_FILE_PATH + '/var/log/django.log')
},
'mail_admins': {
'level': 'ERROR',
#'filters': ['require_debug_false'],
'class': 'django.utils.log.AdminEmailHandler'
},
'cqc_file' :{
'level': log_level,
'class': 'logging.FileHandler',
#'filters': ['require_debug_false'],
'filename': os.environ.get('LOG_FILE_PATH', LOG_FILE_PATH + '/var/log/cqc.log')
},
'null': {
'class': 'logging.NullHandler',
},
'celery_file': {
'level': 'INFO',
'class': 'logging.FileHandler',
'filename': os.environ.get('LOG_FILE_PATH', LOG_FILE_PATH + '/var/log/celery.log'),
}
},
'loggers': {
'': {
'level': 'WARNING',
'handlers': ['file'],
},
'debug' : {
'level': log_level,
'handlers': ['file'],
},
'django.security.DisallowedHost': {
'handlers': ['null'],
'level' : 'CRITICAL',
'propagate': False,
},
'django.request': {
'handlers': ['file','mail_admins'],
'level': 'DEBUG',
'propagate': True,
},
'cqc_report' : {
'level' : 'INFO',
'handlers' : ['cqc_file']
},
'celery.task' : {
'handlers': ['console', 'celery_file'],
'level': 'INFO',
'propagate': True,
}
}
}
I think I need to give celery access to the django.log, celery.log and cqc.log files through and elastic beanstalk container command. I tired this using:
03_change_permissions:
command: chmod g+s /opt/python/log
04_change_owner:
command: chown root:wsgi /opt/python/log
05_add_celery_to_wsgi:
command: usermod -a -G wsgi celery
But this just gave me an error saying no user celery (or something to that effect).
How do I get the File logging to work?
The error tells you that the user who is running the process does not have the necessary privileges it needs in order to write into the log. Your solution
03_change_permissions:
command: chmod g+s /opt/python/log
04_change_owner:
command: chown root:wsgi /opt/python/log
05_add_celery_to_wsgi:
command: usermod -a -G wsgi celery
seems to be on the right track, because when there are no privileges for a user to access a location, the solution is to provide the necessary privileges. However, your assumption that the user is celery seems to be wrong. A big clue for this is that this user does not exist. So, you need to make sure that
the user to run the process exists
you identify the correct user for the process
the user has the necessary privileges for logging

Django logging configuration working on local server but not on remote server

I'm trying to configure django logging in the django settings file so that it logs django info and info for my application to a custom file for easy viewing. Here's my logging config:
'version': 1,
'disable_existing_loggers': False,
'formatters': {
'console': {
# exact format is not important, this is the minimum information
'format': '%(asctime)s %(name)-12s %(levelname)-8s %(message)s',
},
'file': {
# exact format is not important, this is the minimum information
'format': '%(asctime)s %(name)-12s %(levelname)-8s %(message)s',
},
},
'handlers': {
'file': {
'level': 'DEBUG',
'class': 'logging.handlers.RotatingFileHandler',
'formatter': 'file',
'filename': 'logs/django_log.log',
'backupCount': 10, # keep at most 10 log files
'maxBytes': 5242880, # 5*1024*1024 bytes (5MB)
},
'console': {
'level': 'INFO',
'class': 'logging.StreamHandler',
'formatter': 'console',
},
},
'loggers': {
'django': {
'handlers': ['file', 'console'],
'level': 'INFO',
'propagate': True,
},
'py.warnings': {
'handlers': ['console'],
},
'my_application': {
'level': 'INFO',
'handlers': ['file', 'console'],
# required to avoid double logging with root logger
'propagate': False,
},
},
}
This works on my local manage.py test server, both with django events appearing and events that I log, initialized with my_application as the logger name. However, on my web server, the logging file is created and, oddly, only populated with occasional django WARNING messages. So, there is no permissions error or inability to access the log file. Since the same config works on my local, the config can't be the issue, and it clearly has only INFO level logs.
My server setup is taken from this guide: https://www.digitalocean.com/community/tutorials/how-to-set-up-django-with-postgres-nginx-and-gunicorn-on-ubuntu-18-04 and uses Gunicorn with Nginx as a reverse proxy. Could the issue be with the configs for these? I'm stumped.
Additionally, where's a good best practice place to store this django log file?
Also, one related bonus question: What's a good best practice free/cheap service that can notify you if a specific error is logged? It seems like a good idea to set something like that up, but I don't think the django emailer is necessarily the most elegant or best.

Django logger disabled when using Gunicorn

I am trying to use Django logging with gunicorn. I am able to use it with Django development server, but when I use with gunicorn logs are not being written to file or console.
Here is my logging config(stripped down):
LOGGING = {
'version': 1,
'disable_existing_loggers': False,
'formatters': {
'verbose': {
'format': '[%(asctime)s] %(levelname)s [%(filename)s:%(lineno)s] %(message)s'
},
'simple': {
'format': '%(message)s'
}
},
'handlers': {
'console': {
'class': 'logging.StreamHandler',
},
'docs_file_handler': {
'class': 'logging.FileHandler',
'filename': os.path.join(LOGS_DIR, 'docs.log'),
'level': 'DEBUG',
'formatter': 'simple'
}
},
'loggers': {
'docs.views': {
'handlers': ['docs_file_handler', 'console'],
'propagate': True,
'level': 'DEBUG',
},
},
}
When I print the 'logger.disabled' it comes out True.
To get the logger:
logger = logging.getLogger(__name__)
print(logger.handlers) # Returns [<FileHandler /var/efs/docs.log (DEBUG)>]
print(logger.disabled) # Returns True
# When I manually set, logger starts working
logger.disabled = False
Gunicorn command:
gunicorn my_project.wsgi --enable-stdio-inheritance --log-level "info" --error-logfile "/var/efs/gunicorn.error.log" --access-logfile "/var/efs/gunicorn.access.log" --reload
I am unable to get the logs to work with Gunicorn.
I found the solution, in one of my modules I was explicitly specifying a logger with 'disable_existing_loggers' as True.

tornado deploy django and the log repeat output

In my project i deployed my django-project with tornado server, and my tornado main function is:
def main():
tornado.options.options.logging = None
tornado.options.parse_command_line()
os.environ['DJANGO_SETTINGS_MODULE'] = 'Zero.settings'
application = get_wsgi_application()
container = tornado.wsgi.WSGIContainer(application)
http_server = tornado.httpserver.HTTPServer(container, xheaders=True)
http_server.listen(tornado.options.options.port)
tornado.ioloop.IOLoop.current().start()
When i use tornado.options.options.logging = None to disabled tornado logging output, but it still output the log message in my console with twice, my django logging config is:
LOGGING = {
'version': 1,
'disable_existing_loggers': True,
'formatters': {
'standard': {
'format': '%(asctime)s [%(threadName)s] [%(name)s:%(funcName)s] [%(levelname)s]- %(message)s'}
},
'filters': {
},
'handlers': {
'error': {
'level': 'ERROR',
'class': 'logging.handlers.RotatingFileHandler',
'filename': os.path.join(BASE_DIR, 'log', 'error.log'),
'maxBytes': 1024*1024*5,
'backupCount': 5,
'formatter': 'standard',
},
'console':{
'level': 'INFO',
'class': 'logging.StreamHandler',
'formatter': 'standard'
},
},
'loggers': {
'django': {
'handlers': ['console'],
'level': 'INFO',
'propagate': True
},
}
}
The final result is:
2018-06-15 17:40:55,724 [MainThread] [base_views:get] [INFO]- get message correct
INFO:base_views:get message correct
so what can i do to solve this problem.
Thank U.
You've only configured the django logger, not the root logger. When Tornado sees that the root logger is not configured, it adds its own last-resort handler (using logging.basicConfig instead of tornado.log). Because of the way python loggers propagate, this results in any other loggers being duplicated.
When you use tornado.options.options.logging = None, you should make sure that you configure the root logger yourself, or configure all of your other handlers with propagate=False. In this case, move the loggers.django section of your config to a new root section:
'handlers': {...}
'root': {
'handlers': ['console'],
'level': 'INFO',
},

console not showing print message on server, django and uwsgi with nginx

I am currently using django's back end to post another api which works correctly when I am developing locally. But when I push it to a staging cloud server which uses uwsgi with nginx that it is not working properly anymore. I am trying to use print from django to either save the messages into a file or somehow show it in terminal so I can know the post response to start debugging but I have no luck trying to find a way to log any print
I have already did something like this in the settings
LOGGING = {
'version': 1,
'formatters': {
'verbose': {
'format': '%(levelname)s %(asctime)s %(module)s %(process)d %(thread)d %(message)s'
},
'simple': {
'format': '%(levelname)s %(message)s'
},
},
'handlers': {
'console': {
'level': 'DEBUG',
'class': 'logging.StreamHandler',
'formatter': 'simple'
},
'file': {
'level': 'DEBUG',
'class': 'logging.FileHandler',
'filename': '/path/to/your/file.log',
'formatter': 'simple'
},
},
'loggers': {
'django': {
'handlers': ['file'],
'level': 'DEBUG',
'propagate': True,
},
}
}
it does work perfectly and show messages normally but how can I also print messages into the file too?
Can someone give me an idea or advice?
Thanks in advance
So it looks like you already have file logging set up properly. The only thing is that it does not log prints, you have to use django's logging system. Like this:
import logging
logger = logging.getLogger(__name__)
# In your code
logger.debug('Some message') # Logs as debug message
logger.error('Error message') # Logs as error message
Your logger should log all of these to the file since it's level is DEBUG
More info here: https://docs.djangoproject.com/en/2.0/topics/logging/#using-logging