Django logger disabled when using Gunicorn - django

I am trying to use Django logging with gunicorn. I am able to use it with Django development server, but when I use with gunicorn logs are not being written to file or console.
Here is my logging config(stripped down):
LOGGING = {
'version': 1,
'disable_existing_loggers': False,
'formatters': {
'verbose': {
'format': '[%(asctime)s] %(levelname)s [%(filename)s:%(lineno)s] %(message)s'
},
'simple': {
'format': '%(message)s'
}
},
'handlers': {
'console': {
'class': 'logging.StreamHandler',
},
'docs_file_handler': {
'class': 'logging.FileHandler',
'filename': os.path.join(LOGS_DIR, 'docs.log'),
'level': 'DEBUG',
'formatter': 'simple'
}
},
'loggers': {
'docs.views': {
'handlers': ['docs_file_handler', 'console'],
'propagate': True,
'level': 'DEBUG',
},
},
}
When I print the 'logger.disabled' it comes out True.
To get the logger:
logger = logging.getLogger(__name__)
print(logger.handlers) # Returns [<FileHandler /var/efs/docs.log (DEBUG)>]
print(logger.disabled) # Returns True
# When I manually set, logger starts working
logger.disabled = False
Gunicorn command:
gunicorn my_project.wsgi --enable-stdio-inheritance --log-level "info" --error-logfile "/var/efs/gunicorn.error.log" --access-logfile "/var/efs/gunicorn.access.log" --reload
I am unable to get the logs to work with Gunicorn.

I found the solution, in one of my modules I was explicitly specifying a logger with 'disable_existing_loggers' as True.

Related

How to disable logging in pyppeteer

I'm using pyppeteer to take screenshots of images to a make a pdf but pyppeteer auto logs everything I take a screenshot at and because of server limitations and the logs is written to a file the logs are crashing my server.
Is there any way to completely disable logging? I tried this already:
'logLevel': logging.NOTSET,
'env': {'DEBUG': 'puppeteer:*,-not_this'},
I also tried to disable logging like this:
logging.getLogger('pyppeteer').setLevel(logging.NOTSET)
And nothing seems to work.
Update
I managed to found a workaround although not a solution by disabling all logging in the application like this:
logging.disable(logging.CRITICAL)
# pyppeteer code...
logging.disable(logging.NOTSET)
Try this:
logging.getLogger("<logger name>").disabled = True
If you want to disable all loggers you can use:
for name in logging.root.manager.loggerDict:
# print("logger", name)
logging.getLogger(name).disabled = True
Or maybe if you want to disable all except some loggers you created:
allowed_loggers = ['some_logger']
for name in logging.root.manager.loggerDict:
# print("logger", name)
if name in allowed_loggers:
continue
logging.getLogger(name).disabled = True
This should log all of your errors to a file.
LOGGING = {
'version': 1,
'disable_existing_loggers': False,
'formatters': {
'simple': {
'format': '{asctime} {name}] {message}',
'style': '{',
}
},
'handlers': {
'file': {
'level': 'ERROR',
'class': 'logging.FileHandler',
'filename': 'error.log',
'formatter': 'simple',
},
},
'loggers': {
'django': {
'handlers': ['file'],
'level': 'ERROR',
'propagate': True,
},
'': {
'handlers': ['file'],
'level': 'ERROR',
'propagate': False,
},
},
}

Django LOGGING not printing to console and file

I have encountered a strange behavior of Django Loggers.
I am developing a front end application using Django. During the login service, I make some requests to certain components and use log.warning() calls to see the flow of the requests.
The logs worked perfectly, until I decided to add a LOGGING configuration to print the output of the logs in a file, as I want to deploy the application via Docker and I want to periodically check the log files.
When I added the following Django configuration concerning logging:
LOGGING = {
'version': 1,
'disable_existing_loggers': True,
'formatters': {
'detailed': {
'class': 'logging.Formatter',
'format': "[%(asctime)s] - [%(name)s:%(lineno)s] - [%(levelname)s] %(message)s",
}
},
'handlers': {
'console': {
'class': 'logging.StreamHandler',
'level': 'INFO',
'formatter': 'detailed',
},
'file': {
'class': 'logging.handlers.RotatingFileHandler',
'filename': "{}/am.log".format(BASE_DIR),
'mode': 'w',
'formatter': 'detailed',
'level': 'INFO',
'maxBytes': 2024 * 2024,
'backupCount': 5,
},
},
'loggers': {
'am': {
'level': 'INFO',
'handlers': ['console', 'file']
},
}
}
The logging stops working. The file specified in the logging configuration, am.log, is indeed created but nothing gets printed to this file. Even the console logging does not take place.
I have taken this logging configuration from one of my Django projects for the backend of this application, and there it works perfectly. I really don't understand what I am doing wrong. Could you please help me or guide me in the right direction. I would be very grateful.
I wish you all a good day!
By using the key "am" in your 'loggers' configuration, you're defining one logger with name "am":
'loggers': {
'am': { # <-- name of the logger
'level': 'INFO',
'handlers': ['console', 'file']
},
}
So to use that logger, you have to get it by that name:
logger = logging.getLogger("am")
logger.warning("This is a warning")
If you name your loggers by the name of the module in which you're running, which is recommended practice, then you need to define each module logger:
logger = logging.getLogger(__name__) # <-- this logger will be named after the module, e.g. your app name.
Then in your logging configuration you can specify logging behavior per module (per app):
'loggers': {
'my_app': { # <-- logging for my app
'level': 'INFO',
'handlers': ['console', 'file']
},
'django': { # <-- logging for Django module
'level': 'WARNING',
'handlers': ['console', 'file']
},
}
Or if you just want to log everything the same, use the root ('') logger, which doesn't have a name, just empty string:
'loggers': {
'': { # <-- root logger
'level': 'INFO',
'handlers': ['console', 'file']
},
}

tornado deploy django and the log repeat output

In my project i deployed my django-project with tornado server, and my tornado main function is:
def main():
tornado.options.options.logging = None
tornado.options.parse_command_line()
os.environ['DJANGO_SETTINGS_MODULE'] = 'Zero.settings'
application = get_wsgi_application()
container = tornado.wsgi.WSGIContainer(application)
http_server = tornado.httpserver.HTTPServer(container, xheaders=True)
http_server.listen(tornado.options.options.port)
tornado.ioloop.IOLoop.current().start()
When i use tornado.options.options.logging = None to disabled tornado logging output, but it still output the log message in my console with twice, my django logging config is:
LOGGING = {
'version': 1,
'disable_existing_loggers': True,
'formatters': {
'standard': {
'format': '%(asctime)s [%(threadName)s] [%(name)s:%(funcName)s] [%(levelname)s]- %(message)s'}
},
'filters': {
},
'handlers': {
'error': {
'level': 'ERROR',
'class': 'logging.handlers.RotatingFileHandler',
'filename': os.path.join(BASE_DIR, 'log', 'error.log'),
'maxBytes': 1024*1024*5,
'backupCount': 5,
'formatter': 'standard',
},
'console':{
'level': 'INFO',
'class': 'logging.StreamHandler',
'formatter': 'standard'
},
},
'loggers': {
'django': {
'handlers': ['console'],
'level': 'INFO',
'propagate': True
},
}
}
The final result is:
2018-06-15 17:40:55,724 [MainThread] [base_views:get] [INFO]- get message correct
INFO:base_views:get message correct
so what can i do to solve this problem.
Thank U.
You've only configured the django logger, not the root logger. When Tornado sees that the root logger is not configured, it adds its own last-resort handler (using logging.basicConfig instead of tornado.log). Because of the way python loggers propagate, this results in any other loggers being duplicated.
When you use tornado.options.options.logging = None, you should make sure that you configure the root logger yourself, or configure all of your other handlers with propagate=False. In this case, move the loggers.django section of your config to a new root section:
'handlers': {...}
'root': {
'handlers': ['console'],
'level': 'INFO',
},

console not showing print message on server, django and uwsgi with nginx

I am currently using django's back end to post another api which works correctly when I am developing locally. But when I push it to a staging cloud server which uses uwsgi with nginx that it is not working properly anymore. I am trying to use print from django to either save the messages into a file or somehow show it in terminal so I can know the post response to start debugging but I have no luck trying to find a way to log any print
I have already did something like this in the settings
LOGGING = {
'version': 1,
'formatters': {
'verbose': {
'format': '%(levelname)s %(asctime)s %(module)s %(process)d %(thread)d %(message)s'
},
'simple': {
'format': '%(levelname)s %(message)s'
},
},
'handlers': {
'console': {
'level': 'DEBUG',
'class': 'logging.StreamHandler',
'formatter': 'simple'
},
'file': {
'level': 'DEBUG',
'class': 'logging.FileHandler',
'filename': '/path/to/your/file.log',
'formatter': 'simple'
},
},
'loggers': {
'django': {
'handlers': ['file'],
'level': 'DEBUG',
'propagate': True,
},
}
}
it does work perfectly and show messages normally but how can I also print messages into the file too?
Can someone give me an idea or advice?
Thanks in advance
So it looks like you already have file logging set up properly. The only thing is that it does not log prints, you have to use django's logging system. Like this:
import logging
logger = logging.getLogger(__name__)
# In your code
logger.debug('Some message') # Logs as debug message
logger.error('Error message') # Logs as error message
Your logger should log all of these to the file since it's level is DEBUG
More info here: https://docs.djangoproject.com/en/2.0/topics/logging/#using-logging

Uwsgi disables django.request logging

If I start application using uwsgi I don't see logs related to django.requests.
But If I start the same code on the same machine using
manage.py runserver 8080
it works perfectly.
Any ideas why it might happen?
I run uwsgi by this command
/home/gs/python-env/bin/uwsgi --ini /etc/uwsgi.d/uwsgi.ini --static-map /static=/home/gs/api/static/
uwsgi.ini
[uwsgi]
http-socket=:8080
home=/home/gs/python-env
chdir=/home/gs/api
module=server.wsgi
env=server.settings
processes=1
enable-threads=true
My logging configuration from settings.py
LOGGING = {
'version': 1,
'disable_existing_loggers': True,
'formatters': {
'verbose': {
'format': '%(levelname)s %(asctime)s %(process)d %(threadName)s %(module)s %(funcName)s %(message)s'
}
},
'handlers': {
'console': {
'class': 'logging.StreamHandler',
},
'file': {
'level': 'DEBUG',
'class': 'logging.handlers.RotatingFileHandler',
'filename': '/var/log/gs/api.log',
'formatter': 'verbose',
'maxBytes': 1024 * 1024 * 16, # 16Mb
},
'elasticsearch': {
'level': 'DEBUG',
'class': 'api.common.elasticsearch_log_handler.ElasticSearchHandler',
'hosts': [{'host': cluster.ES_HOST, 'port': 443}],
'es_index_name': 'logstash',
'es_additional_fields': {'type': 'api', 'cluser': cluster.CLUSTER_NAME},
'auth_type': ElasticSearchHandler.AuthType.NO_AUTH,
'use_ssl': True,
}
},
'loggers': {
'django': {
'handlers': ['file', 'elasticsearch', 'console'],
'level': 'INFO',
'propagate': True
},
'django.request': {
'handlers': ['file', 'elasticsearch', 'console'],
'level': 'DEBUG',
'propagate':False
}
}
}
If I change info to debug for 'django' I will see my logs from django logger but not from django.request.
UPD: If I write my own middleware I can log requests. But I want to know why django.request doesn't work with uwsgi.
Django's runserver provides the log messages that show up under django.server. When not running under runserver there are still some messages that can be logged to django.request (mostly error messages) but the informational log message for each request only exists in runserver. I verified this by looking at the uWSGI and the Django source.
If you want a similar log message you can use django-request-logging.