I have the following config (based on this):
'loggers': {
'django': {
'handlers': ['console'],
'level': 'INFO',
},
'root': {
'handlers': ['console'],
'level': 'INFO',
},
},
When I run my tests like this (Django 1.8.4)
./manage.py test
I get DEBUG-level output from a source line in my own code that looks like
import logging
logging.debug("Shouldn't be seen, but is")
The line indicates the log message is going to the root logger, as I would expect:
DEBUG:root:blah: Shouldn't be seen, but is
As the tests are running it says
nosetests --verbosity=1
If I say
./manage.py test --verbosity=0
that nosetests message goes away, but the debug logging does not.
What is happening? Is my logging config wrong? Is nosetests interfering? Django?
I think my logging config is being read. I suppressed a django.request WARNING by configuring that logger in this config file.
How do I debug this?
(I read this related post, it didn't help.)
Your logger configuration is not written correctly. The 'root' configuration goes outside the 'loggers' part, like this:
'loggers': {
'django': {
'handlers': ['console'],
'level': 'INFO',
},
},
'root': {
'handlers': ['console'],
'level': 'INFO',
}
Setting it up as this should work.
Related
I have encountered a strange behavior of Django Loggers.
I am developing a front end application using Django. During the login service, I make some requests to certain components and use log.warning() calls to see the flow of the requests.
The logs worked perfectly, until I decided to add a LOGGING configuration to print the output of the logs in a file, as I want to deploy the application via Docker and I want to periodically check the log files.
When I added the following Django configuration concerning logging:
LOGGING = {
'version': 1,
'disable_existing_loggers': True,
'formatters': {
'detailed': {
'class': 'logging.Formatter',
'format': "[%(asctime)s] - [%(name)s:%(lineno)s] - [%(levelname)s] %(message)s",
}
},
'handlers': {
'console': {
'class': 'logging.StreamHandler',
'level': 'INFO',
'formatter': 'detailed',
},
'file': {
'class': 'logging.handlers.RotatingFileHandler',
'filename': "{}/am.log".format(BASE_DIR),
'mode': 'w',
'formatter': 'detailed',
'level': 'INFO',
'maxBytes': 2024 * 2024,
'backupCount': 5,
},
},
'loggers': {
'am': {
'level': 'INFO',
'handlers': ['console', 'file']
},
}
}
The logging stops working. The file specified in the logging configuration, am.log, is indeed created but nothing gets printed to this file. Even the console logging does not take place.
I have taken this logging configuration from one of my Django projects for the backend of this application, and there it works perfectly. I really don't understand what I am doing wrong. Could you please help me or guide me in the right direction. I would be very grateful.
I wish you all a good day!
By using the key "am" in your 'loggers' configuration, you're defining one logger with name "am":
'loggers': {
'am': { # <-- name of the logger
'level': 'INFO',
'handlers': ['console', 'file']
},
}
So to use that logger, you have to get it by that name:
logger = logging.getLogger("am")
logger.warning("This is a warning")
If you name your loggers by the name of the module in which you're running, which is recommended practice, then you need to define each module logger:
logger = logging.getLogger(__name__) # <-- this logger will be named after the module, e.g. your app name.
Then in your logging configuration you can specify logging behavior per module (per app):
'loggers': {
'my_app': { # <-- logging for my app
'level': 'INFO',
'handlers': ['console', 'file']
},
'django': { # <-- logging for Django module
'level': 'WARNING',
'handlers': ['console', 'file']
},
}
Or if you just want to log everything the same, use the root ('') logger, which doesn't have a name, just empty string:
'loggers': {
'': { # <-- root logger
'level': 'INFO',
'handlers': ['console', 'file']
},
}
I'm trying to configure django logging in the django settings file so that it logs django info and info for my application to a custom file for easy viewing. Here's my logging config:
'version': 1,
'disable_existing_loggers': False,
'formatters': {
'console': {
# exact format is not important, this is the minimum information
'format': '%(asctime)s %(name)-12s %(levelname)-8s %(message)s',
},
'file': {
# exact format is not important, this is the minimum information
'format': '%(asctime)s %(name)-12s %(levelname)-8s %(message)s',
},
},
'handlers': {
'file': {
'level': 'DEBUG',
'class': 'logging.handlers.RotatingFileHandler',
'formatter': 'file',
'filename': 'logs/django_log.log',
'backupCount': 10, # keep at most 10 log files
'maxBytes': 5242880, # 5*1024*1024 bytes (5MB)
},
'console': {
'level': 'INFO',
'class': 'logging.StreamHandler',
'formatter': 'console',
},
},
'loggers': {
'django': {
'handlers': ['file', 'console'],
'level': 'INFO',
'propagate': True,
},
'py.warnings': {
'handlers': ['console'],
},
'my_application': {
'level': 'INFO',
'handlers': ['file', 'console'],
# required to avoid double logging with root logger
'propagate': False,
},
},
}
This works on my local manage.py test server, both with django events appearing and events that I log, initialized with my_application as the logger name. However, on my web server, the logging file is created and, oddly, only populated with occasional django WARNING messages. So, there is no permissions error or inability to access the log file. Since the same config works on my local, the config can't be the issue, and it clearly has only INFO level logs.
My server setup is taken from this guide: https://www.digitalocean.com/community/tutorials/how-to-set-up-django-with-postgres-nginx-and-gunicorn-on-ubuntu-18-04 and uses Gunicorn with Nginx as a reverse proxy. Could the issue be with the configs for these? I'm stumped.
Additionally, where's a good best practice place to store this django log file?
Also, one related bonus question: What's a good best practice free/cheap service that can notify you if a specific error is logged? It seems like a good idea to set something like that up, but I don't think the django emailer is necessarily the most elegant or best.
I have this logging configuration set up:
LOGGING = {
'version': 1,
'disable_existing_loggers': False,
'filters': {
'require_debug_false': {
'()': 'django.utils.log.RequireDebugFalse'
}
},
'formatters': {
'verbose': {
'format': '%(asctime)s %(levelname)s [%(name)s:%(lineno)s] %(module)s %(process)d %(thread)d %(message)s'
}
},
'handlers': {
'console': {
'level': 'INFO',
'class': 'logging.StreamHandler',
'stream': sys.stdout,
'formatter': 'verbose'
}
},
'loggers': {
'gunicorn.errors': {
'level': 'INFO',
'handlers': ['console'],
'propagate': True,
},
}
}
It seems to have no effect at all. When the flag DEBUG is set to True then I can see some errors in the console. When however it's set to False, I cannot. So how to log the errors to the console despite DEBUG flag being set one way or the other?
In django 1.9 the default logging configuration has been changed:
Changes to the default logging configuration¶
To make it easier to write custom logging configurations, Django’s
default logging configuration no longer defines ‘django.request’ and
‘django.security’ loggers. Instead, it defines a single ‘django’
logger, filtered at the INFO level, with two handlers:
‘console’: filtered at the INFO level and only active if DEBUG=True.
‘mail_admins’: filtered at the ERROR level and only active if
DEBUG=False. If you aren’t overriding Django’s default logging, you
should see minimal changes in behavior, but you might see some new
logging to the runserver console, for example.
If you are overriding Django’s default logging, you should check to
see how your configuration merges with the new defaults.
So what you need to do is simply override the default config, add this to your loggers:
'django':{
'level': 'INFO',
'handlers': ['console'],
'propagate': True,
}
Logging to console wont depend on DEBUG now.
Hope this helps!
Try adding --capture-output and --enable-stdio-inheritance with gunicorn
/home/ubuntu/inside-env/bin/gunicorn --access-logfile access.log --error-logfile error.log --capture-output --enable-stdio-inheritance --workers 3 --bind unix:/home/ubuntu/path-to-project/webapp.sock project.wsgi:application
and then in your views,
import logging
logging.basicConfig(level='DEBUG')
logging.info('...something...')
When running/debugging individual tests using django.test.TestCase under PyCharm logging.logger messages are not shown. I've tried setting logging.basicConfig(level=logging.DEBUG) as suggested by How can I see log messages when unit testing in PyCharm? but that didn't help either. I suspect it might be django's TestCase setup interferring.
Is there some other way in test setup or runner configuration that I can turn on debug logging for the test run?
The logging I have set up in my settings.py right now is:
LOGGING = {
'version': 1,
'disable_existing_loggers': False,
'handlers': {
'console': {
'class': 'logging.StreamHandler',
'formatter': 'verbose'
},
'file': {
'level': 'DEBUG',
'class': 'logging.handlers.TimedRotatingFileHandler',
'filename': '/var/log/em/mcqueen-dev.log',
'when': 'midnight',
'formatter': 'verbose',
},
},
'formatters': {
'verbose': {
'format': '%(asctime)s.%(msecs).03d - %(process)d - %(thread)d - %(levelname)8s - %(filename)s:%(lineno)d - %(funcName)s - %(message)s'
},
'simple': {
'format': '%(asctime)s - %(levelname)s %(message)s'
},
},
'loggers': {
'mcqueen_api': {
'handlers': ['console', 'file'],
'level': os.getenv('DJANGO_LOG_LEVEL', 'DEBUG')
},
'mcqueen_app': {
'handlers': ['console', 'file'],
'level': os.getenv('DJANGO_LOG_LEVEL', 'DEBUG')
},
'mcqueen_base': {
'handlers': ['console', 'file'],
'level': os.getenv('DJANGO_LOG_LEVEL', 'DEBUG')
},
},
}
When I want to see logs while working on tests for Django project in PyCharm I'm putting this snippet of code in the file with test:
import logging
logger = logging.getLogger(__name__)
logging.disable(logging.NOTSET)
logger.setLevel(logging.DEBUG)
While running Django tests level for disabling logs is set high (to 50), lowering it (as in line #3 in code above) will cause that logs will be displayed (or saved to file - depending on log handler that is in use).
This thread on stackoverflow explains the probable reason your logging output not showing to console. Apparently, django's unittest runner replaces the global sys.stdout/sys.stderr, but the StreamHandler specified from the django settings is still bound up with the original sys.stdout/sys.stderr. The fix is to add a stream handler to your logger in the test module, based on the values of sys.stdout/sys.stderr during execution.
If you want the logger to log to console for all methods of your test case, then you're probably best using a custom base class (see linked to thread for more details) to wrap the logic for adding/removing at setUp/tearDown.
I prefer to use decorators over the individual test methods for wrapping. For example (using the 'django_test' logger config provided by Sơn Lâm's answer):
import logging
import sys
from contextlib import contextmanager
from django.test import TestCase
#contextmanager
def streamhandler_to_console(lggr):
# Use 'up to date' value of sys.stdout for StreamHandler,
# as set by test runner.
stream_handler = logging.StreamHandler(sys.stdout)
lggr.addHandler(stream_handler)
yield
lggr.removeHandler(stream_handler)
def testcase_log_console(lggr):
def testcase_decorator(func):
def testcase_log_console(*args, **kwargs):
with streamhandler_to_console(lggr):
return func(*args, **kwargs)
return testcase_log_console
return testcase_decorator
logger = logging.getLogger('django_test')
class SomeTestCase(TestCase):
#testcase_log_console(logger)
def test_something(self):
logger.info('show something to console.')
I think it will be work
Log configuration on settings.py
LOGGING = {
'version': 1,
'disable_existing_loggers': False,
'formatters': {
'verbose': {
'format': "[%(asctime)s] %(levelname)s %(message)s",
'datefmt': "%d/%b/%Y %H:%M:%S"
}
},
'handlers': {
'file': {
'level': 'DEBUG',
'class': 'logging.FileHandler',
'filename': '/var/log/django_practices.log',
'formatter': 'verbose'
},
'console': {
'level': 'DEBUG',
'class': 'logging.StreamHandler',
'stream': sys.stdout,
'formatter': 'verbose'
},
},
'loggers': {
'django_test': {
'handlers': ['file', 'console'],
'level': 'DEBUG',
},
'name_your_app': {
'handlers': ['file', 'console'],
'level': 'DEBUG',
}
}
}
In UnitTest file
import logging
logger = logging.getLogger('django_test')
logger.info('test_log')
And Log will be appearance.
I think this is to do with the test runner that you're using - assuming you're using the inbuilt Django tests setup in PyCharm then it should already be setting an environment variable PYTHONUNBUFFERED=1 which I think is what makes the output print directly without being buffered and only showing at the end (which is what I presume is happening). Check that this is set in the test configuration and if it isn't then try that.
See also: Pycharm unit test interactive debug command line doesn't work (particularly if you're using a different test runner)
I have my logging configured like so:
LOGGING = {
'version': 1,
'disable_existing_loggers': False,
'handlers': {
'mail_admins': {
'level': 'INFO',
'class': 'common.handlers.SuperAdminEmailHandler'
},
'console': {
'level': 'INFO',
'class': 'logging.StreamHandler'
}
},
'loggers': {
'': {
'handlers': ['console'],
'level': 'INFO',
'propagate': True,
},
'django.request': {
'handlers': ['console'],
'level': 'ERROR',
'propagate': True,
},
},
}
This is with CELERYD_HIJACK_ROOT_LOGGER = False. I also have this task:
#periodic_task(run_every=crontab())
def test_logging():
import logging
print('running test_logging')
logging.info("Here's an info message", extra = {
'tell_admins': True, 'email_content': 'Additional content'
})
logging.error("Here's an error message")
I have a test URL that calls test_logging directly. When I hit that URL I see this output:
running test_logging
Here's an info message
Here's an error message
[05/Jul/2013 11:07:27] "GET /test/ HTTP/1.1" 200 7
Exactly what I expect. When that same function runs through celery though, I get:
Scheduler: Sending due task scheduler.tasks.test_logging (scheduler.tasks.test_logging)
running test_logging
Here's an error message
Where'd the info message go?! I tried adding another celery-specific logger:
'celery': {
'handlers': ['console'],
'level': 'INFO',
'propagate': True,
},
Now I see this from celery:
running test_logging
running test_logging
Here's an error message
Task scheduler.tasks.test_logging[271b8a4a-0c04-4391-81c5-5b009d70b08d] succeeded in 0.00929498672485s: None
Still no info message. Any thoughts on why and how to get it back?
For some reason, even if Celery is using then handlers you specify in Django's logging settings, it still overrides the levels of the loggers. I'm now using this function to get a logger. With the logger set to the appropriate level, things seem to be working as expected.
import logging
from celery.utils.log import get_task_logger
def get_celery_logger(name):
logger = get_task_logger(name)
logger.level = logging.INFO
return logger
First of all, if you use django-celery you can try it run as so:
./manage.py celeryd --loglevel=info
Secondly, if you use Celery >= 3.0 you must get logger object as:
from celery.utils.log import get_task_logger
logger = get_task_logger(__name__)
# and inside your function use
# logger instead logging
#periodic_task(run_every=crontab())
def test_logging():
print('running test_logging')
logger.info("Here's an info message", extra = {
'tell_admins': True, 'email_content': 'Additional content'
})
logger.error("Here's an error message")
If it would not work you also can try change LOGGIN object