How to setup SysLogHandler with Django 1.3 logging dictionary configuration - django

I'm having no luck finding any information on setting up syslog logging with Django 1.3 dictionary configuration. The Django documents don't cover syslog and the python documentation is less than clear and doesn’t cover dictionary config at all. I've started with the following but I'm stuck on how to configure the SysLogHandler.
LOGGING = {
'version': 1,
'disable_existing_loggers': True,
'formatters': {
'verbose': {
'format': '%(levelname)s %(asctime)s %(module)s %(process)d %(thread)d %(message)s'
},
'simple': {
'format': '%(levelname)s %(message)s'
},
},
'handlers': {
'syslog':{
'level':'DEBUG',
'class':'logging.handlers.SysLogHandler',
'formatter': 'verbose'
},
},
'loggers': {
'django': {
'handlers':['syslog'],
'propagate': True,
'level':'INFO',
},
'myapp': {
'handlers': ['syslog'],
'propagate': True,
'level': 'DEBUG',
},
},
}

Finally found the answer, modify the configuration in the original question to have the following for 'syslog':
from logging.handlers import SysLogHandler
...
'syslog':{
'level':'DEBUG',
'class': 'logging.handlers.SysLogHandler',
'formatter': 'verbose',
'facility': SysLogHandler.LOG_LOCAL2,
},
...
Warning to future generations: you pretty much have to do it exactly like the above, weird errors happen if you specify the class directly etc.
Update: I've just moved to Amazon Linux (and Django 1.5) and used the following change to the configuration for the 'syslog' section in that environment, note the 'address' argument:
'syslog':{
'level':'DEBUG',
'class': 'logging.handlers.SysLogHandler',
'formatter': 'verbose',
'facility': 'local1',
'address': '/dev/log',
},

This works for me (on default debian).
I suspect using an address of /dev/log is the secret to ensure there is no attempt to use the network.
I think using a logger label of '' equates to the root logger so will catch most stuff
In settings.py:
LOGGING = {
'version': 1,
'handlers': {
'syslog':{
'address': '/dev/log',
'class': 'logging.handlers.SysLogHandler'
}
},
'loggers': {
'': {
'handlers': ['syslog'],
'level': 'DEBUG',
}
}
}
in the app:
import logging
logging.info("freakout info")
provides:
john:/var/log$ sudo tail -1 user.log
Dec 14 17:15:52 john freakout info

If you're using remote syslog, be sure to include the port in the address field, otherwise you could get [Errno 9] Bad file descriptor
The logging config would look something like this
LOGGING = {
"version": 1,
"disable_existing_loggers": False,
"formatters": {
"verbose": {
"format": "%(asctime)s %(name)-12s %(levelname)-8s %(message)s",
},
},
"handlers": {
"console": {
"level": "INFO",
"class": "logging.StreamHandler",
"stream": sys.stdout,
"formatter": "verbose",
},
"syslog": {
"level": "INFO",
"class": "logging.handlers.SysLogHandler",
"formatter": "verbose",
"facility": "user",
"address": ["192.168.1.10", 514],
}
},
"loggers": {
"": {
"handlers": ["console", "syslog"],
"level": "INFO",
"propagate": True,
},
},
}

Related

Export Django logs to Azure AppInsights with OpenCensus

I'm following this guidance for Django and Azure. I'm able to get dependancies and requests, but not traces.
I added this to middleware:
'opencensus.ext.django.middleware.OpencensusMiddleware'
Here is the LOGGING and OPENCENSUS portions of settings.py
LOGGING = {
'version': 1,
'disable_existing_loggers': False,
'formatters': {
'default': {
'format': '%(asctime)s - %(levelname)s - %(processName)s - %(name)s\n%(message)s',
},
},
"handlers": {
"azure": {
"level": "DEBUG",
"class": "opencensus.ext.azure.log_exporter.AzureLogHandler",
"instrumentation_key": assert_env('APPINSIGHTS_INSTRUMENTATIONKEY'),
},
"console": {
"level": "DEBUG",
"class": "logging.StreamHandler",
"formatter": "default",
},
},
"loggers": {
"logger_name": {"handlers": ["azure", "console"]},
},
# For some reason, this is needed or logging doesn't show up in the
# celery log file.
'skyforge.tasks': {
'handlers': ['azure','console'],
'level': assert_env('DJANGO_LOG_LEVEL'),
},
}
OPENCENSUS = {
'TRACE': {
'SAMPLER': 'opencensus.trace.samplers.ProbabilitySampler(rate=1)',
'EXPORTER': '''opencensus.ext.azure.trace_exporter.AzureExporter(
service_name='skyforge'
)'''
#Assumes Environmental Variable 'APPINSIGHTS_INSTRUMENTATIONKEY'
}
}
Any guidance on where to look for why no trace logs. The django-critical and django-tasks are still going to the console.
This is the part that is wrong:
"loggers": {
"logger_name": {"handlers": ["azure", "console"]},
The "logger_name" needed to be populated with the App name so that
logger = logging.getLogger(__name__) worked properly.
Also a level is needed in the logger.
Here is what worked for me.
LOGGING = {
'version': 1,
'disable_existing_loggers': False,
'formatters': {
'default': {
'format': '%(asctime)s - %(levelname)s - %(processName)s - %(name)s\n%(message)s',
},
},
'handlers': {
'azure': {
'class': 'opencensus.ext.azure.log_exporter.AzureLogHandler',
'formatter':'default'
# https://pypi.org/project/opencensus-ext-azure/
#Assumed ENV APPLICATIONINSIGHTS_CONNECTION_STRING
},
'console': {
'class': 'logging.StreamHandler',
'formatter': 'default',
},
},
'loggers': {
'polls': {
'handlers': ['azure', 'console'],
'level':'DEBUG'
},
}
}
The following options did not make a difference:
'level' parameter in handler
'filter':'[]' in handler
'stream':sys.stdout in handler.
I put the Appinsights connection string in an environmental variable, but the reference shows how to place in settings also. Also the original key in settings seems to work also.
APPLICATIONINSIGHTS_CONNECTION_STRING=InstrumentaionKey=<key uuid>

Django server not working with AzureLogHandler (opencensus)

I'm trying to connect my django project logs to Azure Application Insights using OpenCensus. The middleware for montirong requests works well but I also want to send telemetry logs (not just requests) to Azure. Here is my django LOGGING configuration :
LOGGING = {
'version': 1,
'disable_existing_loggers': False,
'formatters': {
'verbose': {
'format': '%(asctime)s %(levelname).3s %(process)d %(name)s : %(message)s',
},
'simple': {
'format': '%(asctime)s %(levelname)-7s : %(message)s',
},
},
'filters': {
'require_debug_false': {
'()': 'django.utils.log.RequireDebugFalse',
}
},
'handlers': {
'console': {
'class': 'logging.StreamHandler',
'formatter': 'verbose',
},
'azure': {
'formatter': 'simple',
'class': 'opencensus.ext.azure.log_exporter.AzureLogHandler',
'connection_string': 'InstrumentationKey=XXXX-XXXX-XXXX-XXXX'
},
'mail_admins': {
'level': 'ERROR',
'filters': ['require_debug_false'],
'class': 'django.utils.log.AdminEmailHandler'
}
},
'loggers': {
'': {
'level': os.environ.get('LOGLEVEL', 'INFO'),
'handlers': ['console', 'azure'],
},
'devdebug': {
'handlers': ['console'],
'level': 'INFO',
'propagate': False,
},
'django': {
'handlers': ['console', 'mail_admins'],
'level': os.environ.get('LOGLEVEL', 'INFO'),
'propagate': False,
}
},
}
Without 'azure' handler in my root logger config, everything works fine. With 'azure' handler, the server starts but doesn't work : I am unable to connect to it. I really don't know what is happening as it doesn't show me unusual logs (even with LOGLEVEL=DEBUG).
My handler configuration should be good as I can receive logs in Azure (when I run any manage command). Even when I run manage runsslserver localhost:53215, I receive logs but it is like my server is not running when I try to reach it.
I have setup the logging via settings and have it working (although not using runsslserver. I have set the connection_string as an environment variable.
The problem I am having is that the 'django' logger does not work in production. All other loggers are sending messages but not that one??? I have a classic case of it works on my machine and our dev servers but not prod... FML!!
INTEGRATIONS = ['postgresql', 'httplib','logging', 'threading']
config_integration.trace_integrations(INTEGRATIONS)
# Set the AppInsights Key as an env variable so it can be used by the logging system
os.environ['APPLICATIONINSIGHTS_CONNECTION_STRING'] = 'InstrumentationKey=XXXXXXXXXXXXXXXXXXXXXXX'
LOGGING = {
'disable_existing_loggers': True, #False, #<-- if true then make sure that you have a 'django' and '' logger
'filters': {
"require_debug_false": {
'()': 'django.utils.log.RequireDebugFalse',
},
'require_debug_true': {
'()': 'django.utils.log.RequireDebugTrue',
},
},
'formatters': {
'simple': {
'format': '[%(asctime)s] %(levelname)s %(message)s',
'datefmt': '%Y-%m-%d %H:%M:%S'
},
'verbose': {
'format': '[%(asctime)s] %(levelname)s [%(name)s.%(funcName)s:%(lineno)d] %(message)s',
'datefmt': '%Y-%m-%d %H:%M:%S'
},
'azure_verbose': {
'format': '[%(asctime)s] %(levelname)s [%(name)s.%(funcName)s:%(lineno)d] %(message)s traceId=%(traceId)s spanId=%(spanId)s',
'datefmt': '%Y-%m-%d %H:%M:%S'
},
},
'handlers': {
'console': {
'class': 'logging.StreamHandler',
'level': 'INFO',
'filters': ['require_debug_true'],
'formatter': 'verbose'
},
'azure':{
'level': 'INFO',
'filters': ['require_debug_false'],
'class': 'opencensus.ext.azure.log_exporter.AzureLogHandler',
'formatter': 'azure_verbose'
},
},
'loggers': {
'mylogger': {
"handlers": [
"azure",
"console",
],
},
'django': {
'handlers': [
'azure',
'console',
],
},
'py.warnings': {
'handlers': [
'azure',
'console',
],
},
'': {
'handlers': [
'azure',
'console',
],
},
},
'version': 1
}

How can I use a custom logging handler with django?

I'm trying to integrate logging with loki in my django app like this:
handler = logging_loki.LokiHandler(
url="http://localhost:3100/loki/api/v1/push",
tags={"app": "django", "env": ENV},
version="1",
)
LOGGING = {
'version': 1,
'disable_existing_loggers': False,
'formatters': {
'standard': {
'format': '[%(asctime)s] {%(module)s} [%(levelname)s] - %(message)s',
'datefmt': '%d-%m-%Y %H:%M:%S'
},
},
'handlers': {
'console': {
'level': 'INFO',
'class': 'logging.StreamHandler',
'formatter': 'standard',
}
},
'loggers': {
'': {
'handlers': ['console', handler], # this doesnt work
'level': 'DEBUG',
'propagate': True,
# 'name': ENV
}
}
}
What do I need to change so that django uses this custom handler? I tried just referencing the handler object in the logging dict but that didn't need to be the right approach.
I also tried this:
LOGGING_CONFIG = None
logging.config.dictConfig(LOGGING)
logging.getLogger(__name__).addHandler(handler)
but that's not sending any logs to loki
Try this:
LOGGING = {
'version': 1,
'disable_existing_loggers': False,
'formatters': {
'standard': {
'format': '[%(asctime)s] {%(module)s} [%(levelname)s] - %(message)s',
'datefmt': '%d-%m-%Y %H:%M:%S'
},
},
'handlers': {
'console': {
'level': 'INFO',
'class': 'logging.StreamHandler',
'formatter': 'standard',
},
'loki': {
'level': 'INFO',
'class': 'logging_loki.LokiHandler',
'url': "http://localhost:3100/loki/api/v1/push",
'tags' {"app": "django", "env": ENV},
'version': "1",
},
},
'loggers': {
'': {
'handlers': ['console', 'loki'],
'level': 'DEBUG',
'propagate': True,
}
}
}
Specifically, you have to:
Define you handler in the handlers dictionary in the LOGGING setting. The configuration here is used to specify the initialisation arguments for the handler.
Then assign this handler using its key (loki in the example I've given above) to the logger(s) you want it to handle.

Django creates log file but doesn't write into it

I am trying to use Django logger by giving the following settings:
LOGGING = {
'version': 1,
'disable_existing_loggers': False,
'filters': {
'require_debug_false': {
'()': 'django.utils.log.RequireDebugFalse',
},
'require_debug_true': {
'()': 'django.utils.log.RequireDebugTrue',
},
},
'formatters': {
'simple': {
'format': '[%(asctime)s] %(levelname)s %(message)s',
'datefmt': '%Y-%m-%d %H:%M:%S'
},
},
'handlers': {
'logfile': {
'level': 'INFO',
'filters': ['require_debug_false','require_debug_true'],
'class': 'logging.FileHandler',
'filename': '/tmp/django-log.log',
'formatter': 'simple'
},
},
'loggers': {
'user_activity': {
'handlers': ['logfile',]
},
}
}
and then using it in the view like following:
user_logger = logging.getLogger('user_activity')
...
user_logger.info("opened page")
Although Django creates file django-log.log, it doesn't write to it.
Is there a problem with my settings?
Thanks
In the handler filters, you can use either django.utils.log.RequireDebugFalse or django.utils.log.RequireDebugTrue, not both. They are mutually exclusive.
You would need to set the level of the logging when you create the log object using logging.getLogger(name):
Try this before writing to the logger:
user_logger.setLevel(logging.INFO)

Django logging setting configuration issue

My Logging settings is like below:
LOGGING = {
'version': 1,
'disable_existing_loggers': True,
'formatters': {
'standard': {
'format' : "[%(asctime)s] %(levelname)s [%(name)s:%(lineno)s] %(message)s",
'datefmt' : "%d/%b/%Y %H:%M:%S"
},
},
'handlers': {
'null': {
'level':'DEBUG',
'class':'django.utils.log.NullHandler',
},
'logfile': {
'level':'DEBUG',
'class':'logging.handlers.RotatingFileHandler',
'filename': LOG_ROOT + "/logfile",
'maxBytes': 50000,
'backupCount': 2,
'formatter': 'standard',
},
'console':{
'level':'INFO',
'class':'logging.StreamHandler',
'formatter': 'standard'
},
},
'loggers': {
'django': {
'handlers':['console'],
'propagate': True,
'level':'WARN',
},
'django.db.backends': {
'handlers': ['console'],
'level': 'DEBUG',
'propagate': False,
},
'home': {
'handlers': ['console', 'logfile'],
'level': 'DEBUG',
},
}
}
I am calling it like
import logging
log = logging.getLogger('home')
log.error("Hey there it works!!")
getting error No handlers could be found for logger "home"
Seems I am missing something during configuration, also went through the past SQ questions to figure it out, but couldn't figure it out.
What is wrong, any clue?
Thanks in advance.