Logging shows wrong linenumber - python-2.7

Im trying to learn python 2.7s logging module, and have copied this clip:
def _logging():
# set up program logging
global logger
# valid logging levels:
# =====================
# logger.debug('debug message')
# logger.info('info message')
# logger.warn('warn message')
# logger.error('error message')
# logger.critical('critical message')
#
# to show traceback:
# logger.critical(str(e), exc_info = True) # if 'error' & 'True' - halts script
logger = logging.getLogger()
logger.setLevel(logging.DEBUG)
# create file handler which logs INFO & above messages
fh = logging.FileHandler('c:/temp/pgmLog.log','w')
fh.setLevel(logging.INFO)
# create console handler that logs all messages
ch = logging.StreamHandler()
ch.setLevel(logging.DEBUG)
# create formatter and add it to the handlers
formatter = logging.Formatter('%(asctime)s %(lineno)-5d %(levelname)-12s %(message)s', datefmt='%m-%d-%Y %H:%M:%S')
ch.setFormatter(formatter)
fh.setFormatter(formatter)
# add the handlers to logger
logger.addHandler(ch) # console logger
logger.addHandler(fh) # file logger
when I run the following piece:
logger.info('Program start')
try:
print 3/0
except Exception, err:
logger.error(err)
the logger is showing me:
11-22-2014 15:48:32 46 INFO Program start
11-22-2014 15:48:32 1415 ERROR integer division or modulo by zero
Can someone please tell me why the error is showing as line 1415, and not the real location of the error?
Many thanks

Related

Celery class based task is ran synchronously instead of asynchronously

I have used cookiecutter to setup a django project. The relevant parts of the project should be:
File config/settings/base.py:
INSTALLED_APPS += ['foo.taskapp.celery.CeleryAppConfig']
if USE_TZ:
CELERY_TIMEZONE = TIME_ZONE
CELERY_BROKER_URL = env('CELERY_BROKER_URL', default='redis://redis:6379/0')
CELERY_RESULT_BACKEND = CELERY_BROKER_URL
CELERY_ACCEPT_CONTENT = ['json']
CELERY_TASK_SERIALIZER = 'json'
CELERY_RESULT_SERIALIZER = 'json'
CELERYD_TASK_TIME_LIMIT = 5 * 60
CELERYD_TASK_SOFT_TIME_LIMIT = 60
### EDIT: I had forgotten these: ###
CELERY_ALWAYS_EAGER=True # thats the problem
CELERY_TASK_EAGER_PROPAGATES = True
File foo/taskapp/celery.py (should be same as cookiecutter):
import os
from celery import Celery
from django.apps import apps, AppConfig
from django.conf import settings
if not settings.configured:
# set the default Django settings module for the 'celery' program.
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'config.settings.local') # pragma: no cover
app = Celery('foo')
class CeleryAppConfig(AppConfig):
name = 'foo.taskapp'
verbose_name = 'Celery Config'
def ready(self):
# Using a string here means the worker will not have to
# pickle the object when using Windows.
# - namespace='CELERY' means all celery-related configuration keys
# should have a `CELERY_` prefix.
app.config_from_object('django.conf:settings', namespace='CELERY')
installed_apps = [app_config.name for app_config in apps.get_app_configs()]
app.autodiscover_tasks(lambda: installed_apps, force=True)
if hasattr(settings, 'RAVEN_CONFIG'):
# Celery signal registration
# Since raven is required in production only,
# imports might (most surely will) be wiped out
# during PyCharm code clean up started
# in other environments.
# #formatter:off
from raven import Client as RavenClient
from raven.contrib.celery import register_signal as raven_register_signal
from raven.contrib.celery import register_logger_signal as raven_register_logger_signal
# #formatter:on
raven_client = RavenClient(dsn=settings.RAVEN_CONFIG['dsn'])
raven_register_logger_signal(raven_client)
raven_register_signal(raven_client)
#app.task(bind=True)
def debug_task(self):
print(f'Request: {self.request!r}') # pragma: no cover
File foo/foo/tasks.py:
class TaskHandler(Task):
name = "..."
def run(self, *args, **kwargs):
# this will block the main django thread
time.sleep(10)
def on_success(self, retval, task_id, args, kwargs):
# enters here succesfully
...
def on_failure(self, exc, task_id, args, kwargs, einfo):
# does NOT enter here if exception is raised in run()
...
tasktype = app.register_task(TaskHandler())
# app.tasks.register(TaskHandler()) # also tried this
File foo/foo/views.py:
class FooCreateView(FormValidMessageMixin, CreateView):
...
def form_valid(self, form):
form.instance.user = self.request.user
with transaction.atomic():
# Save the self.object and render the success_url response page
response = super().form_valid(form)
...
kwargs = { ... } # json serializable
transaction.on_commit(lambda: tasktype.delay(**kwargs))
# transaction.on_commit(lambda: TaskHandler().delay(**kwargs)) # also tried
return response
The http response is returned to the browser only after the celery task is completed. There is no celery log produced neither in
$ celery worker -A foo.taskapp -l info
nor in the devserver (runserver) console. I can only see the django root logger's messages (no info either).
Any ideas?

Running a python subprocess in Flask

I have a Flask web app running a crawling process in this fashion:
on terminal tab 1:
$ cd /path/to/scraping
$ scrapyrt
http://scrapyrt.readthedocs.io/en/latest/index.html
on terminal tab 2:
$ python app.pp
and in app.py:
params = {
'spider_name': spider,
'start_requests':True
}
response = requests.get('http://localhost:9080/crawl.json', params)
print ('RESPONSE',response)
data = json.loads(response.text)
which works.
now I'de like to move everthing into app.py, and for that I've tried:
import subprocess
from time import sleep
try:
subprocess.check_output("scrapyrt", shell=True, cwd='path/to/scraping')
except subprocess.CalledProcessError as e:
raise RuntimeError("command '{}' return with error (code {}): {}".format(e.cmd, e.returncode, e.output))
sleep(3)
params = {
'spider_name': spider,
'start_requests':True
}
response = requests.get('http://localhost:9080/crawl.json', params)
print ('RESPONSE',response)
data = json.loads(response.text)
this starts twisted, like so:
2018-02-03 17:29:35-0200 [-] Log opened.
2018-02-03 17:29:35-0200 [-] Site starting on 9080
2018-02-03 17:29:35-0200 [-] Starting factory <twisted.web.server.Site instance at 0x104effa70>
but crawling process hangs and does not go through.
what am I missing here?
Have you considered using a scheduler, such as APScheduler or similar?
You can run code using crons or in intervals, and it integrates well with Flask.
Take a look here:
http://apscheduler.readthedocs.io/en/stable/userguide.html
Here is an example:
from flask import Flask
from apscheduler.schedulers.background import BackgroundScheduler
app = Flask(__name__)
cron = BackgroundScheduler()
cron.start()
#app.route('/')
def index():
return render_template("index.html")
#cron.scheduled_job('interval', minutes=3)
def my_scrapper():
# Do the stuff you need
return True

django tornado getting 403 error on start

Following a tutorial I have written a management command to start tornado and it looks like:
import signal
import time
import tornado.httpserver
import tornado.ioloop
from django.core.management.base import BaseCommand, CommandError
from privatemessages.tornadoapp import application
class Command(BaseCommand):
args = '[port_number]'
help = 'Starts the Tornado application for message handling.'
def sig_handler(self, sig, frame):
"""Catch signal and init callback"""
tornado.ioloop.IOLoop.instance().add_callback(self.shutdown)
def shutdown(self):
"""Stop server and add callback to stop i/o loop"""
self.http_server.stop()
io_loop = tornado.ioloop.IOLoop.instance()
io_loop.add_timeout(time.time() + 2, io_loop.stop)
def handle(self, *args, **options):
if len(args) == 1:
try:
port = int(args[0])
except ValueError:
raise CommandError('Invalid port number specified')
else:
port = 8888
self.http_server = tornado.httpserver.HTTPServer(application)
self.http_server.listen(port, address="127.0.0.1")
# Init signals handler
signal.signal(signal.SIGTERM, self.sig_handler)
# This will also catch KeyboardInterrupt exception
signal.signal(signal.SIGINT, self.sig_handler)
print "start"
tornado.ioloop.IOLoop.instance().start()
print "end"
Here I when I run this management command I get tornado tornado.access:403 GET /2/ (127.0.0.1) 1.92ms error
For test purpose I have printed "start" and "end". I guess when this command is executed successfully "end" should be printed.
Here only "start" is printed not "end". I guess there is something error on tornado.ioloop.IOLoop.instance().start() but I dont know what it is.
Can anyone guide me what is wrong in here ?
You forgot to put self.http_server.start() before starting the ioloop.
...
self.http_server = tornado.httpserver.HTTPServer(application)
self.http_server.listen(port, address="127.0.0.1")
self.http_server.start()
...
Update
Along with the httpserver start missing, are you using this library?
Your management command is exactly this one
Their tutorial is in russian and honestly, I don't read russian.
Update2:
What is see in their code, is that the url /(?P<thread_id>\d+)/ is a websocket handler:
application = tornado.web.Application([
(r"/", MainHandler),
(r'/(?P<thread_id>\d+)/', MessagesHandler),
])
...
class MessagesHandler(tornado.websocket.WebSocketHandler):
...
But the error you posted seems like something tried to access it in http via GET.
Honestly, without a debugger and the same environment, I can't figure out the issue.

Django - custom logger

I have some interesting task - write custom logger for Django, that stores request (INFO level) and show data (DEBUG level) in file.
I cannot use standard Django logger.
Can you point me in right direction?
Should I use middleware?
Thanks. I appreciate your time.
So far i figure out something like:
import os
import logging
from django.conf import settings
# logging config
logging.basicConfig(filename=os.path.join(settings.BASE_DIR, 'debug.log'),
level=logging.DEBUG)
logger = logging.getLogger(__name__)
class FileLoggerMiddleware(object):
def process_response(self, request, response):
request_log = 'Path: {}, GET: {}, POST: {}'.format(
request.path, request.GET, request.POST)
logger.info(request_log)
# We have queries logging out of box
# But here what i wood wrote:
# from django.db import connection
# for q in connection.queries:
# query_log = "({}) QUERY ".format(q.time, q.sql ) //and more info
# logger.debug(query_log)
return response

Stop a python 2.7 daemon by the signal

I use python 2.7.3 and daemon runner in my script. In a run(loop) method i want to sleep for the some time, but not with the such code:
while True:
time.sleep(10)
I want wait on a some synchronizing primitive, for example multiprocessing.Event. There is my code:
# -*- coding: utf-8 -*-
import logging
from daemon import runner
import signal
import multiprocessing
import spyder_cfg
logging.basicConfig(level=logging.DEBUG, format='%(asctime)s %(name)-12s %(levelname)-8s %(message)s', datefmt='%m-%d %H:%M', filename=spyder_cfg.log_file)
class Daemon(object):
def __init__(self, pidfile_path):
self.stdin_path = '/dev/null'
self.stdout_path = '/dev/tty'
self.stderr_path = '/dev/tty'
self.pidfile_path = None
self.pidfile_timeout = 5
self.pidfile_path = pidfile_path
def setup_daemon_context(self, daemon_context):
self.daemon_context = daemon_context
def run(self):
logging.info('Spyder service has started')
logging.debug('event from the run() = {}'.format(self.daemon_context.stop_event))
while not self.daemon_context.stop_event.wait(10):
try:
logging.info('Spyder is working...')
except BaseException as exc:
logging.exception(exc)
logging.info('Spyder service has been stopped')
def handle_exit(self, signum, frame):
try:
logging.info('Spyder stopping...')
self.daemon_context.stop_event.set()
except BaseException as exc:
logging.exception(exc)
if __name__ == '__main__':
app = Daemon(spyder_cfg.pid_file)
d = runner.DaemonRunner(app)
d.daemon_context.working_directory = spyder_cfg.work_dir
d.daemon_context.files_preserve = [h.stream for h in logging.root.handlers]
d.daemon_context.signal_map = {signal.SIGUSR1: app.handle_exit}
d.daemon_context.stop_event = multiprocessing.Event()
app.setup_daemon_context(d.daemon_context)
logging.debug('event from the main = {}'.format(d.daemon_context.stop_event))
d.do_action()
It is my log file records:
06-04 11:32 root DEBUG event from the main = <multiprocessing.synchronize.Event object at 0x7f0ef0930d50>
06-04 11:32 root INFO Spyder service has started
06-04 11:32 root DEBUG event from the run() = <multiprocessing.synchronize.Event object at 0x7f0ef0930d50>
06-04 11:32 root INFO Spyder is working...
06-04 11:32 root INFO Spyder stopping...
There is not 'Spyder service has been stopped' print in the log, my program hang on the set() call. While debugging i see that it hang when Event.set() call, the set method hang on semaphore while all waiting entities wake up. There is no reason if Event will be global object or threading.Event. I see this one answer, but its not good for me. Is there an alternative for wait with the timeout wait with the same behavior as multiprocessing.Event?
I do print stack from the signal handler and i think there is deadlock, because signal handler use same stack with the my main process and when i call Event.set(), method wait() higher on the stack...
def handle_exit(self, signum, frame):
try:
logging.debug('Signal handler:{}'.format(traceback.print_stack()))
except BaseException as exc:
logging.exception(exc)
d.do_action()
File ".../venv/work/local/lib/python2.7/site-packages/daemon/runner.py", line 189, in do_action
func(self)
File ".../venv/work/local/lib/python2.7/site-packages/daemon/runner.py", line 134, in _start
self.app.run()
File ".../venv/work/skelet/src/spyder.py", line 32, in run
while not self.daemon_context.stop_event.wait(10):
File "/usr/lib/python2.7/multiprocessing/synchronize.py", line 337, in wait
self._cond.wait(timeout)
File "/usr/lib/python2.7/multiprocessing/synchronize.py", line 246, in wait
self._wait_semaphore.acquire(True, timeout)
File ".../venv/work/skelet/src/spyder.py", line 41, in handle_exit
logging.debug('Signal handler:{}'.format(traceback.print_stack()))
that is why this fix solve the problem:
def handle_exit(self, signum, frame):
t = Timer(1, self.handle_exit2)
t.start()
def handle_exit2(self):
self.daemon_context.stop_event.set()