I am new to tornado web server. When I start the tornado web server using python main_tornado.py It is working. Please see the below code.
import tornado.ioloop
import tornado.web
class MainHandler(tornado.web.RequestHandler):
def get(self):
self.write("Hello, world")
application = tornado.web.Application([
(r"/", MainHandler),
])
if __name__ == "__main__":
application.listen(8888)
tornado.ioloop.IOLoop.instance().start()
When I stop the server using CTRL+C it gave the following error.
^CTraceback (most recent call last):
File "main_tornado.py", line 19, in <module>
tornado.ioloop.IOLoop.instance().start()
File "/home/nyros/Desktop/NewWeb/venv/lib/python3.2/site-packages/tornado/ioloop.py", line 301, in start
event_pairs = self._impl.poll(poll_timeout)
KeyboardInterrupt
Please solve my problem. Thanks..
You can stop Tornado main loop with tornado.ioloop.IOLoop.instance().stop(). To have this method called after passing signal with Ctrl+C you can periodically check global flag to test if main loop should end and register handler for SIGINT signal which will change value of this flag:
#!/usr/bin/python
# -*- coding: utf-8 -*-
import signal
import logging
import tornado.ioloop
import tornado.web
import tornado.options
class MainHandler(tornado.web.RequestHandler):
def get(self):
self.write("Hello, world")
class MyApplication(tornado.web.Application):
is_closing = False
def signal_handler(self, signum, frame):
logging.info('exiting...')
self.is_closing = True
def try_exit(self):
if self.is_closing:
# clean up here
tornado.ioloop.IOLoop.instance().stop()
logging.info('exit success')
application = MyApplication([
(r"/", MainHandler),
])
if __name__ == "__main__":
tornado.options.parse_command_line()
signal.signal(signal.SIGINT, application.signal_handler)
application.listen(8888)
tornado.ioloop.PeriodicCallback(application.try_exit, 100).start()
tornado.ioloop.IOLoop.instance().start()
Output:
$ python test.py
[I 181209 22:13:43 web:2162] 200 GET / (127.0.0.1) 0.92ms
^C[I 181209 22:13:45 test:21] exiting...
[I 181209 22:13:45 test:28] exit success
UPDATE
I've just saw in question Tornado long polling requests this simple solution:
try:
tornado.ioloop.IOLoop.instance().start()
except KeyboardInterrupt:
tornado.ioloop.IOLoop.instance().stop()
Obviously, this is a less safe way.
UPDATE
Edited the code to remove use of global.
You can simply stop the Tornado ioloop from a signal handler. It should be safe thanks to add_callback_from_signal() method, the event loop will exit nicely, finishing any eventually concurrently running task.
import tornado.ioloop
import tornado.web
import signal
class MainHandler(tornado.web.RequestHandler):
def get(self):
self.write("Hello, world")
application = tornado.web.Application([
(r"/", MainHandler),
])
def sig_exit(signum, frame):
tornado.ioloop.IOLoop.instance().add_callback_from_signal(do_stop)
def do_stop(signum, frame):
tornado.ioloop.IOLoop.instance().stop()
if __name__ == "__main__":
application.listen(8888)
signal.signal(signal.SIGINT, sig_exit)
tornado.ioloop.IOLoop.instance().start()
The code is OK. The CTRL+C generates KeyboardInterrupt. To stop the server you can use CTRL+Pause Break(on windows) instead of CTRL+C. On linux CTRL+C also generates the KeyboardInterrupt also. If you will use CTRL+Z program will stop but port gets busy.
I'd say the cleanest, safest and most portable solution would be to put all closing and clean-up calls in a finally block instead of relying on KeyboardInterrupt exception:
import tornado.ioloop
import tornado.web
class MainHandler(tornado.web.RequestHandler):
def get(self):
self.write("Hello, world")
application = tornado.web.Application([
(r"/", MainHandler),
])
# .instance() is deprecated in Tornado 5
loop = tornado.ioloop.IOLoop.current()
if __name__ == "__main__":
try:
print("Starting server")
application.listen(8888)
loop.start()
except KeyboardInterrupt:
pass
finally:
loop.stop() # might be redundant, the loop has already stopped
loop.close(True) # needed to close all open sockets
print("Server shut down, exiting...")
Related
I have a Flask web app running a crawling process in this fashion:
on terminal tab 1:
$ cd /path/to/scraping
$ scrapyrt
http://scrapyrt.readthedocs.io/en/latest/index.html
on terminal tab 2:
$ python app.pp
and in app.py:
params = {
'spider_name': spider,
'start_requests':True
}
response = requests.get('http://localhost:9080/crawl.json', params)
print ('RESPONSE',response)
data = json.loads(response.text)
which works.
now I'de like to move everthing into app.py, and for that I've tried:
import subprocess
from time import sleep
try:
subprocess.check_output("scrapyrt", shell=True, cwd='path/to/scraping')
except subprocess.CalledProcessError as e:
raise RuntimeError("command '{}' return with error (code {}): {}".format(e.cmd, e.returncode, e.output))
sleep(3)
params = {
'spider_name': spider,
'start_requests':True
}
response = requests.get('http://localhost:9080/crawl.json', params)
print ('RESPONSE',response)
data = json.loads(response.text)
this starts twisted, like so:
2018-02-03 17:29:35-0200 [-] Log opened.
2018-02-03 17:29:35-0200 [-] Site starting on 9080
2018-02-03 17:29:35-0200 [-] Starting factory <twisted.web.server.Site instance at 0x104effa70>
but crawling process hangs and does not go through.
what am I missing here?
Have you considered using a scheduler, such as APScheduler or similar?
You can run code using crons or in intervals, and it integrates well with Flask.
Take a look here:
http://apscheduler.readthedocs.io/en/stable/userguide.html
Here is an example:
from flask import Flask
from apscheduler.schedulers.background import BackgroundScheduler
app = Flask(__name__)
cron = BackgroundScheduler()
cron.start()
#app.route('/')
def index():
return render_template("index.html")
#cron.scheduled_job('interval', minutes=3)
def my_scrapper():
# Do the stuff you need
return True
I wrote a SeleniumTestCase class that launches PhantomJS in its setUpClass and kills it in its tearDownClass. However, if a derived class' setUpClass raises an error, the PhantomJS process is left hanging because SeleniumTestCase.tearDownClass doesn't get called.
from django.test import LiveServerTestCase
import sys, signal, os
from selenium import webdriver
errorShots = os.path.join(os.path.dirname(os.path.abspath(__file__)), '..', "errorShots")
class SeleniumTestCase(LiveServerTestCase):
#classmethod
def setUpClass(cls):
"""
Launches PhantomJS
"""
super(SeleniumTestCase, cls).setUpClass()
cls.browser = webdriver.PhantomJS()
#classmethod
def tearDownClass(cls):
"""
Saves a screenshot if the test failed, and kills PhantomJS
"""
print 'Tearing down...'
if cls.browser:
if sys.exc_info()[0]:
try:
os.mkdir(errorShots)
except:
pass
errorShotPath = os.path.join(
errorShots,
"ERROR_phantomjs_%s_%s.png" % (cls._testMethodName, datetime.datetime.now().isoformat())
)
cls.browser.save_screenshot(errorShotPath)
print 'Saved screenshot to', errorShotPath
cls.browser.service.process.send_signal(signal.SIGTERM)
cls.browser.quit()
class SetUpClassTest(SeleniumTestCase):
#classmethod
def setUpClass(cls):
print 'Setting Up'
super(SetUpClassTest, cls).setUpClass()
raise Error('gotcha!')
def test1(self):
pass
Output (note that "Tearing Down" doesn't get printed)
$ ./manage.py test
Creating test database for alias 'default'...
Setting Up
E
======================================================================
ERROR: setUpClass (trucks.tests.SetUpClassTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/Users/andy/leased-on/trucks/tests.py", line 1416, in setUpClass
raise Error('gotcha!')
NameError: global name 'Error' is not defined
----------------------------------------------------------------------
Ran 0 tests in 1.034s
FAILED (errors=1)
Destroying test database for alias 'default'...
How can I kill PhantomJS after a suite's setUpClass fails?
I know I could switch to using setUp and addCleanup, but I want to avoid relaunching PhantomJS (and logging back into my app with it) before every single test.
I decided to use setUpModule and tearDownModule to launch and kill PhantomJS. I put the screenshot-saving code in an addCleanup hook.
from django.test import LiveServerTestCase
from selenium import webdriver
import sys
import signal
import os
import unittest
errorShots = os.path.join(os.path.dirname(os.path.abspath(__file__)), '..', "errorShots")
browser = None
def setUpModule():
"""
Launches PhantomJS
"""
global browser
sys.stdout.write('Starting PhantomJS...')
sys.stdout.flush()
browser = webdriver.PhantomJS()
print 'done'
def tearDownModule():
"""
kills PhantomJS
"""
if browser:
sys.stdout.write('Killing PhantomJS...')
sys.stdout.flush()
browser.service.process.send_signal(signal.SIGTERM)
browser.quit()
print 'done'
class SeleniumTestCase(LiveServerTestCase):
def setUp(self):
self.addCleanup(self.cleanup)
def cleanup(self):
"""
Saves a screenshot if the test failed
"""
if sys.exc_info()[0]:
try:
os.mkdir(errorShots)
except:
pass
errorShotPath = os.path.join(
errorShots,
"ERROR_phantomjs_%s_%s.png" % (self._testMethodName, datetime.datetime.now().isoformat())
)
browser.save_screenshot(errorShotPath)
print '\nSaved screenshot to', errorShotPath
Following a tutorial I have written a management command to start tornado and it looks like:
import signal
import time
import tornado.httpserver
import tornado.ioloop
from django.core.management.base import BaseCommand, CommandError
from privatemessages.tornadoapp import application
class Command(BaseCommand):
args = '[port_number]'
help = 'Starts the Tornado application for message handling.'
def sig_handler(self, sig, frame):
"""Catch signal and init callback"""
tornado.ioloop.IOLoop.instance().add_callback(self.shutdown)
def shutdown(self):
"""Stop server and add callback to stop i/o loop"""
self.http_server.stop()
io_loop = tornado.ioloop.IOLoop.instance()
io_loop.add_timeout(time.time() + 2, io_loop.stop)
def handle(self, *args, **options):
if len(args) == 1:
try:
port = int(args[0])
except ValueError:
raise CommandError('Invalid port number specified')
else:
port = 8888
self.http_server = tornado.httpserver.HTTPServer(application)
self.http_server.listen(port, address="127.0.0.1")
# Init signals handler
signal.signal(signal.SIGTERM, self.sig_handler)
# This will also catch KeyboardInterrupt exception
signal.signal(signal.SIGINT, self.sig_handler)
print "start"
tornado.ioloop.IOLoop.instance().start()
print "end"
Here I when I run this management command I get tornado tornado.access:403 GET /2/ (127.0.0.1) 1.92ms error
For test purpose I have printed "start" and "end". I guess when this command is executed successfully "end" should be printed.
Here only "start" is printed not "end". I guess there is something error on tornado.ioloop.IOLoop.instance().start() but I dont know what it is.
Can anyone guide me what is wrong in here ?
You forgot to put self.http_server.start() before starting the ioloop.
...
self.http_server = tornado.httpserver.HTTPServer(application)
self.http_server.listen(port, address="127.0.0.1")
self.http_server.start()
...
Update
Along with the httpserver start missing, are you using this library?
Your management command is exactly this one
Their tutorial is in russian and honestly, I don't read russian.
Update2:
What is see in their code, is that the url /(?P<thread_id>\d+)/ is a websocket handler:
application = tornado.web.Application([
(r"/", MainHandler),
(r'/(?P<thread_id>\d+)/', MessagesHandler),
])
...
class MessagesHandler(tornado.websocket.WebSocketHandler):
...
But the error you posted seems like something tried to access it in http via GET.
Honestly, without a debugger and the same environment, I can't figure out the issue.
I use python 2.7.3 and daemon runner in my script. In a run(loop) method i want to sleep for the some time, but not with the such code:
while True:
time.sleep(10)
I want wait on a some synchronizing primitive, for example multiprocessing.Event. There is my code:
# -*- coding: utf-8 -*-
import logging
from daemon import runner
import signal
import multiprocessing
import spyder_cfg
logging.basicConfig(level=logging.DEBUG, format='%(asctime)s %(name)-12s %(levelname)-8s %(message)s', datefmt='%m-%d %H:%M', filename=spyder_cfg.log_file)
class Daemon(object):
def __init__(self, pidfile_path):
self.stdin_path = '/dev/null'
self.stdout_path = '/dev/tty'
self.stderr_path = '/dev/tty'
self.pidfile_path = None
self.pidfile_timeout = 5
self.pidfile_path = pidfile_path
def setup_daemon_context(self, daemon_context):
self.daemon_context = daemon_context
def run(self):
logging.info('Spyder service has started')
logging.debug('event from the run() = {}'.format(self.daemon_context.stop_event))
while not self.daemon_context.stop_event.wait(10):
try:
logging.info('Spyder is working...')
except BaseException as exc:
logging.exception(exc)
logging.info('Spyder service has been stopped')
def handle_exit(self, signum, frame):
try:
logging.info('Spyder stopping...')
self.daemon_context.stop_event.set()
except BaseException as exc:
logging.exception(exc)
if __name__ == '__main__':
app = Daemon(spyder_cfg.pid_file)
d = runner.DaemonRunner(app)
d.daemon_context.working_directory = spyder_cfg.work_dir
d.daemon_context.files_preserve = [h.stream for h in logging.root.handlers]
d.daemon_context.signal_map = {signal.SIGUSR1: app.handle_exit}
d.daemon_context.stop_event = multiprocessing.Event()
app.setup_daemon_context(d.daemon_context)
logging.debug('event from the main = {}'.format(d.daemon_context.stop_event))
d.do_action()
It is my log file records:
06-04 11:32 root DEBUG event from the main = <multiprocessing.synchronize.Event object at 0x7f0ef0930d50>
06-04 11:32 root INFO Spyder service has started
06-04 11:32 root DEBUG event from the run() = <multiprocessing.synchronize.Event object at 0x7f0ef0930d50>
06-04 11:32 root INFO Spyder is working...
06-04 11:32 root INFO Spyder stopping...
There is not 'Spyder service has been stopped' print in the log, my program hang on the set() call. While debugging i see that it hang when Event.set() call, the set method hang on semaphore while all waiting entities wake up. There is no reason if Event will be global object or threading.Event. I see this one answer, but its not good for me. Is there an alternative for wait with the timeout wait with the same behavior as multiprocessing.Event?
I do print stack from the signal handler and i think there is deadlock, because signal handler use same stack with the my main process and when i call Event.set(), method wait() higher on the stack...
def handle_exit(self, signum, frame):
try:
logging.debug('Signal handler:{}'.format(traceback.print_stack()))
except BaseException as exc:
logging.exception(exc)
d.do_action()
File ".../venv/work/local/lib/python2.7/site-packages/daemon/runner.py", line 189, in do_action
func(self)
File ".../venv/work/local/lib/python2.7/site-packages/daemon/runner.py", line 134, in _start
self.app.run()
File ".../venv/work/skelet/src/spyder.py", line 32, in run
while not self.daemon_context.stop_event.wait(10):
File "/usr/lib/python2.7/multiprocessing/synchronize.py", line 337, in wait
self._cond.wait(timeout)
File "/usr/lib/python2.7/multiprocessing/synchronize.py", line 246, in wait
self._wait_semaphore.acquire(True, timeout)
File ".../venv/work/skelet/src/spyder.py", line 41, in handle_exit
logging.debug('Signal handler:{}'.format(traceback.print_stack()))
that is why this fix solve the problem:
def handle_exit(self, signum, frame):
t = Timer(1, self.handle_exit2)
t.start()
def handle_exit2(self):
self.daemon_context.stop_event.set()
I'm running a BaseHTTPServer, passed through ThreadedHTTPServer so I get threading.
server = ThreadedHTTPServer(('', int(port)), MainHandler)
Next I fork according to the info here: Daemonizing python's BaseHTTPServer
Then I do:
server.serve_forever()
What I am trying to do is have the same Python script run a Tornado WebSocket server as well, I tried creating the second handler and in my main creating the second server similar to above, but then the serve_forever() blocks (I assume) and I can't start the Tornado WebSocket server.
I had considered using Tornado to serve my general web stuff too but performance was aweful and unusable, so I'd prefer to run it alongside, unless there is a simpler alternative to adding WebSockets to the BaseHTTPServer.
Can anyone offer a solution please?
Yes, serve_forever() blocks it all. You can use handle_request to serve one request at a time. To assure it won't block you have to set timeout. To run it periodically you can use tornado.ioloop.PeriodicCallback. Example:
#!/usr/bin/python
# -*- coding: utf-8 -*-
from BaseHTTPServer import HTTPServer, BaseHTTPRequestHandler
from SocketServer import ThreadingMixIn
import threading
import tornado.httpserver
import tornado.ioloop
import tornado.options
import tornado.web
from tornado.options import define, options
define("port", default=8000, help="run on the given port", type=int)
class Handler(BaseHTTPRequestHandler):
def do_GET(self):
self.send_response(200)
self.end_headers()
message = threading.currentThread().getName()
self.wfile.write(message)
self.wfile.write('\n')
return
class ThreadedHTTPServer(ThreadingMixIn, HTTPServer):
"""Handle requests in a separate thread."""
class IndexHandler(tornado.web.RequestHandler):
def get(self):
greeting = self.get_argument('greeting', 'Hello')
self.write(greeting + ', friendly user!\n')
if __name__ == '__main__':
# create Tornado Server
tornado.options.parse_command_line()
app = tornado.web.Application(handlers=[(r"/", IndexHandler)])
http_server = tornado.httpserver.HTTPServer(app)
http_server.listen(options.port)
# create BaseHTTPServer
server = ThreadedHTTPServer(('localhost', 8080), Handler)
server.timeout = 0.01
tornado.ioloop.PeriodicCallback(server.handle_request, 100).start() # every 100 milliseconds
tornado.ioloop.IOLoop.instance().start()
Running:
$ curl http://localhost:8080/
Thread-1
$ curl http://localhost:8080/
Thread-2
$ curl http://localhost:8000/
Hello, friendly user!
$ curl http://localhost:8080/
Thread-3
$ curl http://localhost:8000/
Hello, friendly user!
$ curl http://localhost:8080/
Thread-4
$ curl http://localhost:8000/
Hello, friendly user!
$ curl http://localhost:8000/
Hello, friendly user!
I used here timeout attribute to set timeout. I'm not sure if it's proper way to do it. Other method: http://code.activestate.com/recipes/499376/
Another solution: running every server in its own thread:
#!/usr/bin/python
# -*- coding: utf-8 -*-
from BaseHTTPServer import HTTPServer, BaseHTTPRequestHandler
from SocketServer import ThreadingMixIn
import threading
import tornado.httpserver
import tornado.ioloop
import tornado.options
import tornado.web
from tornado.options import define, options
define("port", default=8000, help="run on the given port", type=int)
class Handler(BaseHTTPRequestHandler):
def do_GET(self):
self.send_response(200)
self.end_headers()
message = threading.currentThread().getName()
self.wfile.write(message)
self.wfile.write('\n')
return
class ThreadedHTTPServer(ThreadingMixIn, HTTPServer):
"""Handle requests in a separate thread."""
class IndexHandler(tornado.web.RequestHandler):
def get(self):
greeting = self.get_argument('greeting', 'Hello')
self.write(greeting + ', friendly user!\n')
def run_tornado():
tornado.options.parse_command_line()
app = tornado.web.Application(handlers=[(r"/", IndexHandler)])
http_server = tornado.httpserver.HTTPServer(app)
http_server.listen(options.port)
tornado.ioloop.IOLoop.instance().start()
def run_base_http_server():
server = ThreadedHTTPServer(('localhost', 8080), Handler)
server.serve_forever()
if __name__ == '__main__':
threading.Thread(target=run_tornado).start()
threading.Thread(target=run_base_http_server).start()