Flask socketio debug with eventlet and Redis spawns extra greenthreads? - flask

Question

I'm trying to put together a simple Flask / socketio / eventlet server that subscribes to Redis events. The behavior I'm seeing is that with Flask debug enabled, every time Werkzeug detects changes and restarts socketio, another one of my redis listeners is started as well (except the old listener doesn't exit).
Here's a working version with all of the socketio handlers removed:
import json
from flask import Flask, render_template
from flask_socketio import SocketIO, emit
from flask.ext.redis import FlaskRedis
import eventlet
eventlet.monkey_patch()
with open('config/flask.json') as f:
config_flask = json.load(f)
app = Flask(__name__, static_folder='public', static_url_path='')
app.config.update(
DEBUG= True,
PROPAGATE_EXCEPTIONS= True,
REDIS_URL= "redis://localhost:6379/0"
)
redis_cache = FlaskRedis(app)
socketio = SocketIO(app)
#app.route('/')
def index():
cache = {}
return render_template('index.html', **cache)
def redisReader():
print 'Starting Redis subscriber'
pubsub = redis_cache.pubsub()
pubsub.subscribe('msg')
for msg in pubsub.listen():
print '>>>>>', msg
def runSocket():
print "Starting webserver"
socketio.run(app, host='0.0.0.0')
if __name__ == '__main__':
pool = eventlet.GreenPool()
pool.spawn(redisReader)
pool.spawn(runSocket)
pool.waitall()
Throw in some manual redis-cli publishing (PUBLISH msg himom)
This produces the following output:
Starting Redis subscriber
Starting webserver
* Restarting with stat
>>>>> {'pattern': None, 'type': 'subscribe', 'channel': 'msg', 'data': 1L}
Starting Redis subscriber
Starting webserver
* Debugger is active!
* Debugger pin code: 789-323-740
(22252) wsgi starting up on http://0.0.0.0:5000
>>>>> {'pattern': None, 'type': 'subscribe', 'channel': 'msg', 'data': 1L}
>>>>> {'pattern': None, 'type': 'message', 'channel': 'msg', 'data': 'himom'}
>>>>> {'pattern': None, 'type': 'message', 'channel': 'msg', 'data': 'himom'}
Why is the Redis listener getting started multiple times? If I make changes and save them, Werkzeug will start another one every time. How do I deal with this correctly?
Here's a list of the packages involved and their versions:
Python 2.7.6
Flask 0.10.1
Werkzeug 0.11.4
eventlet 0.18.4
greenlet 0.4.9
Flask-Redis 0.1.0
Flask-SocketIO 2.2
** UPDATE **
I now have a partial solution. Everything above stays the same, except the pool behavior has been moved into Flask's 'before_first_request' function:
def setupRedis():
print "setting up redis"
pool = eventlet.GreenPool()
pool.spawn(redisReader)
def runSocket():
print "Starting Webserver"
socketio.run(app, host='0.0.0.0')
if __name__ == '__main__':
app.before_first_request(setupRedis)
print app.before_first_request_funcs
runSocket()
The remaining issue is that 'before_first_request' does not handle the case where there are existing websockets, but that is a separate question.

Answer 1

Add simple print(os.getpid()) and observe ps aux. You should notice there are two Python processes. Change DEBUG=False, and the "problem" is not reproducing, also there is one Python process now.
So the problem is that your code creates redisReader no matter if it's a "worker manager" or an actual "request handler" process. So, don't create and start that pool unconditionally. Consult Werkzeug documentation where is your "application init" event, and only start redisReader there.

Answer 2

The solution here was to put my threading into a function called by Flask:
def setupRedis():
pool = eventlet.GreenPool()
pool.spawn(redisReader)
...
app.before_first_request(setupRedis)
This solved the extra threads left behind on Werkzeug restarts.