I'm writing my first Flask application. The application itself runs fine. I just have a newbie question about logging in production mode.
The basic structure:
app/
app/templates/
app/static
config.py
flask/... <- virtual env with flask + extensions
run.py
The application is started by run.py script:
#!flask/bin/python
import os.path
import sys
appdir = os.path.dirname(os.path.abspath(__file__))
if appdir not in sys.path:
sys.path.insert(1, appdir)
from app import app as application
if __name__ == '__main__':
application.run(debug=True)
and is started either directly or from an Apache 2.4 web server. I have these lines in the apache config:
WSGIPythonHome /usr/local/opt/app1/flask
WSGIScriptAlias /app1 /usr/local/opt/app1/run.py
In the former case, the debug=True is all I need for the development.
I'd like to have some logging also for the latter case, i.e. when running under Apache on a production server. Following is a recommendation from the Flask docs:
if not app.debug:
import logging
from themodule import TheHandlerYouWant
file_handler = TheHandlerYouWant(...)
file_handler.setLevel(logging.WARNING)
app.logger.addHandler(file_handler)
It needs some customization, but that's what I want - instructions for the case when app.debug flag is not set. Similar recommendation was given also here:
How do I write Flask's excellent debug log message to a file in production?
Please help: where do I have to put this code?
UPDATE: based on the comments by davidism and the first answer I've got I think the app in the current simple form is not suitable for what I was asking for. I will modify it to use different sets of configuration data as recommended here: http://flask.pocoo.org/docs/0.10/config/#development-production . If my application were larger, I would follow the pech0rin's answer.
UPDATE2: I think the key here is that the environment variables should control how the application is to be configured.
I have had a lot of success with setting up my logging configurations inside a create_app function. This uses the application factory pattern. This allows you to pass in some arguments or a configuration class. The application is then specifically created using your parameters.
This allows you initialize the application, setup logging, and do whatever else you want to do, before the application is sent back to be run.
For example:
def create_app(dev=False):
app = Flask(__name__)
if dev:
app.config['DEBUG'] = True
else:
...
app.logger.addHandler(file_handler)
return app
This has worked very well for me in production environments. YMMV
Related
Deploying my django website with S3 as storage which runs fine locally to pythonanywhere gives a strange error I can't google a solution for:
"TypeError: a bytes-like object is required, not 'str'"
What I'm doing wrong?
I've tried to put my environment variables out of settings.env (aws keys, secret_key, etc) ad set them directly in my settings.py app. + every suggestion I could find but it's still the same :(
here's my /var/www/username_pythonanywhere_com_wsgi.py:
# +++++++++++ DJANGO +++++++++++
# To use your own Django app use code like this:
import os
import sys
from dotenv import load_dotenv
project_folder = os.path.expanduser('~/portfolio_pa/WEB') # adjust as appropriate
load_dotenv(os.path.join(project_folder, 'settings.env'))
# assuming your Django settings file is at '/home/myusername/mysite/mysite/settings.py'
path = '/home/corebots/portfolio_pa'
if path not in sys.path:
sys.path.insert(0, path)
os.environ['DJANGO_SETTINGS_MODULE'] = 'WEB.settings'
## Uncomment the lines below depending on your Django version
###### then, for Django >=1.5:
from django.core.wsgi import get_wsgi_application
application = get_wsgi_application()
###### or, for older Django <=1.4
#import django.core.handlers.wsgi
#application = django.core.handlers.wsgi.WSGIHandler()
I'd expect the website to run fine just like it does locally.
Boto library doesn't have a good Python3 support. This particular issue is known in the boto bugtracker: https://github.com/boto/boto/issues/3837
The best way of fixing this is to use boto3 which has decent Python3 support and is a generally most supported AWS SDK for Python.
The reason why it works on your local machine and doesn't work on production is that pythonanywhere setup seems to be using proxy which triggers this incompatible boto code. See the actual calling code: https://github.com/boto/boto/blob/master/boto/connection.py#L747
Your error traceback confirms this.
Unfortunately, I'm not familliar with the django-photologue, but a brief look doesn't suggest that it strongly depends on boto3. Maybe I'm wrong.
I still think that the best way is to go with boto3. As a backup strat you can fork boto with a fix for this issue and install that instead of the official one from PyPI: https://github.com/boto/boto/pull/3699
I developed my first flask application that is currently running and everything work fine. The application is loaded using "python application.py" or using gUnicorn. There is no fancy fancy folder structure everything is in the same folder, with exception of the static\ and templates\
Folder Structure:
- application\hello.py
- application\static\
- application\templates\
To Run:
- python hello.py
#hello.py
from flask import Flask
app = Flask(__name__)
#app.route('/')
def hello_world():
return 'Hello, World!'
if __name__ == '__main__':
app.run(host='0.0.0.0', load_dotenv=True, debug=True, threaded=True)
Now that looking to add more functionalities to the application noticed most tutorials uses "flask run" to launch the instead. There are two different tutorials I am following one using blueprints and the other one is the microblog tutorial both using similar folder structure:
- application\run.py
- application\app\main.py
- application\app\static\
- application\app\templates\
To run:
- export Flask_APP=run.py
- flask run
The application will continue to grow and I want to follow best practices.
Question 1:
How do I enable the following parameters when using "flask run"??:
if __name__ == '__main__':
app.run(host='0.0.0.0', load_dotenv=True, debug=True, threaded=True)
Question 2:
Is there any pro/cons configuring the app to run using flask run vs python app.py????? There was another post with this title but context was unrelated.
When can I read more about this??
Threaded mode is enabled by default. You don't have to pass it in. Source
For debug mode, use export FLASK_DEBUG=1. Source
For load_dotenv use export FLASK_SKIP_DOTENV=0 Source
For specifying port use export FLASK_RUN_PORT=8000. Source
To bind the app to 0.0.0.0, set SERVER_NAME config like app.config['SERVER_NAME']. Source
Also see:
http://flask.pocoo.org/docs/1.0/cli/#setting-command-options
I'm trying to separate out Django's secret key and DB pass into environmental variables, as widely suggested, so I can use identical code bases between local/production servers.
The problem I am running into is correctly setting and then reading the environmental vars on the production server running Apache + mod_wsgi.
Vars set in my user profile aren't available because Apache isn't run as that user. Vars set in the Virtual Hosts file with SetEnv are not available because the scope is somehow different.
I've read a couple 1,2 of SO answers, which leads to this blog with a solution.
I can't figure out how to apply the solution to current versions of Django which use a wsgi.py file, which looks like:
import os
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "project.settings")
from django.core.wsgi import get_wsgi_application
application = get_wsgi_application()
How can I apply that blog solution to the wsgi.py file, or is there a better place to store env-vars where Django can get at them?
If anyone else is frustrated by Graham's answer, here is a solution that actually works for the original question. I personally find setting environmental variables from Apache to be extremely useful and practical, especially since I configure my own hosting environment and can do whatever I want.
wsgi.py (tested in Django 1.5.4)
from django.core.handlers.wsgi import WSGIHandler
class WSGIEnvironment(WSGIHandler):
def __call__(self, environ, start_response):
os.environ['SETTINGS_CONFIG'] = environ['SETTINGS_CONFIG']
return super(WSGIEnvironment, self).__call__(environ, start_response)
application = WSGIEnvironment()
Of minor note, you lose the future-proof method of django.core.wsgi.get_wsgi_application, which currently only returns WSGIHandler(). If the WSGIHandler.__call__ method is ever updated and you update Django also, you may have to update the WSGIEnvironment class if the arguments change. I consider this a very small penalty to pay for the convenience.
FWIW. Relying on environment variables for fine grained configuration settings are in general not a good idea. This is because not all WSGI hosting environments or commercial PaaS offerings support the concept. Using environment variables for fine grained settings can also effectively lock you into a specific PaaS offering where you have directly embedded a lookup of a specifically named environment variable directly into your code, where the naming convention of that environment variable is specific to that hosting service. So although use of environment variables is pushed by certain services, always be careful of being dependent on environment variables as it will reduce the portability of your WSGI application and make it harder to move between deployment mechanisms.
That all said, the blog post you mention will not usually help. This is because it is using the nasty trick of setting the process environment variables on each request based on the per request WSGI environ settings set using SetEnv in Apache. This can cause various issues in a multi threading configuration if the values of the environment variables can differ based on URL context. For the case of Django, it isn't helpful because the Django settings module would normally be imported before any requests had been handled, which means that the environment variables would not be available at the time required.
This whole area of deployment configuration is in dire need of a better way of doing things, but frankly it is mostly a lost cause because hosting services will not change things to accommodate a better WSGI deployment strategy. They have done their work, have their customers locked into the way they have done it already and aren't about to create work for themselves and change things even if a better way existed.
Anyway, 'all problems in computer science can be solved by another level of indirection'. (http://en.wikipedia.org/wiki/Indirection) and that is what you can do here.
Don't have your application lookup environment variables. Have it import a deployment specific Python configuration module which contains a means of using an API to get the configuration settings. This configuration module would implement different ways of getting the actual settings based on the deployment mechanism. In some cases it could grab the values from environment variables. For other such as with Apache/mod_wsgi, the values could be in that configuration module, or read from a separate configuration file which could be an ini, json or yaml format. In providing a API, it can also map names of configuration settings to different names used by different PaaS offerings.
This configuration module doesn't need to be a part of your application code, but could be manually place into a subdirectory of '/etc/' on the target system. You just then need to set the Python module search path so your application can see it. The whole system could be made quite elegant as part of a wider better standard for WSGI deployment, but as I said, little incentive to do the hard work of creating such a thing when existing PaaS offerings are highly unlikely to change to use such a standard.
Here's an alternative solution that's as future-proof as get_wsgi_application. It even lets you set environment variables to use in your Django initialization.
# in wsgi.py
KEYS_TO_LOAD = [
# A list of the keys you'd like to load from the WSGI environ
# into os.environ
]
def loading_app(wsgi_environ, start_response):
global real_app
import os
for key in KEYS_TO_LOAD:
try:
os.environ[key] = wsgi_environ[key]
except KeyError:
# The WSGI environment doesn't have the key
pass
from django.core.wsgi import get_wsgi_application
real_app = get_wsgi_application()
return real_app(wsgi_environ, start_response)
real_app = loading_app
application = lambda env, start: real_app(env, start)
I'm not 100% clear how mod_wsgi manages its processes, but I assume it doesn't re-load the WSGI app very often. If so, the performance penalty from initializing Django will only happen once, inside the first request.
Alternatively, if you don't need to set the environment variables before initializing Django, you can use the following :
# in wsgi.py
KEYS_TO_LOAD = [
# A list of the keys you'd like to load from the WSGI environ
# into os.environ
]
from django.core.wsgi import get_wsgi_application
django_app = get_wsgi_application()
def loading_app(wsgi_environ, start_response):
global real_app
import os
for key in KEYS_TO_LOAD:
try:
os.environ[key] = wsgi_environ[key]
except KeyError:
# The WSGI environment doesn't have the key
pass
real_app = django_app
return real_app(wsgi_environ, start_response)
real_app = loading_app
application = lambda env, start: real_app(env, start)
For Django 1.11:
Apache config:
<VirtualHost *:80 >
...
SetEnv VAR_NAME VAR_VALUE
</VirtualHost>
wsgi.py:
import os
import django
from django.core.handlers.wsgi import WSGIHandler
class WSGIEnvironment(WSGIHandler):
def __call__(self, environ, start_response):
os.environ["VAR_NAME"] = environ.get("VAR_NAME", "")
return super(WSGIEnvironment, self).__call__(environ, start_response)
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "project.settings")
django.setup(set_prefix=False)
application = WSGIEnvironment()
I ran into the same problem of separating production and development code while tracking both of them into version control. I solved the problem using different wsgi scripts, one for production server and one for development server. Create two different setting files as mentioned here. And reference them in wsgi scripts. For example the following is wsgi_production.py file
...
import os
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "project.settings.production")
...
and in wsgi_development.py file
...
import os
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "project.settings.development")
...
I'm trying to use server push with my Django app. Of all the different solutions I've seen, django-socketio seemed to be the easiest to implement, and I got it working when launching the server through manage.py. However, when it goes to production, I'd like it to be served through Apache. In the past, I've done this with wsgi, but django-socketio isn't playing nicely with the default wsgi script. Is there something simple I can just change in my django.wsgi that lets apache do the right thing? If not, what would be the suggested way to handle this?
EDIT: Here's the WSGI script I was normally using (without any kind of push server), plus a bit more explanation.
import os, sys
locale = os.path.realpath(__file__)
ROOT_DIR = locale[:locale.find('/server/apache/')]
sys.path.append(ROOT_DIR)
sys.path.append(ROOT_DIR+'/server')
os.environ['DJANGO_SETTINGS_MODULE'] = 'server.settings'
os.environ['PYTHON_EGG_CACHE'] = '/var/www/.python-eggs'
import django.core.handlers.wsgi
application = django.core.handlers.wsgi.WSGIHandler()
sys.stdout = sys.stderr
The sys.stdout = sys.stderr is just to redirect any test print statements to apache's error log. The problem is, when I use this, I get an error complaining that request.environ doesn't have the key "socketio" or "DJANGO_SOCKETIO_PORT". I can add DJANGO_SOCKETIO_PORT easily enough (os.environ['DJANGO_SOCKETIO_PORT']="9000"), but from what I can tell, request.environ['socketio'] is set to an instance of SocketIOProtocol somewhere in django-socketio's internals. Also, after looking at the command that django-socketio added to manage.py, I noticed that it creates an instance of SocketIOServer, and calls serve_forever on it, but I have no idea where to put that in my code. Hopefully this will make it easier to see what I'm trying to get done.
Last two day I'm trying to find a way to run a working django project under twisted. After detailed searching I found several methods to configure it. But most of them are deal with how to run the app via command line not as a daemon.I want to run the django project as daemon.
I tried following links to implement this,
Twisted: Creating a ThreadPool and then daemonizing leads to uninformative hangs
http://www.robgolding.com/blog/2011/02/05/django-on-twistd-web-wsgi-issue-workaround/
But this also not working for me.By this method TCP server is not even listen to the given port.
Please help me to figure it out.
UPDATE
I'm sorry for the missing informations.Here is my objectives.
I'm beginner in twisted world, so first I'm trying to get my working django project configured under twisted,currently its working well on django testing server or apache via mod_wsgi.
To configure it with twisted I used the biding code given below, that code is a combination of two sample's found in the links that I given in the first post.
So in-order to integrate django app with twisted I used the following code, file name: "server.py".
import sys
import os
from twisted.application import internet, service
from twisted.web import server, resource, wsgi, static
from twisted.python import threadpool
from twisted.internet import reactor
from django.conf import settings
import twresource # This file hold implementation of "Class Root".
class ThreadPoolService(service.Service):
def __init__(self, pool):
self.pool = pool
def startService(self):
service.Service.startService(self)
self.pool.start()
def stopService(self):
service.Service.stopService(self)
self.pool.stop()
class Root(resource.Resource):
def __init__(self, wsgi_resource):
resource.Resource.__init__(self)
self.wsgi_resource = wsgi_resource
def getChild(self, path, request):
path0 = request.prepath.pop(0)
request.postpath.insert(0, path0)
return self.wsgi_resource
PORT = 8080
# Environment setup for your Django project files:
#insert it to first so our project will get first priority.
sys.path.insert(0,"django_project")
sys.path.insert(0,".")
os.environ['DJANGO_SETTINGS_MODULE'] = 'django_project.settings'
from django.core.handlers.wsgi import WSGIHandler
def wsgi_resource():
pool = threadpool.ThreadPool()
pool.start()
# Allow Ctrl-C to get you out cleanly:
reactor.addSystemEventTrigger('after', 'shutdown', pool.stop)
wsgi_resource = wsgi.WSGIResource(reactor, pool, WSGIHandler())
return wsgi_resource
# Twisted Application Framework setup:
application = service.Application('twisted-django')
# WSGI container for Django, combine it with twisted.web.Resource:
# XXX this is the only 'ugly' part: see the 'getChild' method in twresource.Root
wsgi_root = wsgi_resource()
root = Root(wsgi_root)
#multi = service.MultiService()
#pool = threadpool.ThreadPool()
#tps = ThreadPoolService(pool)
#tps.setServiceParent(multi)
#resource = wsgi.WSGIResource(reactor, tps.pool, WSGIHandler())
#root = twresource.Root(resource)
#Admin Site media files
#staticrsrc = static.File(os.path.join(os.path.abspath("."), "/usr/haridas/eclipse_workplace/skgargpms/django/contrib/admin/media/"))
#root.putChild("admin/media", staticrsrc)
# Serve it up:
main_site = server.Site(root)
#internet.TCPServer(PORT, main_site).setServiceParent(multi)
internet.TCPServer(PORT, main_site).setServiceParent(application)
#EOF.
Using above code It worked well from command line using "twisted -ny server.py", but when we run it as daemon "twisted -y server.py" it will hang, but the app is listening to the port 8080. I can access it using telnet.
I found some fixes for this hanging issue from stackoverflow itself. It helped me to use the code sections given below, which is commented in the above server.py file.
multi = service.MultiService()
pool = threadpool.ThreadPool()
tps = ThreadPoolService(pool)
tps.setServiceParent(multi)
resource = wsgi.WSGIResource(reactor, tps.pool, WSGIHandler())
root = twresource.Root(resource)
and :-
internet.TCPServer(PORT, main_site).setServiceParent(multi)
instead of using the:-
wsgi_root = wsgi_resource()
root = Root(wsgi_root)
and :-
internet.TCPServer(PORT, main_site).setServiceParent(application)
The modified method also didn't helped me to avoid the hanging issue.Is any body out there who successfully run the django apps under twisted daemon mode?.
I maid any mistakes while combining these codes?, Currently I'm only started to learn the twisted architectures in detail. Please help me to solve this problem
Thanks and Regards,
Haridas N.
Note:- Im looking for the Twisted Application configuration (TAC) file, which integrate django app with twisted and run with out any problem in the daemon mode also.
Thank you,
Haridas N.
twistd is the Twisted Daemonizer. Anything you run with twistd will be easy to daemonize. All you have to do is not pass the --nodaemon option.
As far as why your code is "not working", you need to provide more details about what you did, what you expected to happen, and how what actually happened differed from your expectations. Otherwise, only a magician can answer your question.
Since you said the TCP port doesn't even get set up, the only guess I can think of is that you're trying to listen on a privileged port (such as 80) without having permissions to do so (ie, you're not root and you're not using authbind or something similar).