Django / Azure : serving static files in production - django

I want to deploy a Django website hosted in an Azure Web App.
The static files are served perfectly in DEBUG mode (DEBUG=False), but I can't find the right settings to have the server take care of it in production. All my static files are collected in a "static" directory at the app root wwwroot/static/.
Here is the web.config, at the root of the application.
<?xml version="1.0" encoding="utf-8"?>
<configuration>
<appSettings>
<add key="WSGI_ALT_VIRTUALENV_HANDLER" value="MyApp.wsgi.application" />
<add key="WSGI_ALT_VIRTUALENV_ACTIVATE_THIS" value="D:\home\site\wwwroot\env\Scripts\python.exe" />
<add key="WSGI_HANDLER" value="ptvs_virtualenv_proxy.get_venv_handler()" />
<add key="PYTHONPATH" value="D:\home\site\wwwroot" />
<add key="DJANGO_SETTINGS_MODULE" value="MyApp.settings" />
<add key="WSGI_LOG" value="D:\home\LogFiles\wfastcgit.log"/>
</appSettings>
<system.webServer>
<handlers>
<add name="PythonHandler" path="handler.fcgi" verb="*" modules="FastCgiModule" scriptProcessor="D:\Python34\python.exe|D:\Python34\Scripts\wfastcgi.py" resourceType="Unspecified" requireAccess="Script"/>
</handlers>
<rewrite>
<rules>
<rule name="Static Files" stopProcessing="true">
<conditions>
<add input="true" pattern="false" />
</conditions>
</rule>
<rule name="Configure Python" stopProcessing="true">
<match url="(.*)" ignoreCase="false" />
<action type="Rewrite" url="handler.fcgi/{R:1}" appendQueryString="true" />
</rule>
</rules>
</rewrite>
</system.webServer>
I also added the following web.config in the static directory:
<?xml version="1.0"?>
<configuration>
<system.webServer>
<!-- this configuration overrides the FastCGI handler to let IIS serve the static files -->
<handlers>
<clear />
<add
name="StaticFile"
path="*" verb="*" modules="StaticFileModule,DefaultDocumentModule,DirectoryListingModule"
resourceType="Either"
requireAccess="Read" />
</handlers>
</system.webServer>
I also defined the static directory as a "virtual directory" in the web app settings, with "\static" refering to "\site\wwwroot\static", and checking the "application" checkbox (i also tried to leave it unchecked but it does not change anything.
However, this is not working.
1) Is it possible to setup an Azure web app to serve the static files in production without using a CDN?
2) If yes, how to do it?

It's a good idea to move your static content to Azure Blob Storage and use a CDN for a better performance.
1-) You can define your web.config just like the following: https://github.com/prashanthmadi/azure-django-customdeployment/blob/master/web.config
2-) to configure the static file handler, add the following: https://stackoverflow.com/a/2066040/1384539

I've solved this problem by adding this 2 lines to the top of <handlers> section in root web.config file:
<remove name="Python27_via_FastCGI" />
<remove name="Python34_via_FastCGI" />
Everything else is the same as your's web.config and I didn't need separate config file in static/ directory

I was able to solve the problem after contacting Azure support. This is actually not a trivial problem, and other may have experienced it. The solution below also significantly improves the performances of the app.
According to Microsoft Azure support team:
the default Python version I was using (3.4) creates some problems and it's better to install a new Python version using the web app extensions.
For Azure Web App in Python, it's better to use HTTP Platform Handler than fastcgi.
I was also experiencing very slow loading page. This was due to a high number of Python packages that were loaded when the application restart. for some reason, the default Python version tends to fail, so the application has to restart even if the "Always ON" settings is ON. It seems that Azure actually loads the Python environment from a different disk than the Python code, and this creates a significant slow down.
Don't use a virtual environment
Solution
Here is the procedure I followed to solve the problem(s):
Install Python 3.5 as a web app extension. This can easily be done using the Azure portal for the web application : "Settings > Extensions"
In the Azure portal for the web app, go to "Advanced Tools". Open the Kudu advanced tool, then the CMD console (from the "Debug console" menu at the top of the Kudu page). Navigate to the home directory and find where the new Python has been installed. In my case it was in "D:\home\python353x86"
Now it's time to update the web.config using the new Python version. This path is explained in great details here: https://prashanthmadi.github.io/2016/11/15/django-app-with-httpplatformhandler-in-azure-app-services-windows-new.html. Be careful to replace the processPath="D:\home\Python27\python.exe" from the tutorial with your own Python path. Same thing for the deployment script
Add a new web.config file in your static directory, as described in the same tutorial
Install Python packages, by using the debug console and your "requirements.txt" file, running:
D:\home\python353x86\python.exe -m pip install -r requirements.txt
Replace the python path with your actual path. The python packages are now installed along with your Python exe. Using a virtual environment will slow down the loading of your web page in case the application has to restart.
At this point, your web app should work properly
Additional note:
If you are using webjobs, you need to update the way you are running them.
The solution is to create a "run.cmd" file in your webjob directory, containing :
D:\home\python353x86\python.exe start.py
Where start.py is your webjob script. Once again, replace the python path with your actual path.

If after trying all the other proposed solutions you still find yourself at trouble, you may have to understand that depending on the server that's running you application is the way static files are server. Django has it's own server, which is run by using the command:
python manage.py runserver
But PAAS providers do not use this server in most of the cases. gunicorn is in most times the chosen one. Azure sometimes uses IIS's FastCGI handler but at current date it is intelligent enough to detect a django application and try to use django's default server.
So, the first step you have to take is to find out what server is azure using to run your app, and you may know that by reading azure's log:
https:// YOUR APP NAME.scm.azurewebsites.net/api/logstream
If you confirm that azure is using django's default server, you must bear in mind that django does not server static files automatically when in a production environment. You must configure the static folder's url. So in your config/urls.py set the static url, like this:
from django.conf import settings
from django.conf.urls.i18n import i18n_patterns
from django.conf.urls.static import static
from django.contrib import admin
from django.urls import include, path
urlpatterns = i18n_patterns(path('admin/', admin.site.urls), prefix_default_language=False) + \
static(settings.MEDIA_URL, document_root=settings.MEDIA_ROOT) + \
static(settings.STATIC_URL, document_root=settings.STATIC_ROOT)
As you can see, I enable admin, static and media urls. This is enough to let django's default server know where to find and serve static files (and media files too, as well as admin routes)

Related

Difference between config.py and web.config file when deploying Flask app to Windows IIS

Context Information
I am developing a flask web application at work. Due to existing web infrastructure of my company, I have to deploy the flask app on Windows Server via IIS.
For the development process I worked with the normal Flask Werkzeug development server on my local machine. During the development process, a config.py file (please see below) was created.
Now I am deploying the application to production on IIS. There, a web.config file is necessary for the IIS server to interpret the python code of my app (I am using FastCGI via wfastcgi python module).
Question
I now have two files in my IIS production server app directory that I am not sure of how they interact or relate: config.py and web.config.
Can someone explain to me the differences of those files and if config.py remains relevant for my app in a IIS production environment?
Code
config.py
import os
from dotenv import load_dotenv
basedir = os.path.abspath(os.path.dirname(__name__))
load_dotenv(os.path.join(basedir, '.env'))
# Create the super class
class Config(object):
SECRET_KEY = os.environ.get('SECRET_KEY')
SQLALCHEMY_COMMIT_ON_TEARDOWN = True
SQLALCHEMY_TRACK_MODIFICATIONS = False
# Create the development config
class DevelopmentConfig(Config):
DEBUG = True
# Create the testing config
class TestingConfig(Config):
DEBUG = False
TESTING = True
# create the production config
class ProductionConfig(Config):
DEBUG = False
TESTING = False
web.config file:
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<system.webServer>
<handlers>
<add name="FastCGI Handler" path="*" verb="*" modules="FastCgiModule" scriptProcessor="f:\apps\venv\scripts\python.exe|f:\apps\venv\lib\site-packages\wfastcgi.py" resourceType="Unspecified" requireAccess="Script" />
</handlers>
</system.webServer>
</configuration>
A config.py file contains parameters and initial settings for computer programs. Importing a config.py file allows the variables and functions in the config.py file to be used in the current program. This is the main configuration file, and should not be modified. It can be used as a reference for configuration settings.
The web.config is a file that is read by IIS to configure an app hosted with IIS, and web.config is used for ASP.NET Web Projects/Web Services. web.config by default has several configurations required for the web application. It is also called Application Level Configuration File and inherits setting from the machine.config file.

django rest swagger does not display properly

I want to use django-rest-swagger to document my APIs, so I follow the official doc to setting in my django app. But it doesn't display properly like below:
I was using Nginx+supervisor+gunicorn to serve my django app. Is it possible that they cause it?
The proper page should be like below:
(source: django-rest-framework.org)
I assume that your static files is not configured. You can see this in firebug or chrome dev tools.
Read the documentation how to deploy static files (https://docs.djangoproject.com/en/1.8/howto/static-files/deployment/)
Check that rest_swagger in your INSTALLED_APPS.
Check that STATIC_URL, STATIC_ROOT settings is setted properly.
Call collectstatic management command.
Check that nginx configured to serve static files in STATIC_ROOT folder.

Django on Azure not loading static files

I followed the tutorial below to create a django project on azure:
http://www.windowsazure.com/en-us/develop/python/tutorials/web-sites-with-django/
Everything worked fine until I tried to install the django_admin_bootstrapped app. Now all static requests return 404 error. I don't if the new app caused the problem or just exposed it.
I have this:
STATIC_ROOT = ''
# URL prefix for static files.
# Example: "http://media.lawrence.com/static/"
STATIC_URL = '/static/'
# Additional locations of static files
STATICFILES_DIRS = (
# Put strings here, like "/home/html/static" or "C:/www/django/static".
# Always use forward slashes, even on Windows.
# Don't forget to use absolute paths, not relative paths.
#os.path.join(PROJECT_DIR, 'site-packages/django_admin_bootstrapped/static'),
)
I read quite a bit online and some people say you have to configure your server to serve static files. I don't have direct access to the server, it's an azure website and I deploy through git. And like I said, the admin and everything used to work before I tried to install these apps that bootstrap the admin. Now even when the app is not under INSTALLED_APPS I don't get any css or js files.
Thanks!
I had this problem and none of the suggested answers seemed to fit. My bizarre solution was to switch off Python on the Azure web site configure page.
I arrived at this odd conclusion by installing the PTVS Django sample and following the steps in this tutorial http://azure.microsoft.com/en-us/documentation/articles/web-sites-python-ptvs-django-sql/. The only difference I could fine between my site and the working tutorial was that Django was off! If someone has an explanation I would love to hear it (PHP is enabled!!!).
I found my solution on this page: http://mrtn.me/blog/2012/06/27/running-django-under-windows-with-iis-using-fcgi/
I had to create a central static folder and add a web.config for iis to serve the files. web.config below:
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<system.webServer>
<!-- this configuration overrides the FastCGI handler to let IIS serve the static files -->
<handlers>
<clear/>
<add name="StaticFile" path="*" verb="*" modules="StaticFileModule" resourceType="File" requireAccess="Read" />
</handlers>
</system.webServer>
</configuration>
Hope this helps someone!
I don't have enough rep to vote up but the answer does in fact work on Azure for me as well, and appears to be, so far, the only method of doing it. Just remember that the web.config needs to be in the right place...lol...I apparently had multiple static folders as I was trying different ways of solving this.
If after trying all the other proposed solutions you still find yourself at trouble, you may have to understand that depending on the server that's running you application is the way static files are server. Django has it's own server, which is run by using the command:
python manage.py runserver
But PAAS providers do not use this server in most of the cases. gunicorn is in most times the chosen one. Azure sometimes uses IIS's FastCGI handler but at current date it is intelligent enough to detect a django application and try to use django's default server.
So, the first step you have to take is to find out what server is azure using to run your app, and you may know that by reading azure's log:
https:// YOUR APP NAME.scm.azurewebsites.net/api/logstream
If you confirm that azure is using django's default server, you must bear in mind that django does not server static files automatically when in a production environment. You must configure the static folder's url. So in your config/urls.py set the static url, like this:
from django.conf import settings
from django.conf.urls.i18n import i18n_patterns
from django.conf.urls.static import static
from django.contrib import admin
from django.urls import include, path
urlpatterns = i18n_patterns(path('admin/', admin.site.urls), prefix_default_language=False) + \
static(settings.MEDIA_URL, document_root=settings.MEDIA_ROOT) + \
static(settings.STATIC_URL, document_root=settings.STATIC_ROOT)
As you can see, I enable admin, static and media urls. This is enough to let django's default server know where to find and serve static files (and media files too, as well as admin routes)
Yes, I can confirm that the 'right place' to store your static files for a Django deployment on Azure Web Sites is the 'static' directory in the web root.
Step 1: create a directory called static in your web root (www root)
Step 2: edit Django's settings.py and set STATIC_ROOT = 'D:/home/site/wwwroot/static'
Step 3: In your templates use the construct {% static "static_file_name" %}.

Implementing search with Haystack and Solr issues

I'm trying to implement search to my django based website.
While following the tutorial I found this:
If you’re using the Solr backend, you have an extra step. Solr’s
configuration is XML-based, so you’ll need to manually regenerate the
schema. You should run ./manage.py build_solr_schema first, drop the
XML output in your Solr’s schema.xml file and restart your Solr
server.
First, I don't know where to put my schema.xml, after some resarch I figured I'd create a folder inside my project to put it: myprojectname/solr/schema.xml. Is that right?
Second, how do I restart Solr?
UPDATE
I downloaded Solr,unzipped it and I put the schema.xml generated inside example/solr/conf
then I start solr java -jar start.jar
but when I try to build the index :
./manage.py rebuild_index
I get :
WARNING: This will irreparably remove EVERYTHING from your search index.
Your choices after this are to restore from backups or rebuild via the `rebuild_index` command.
Are you sure you wish to continue? [y/N] y
Removing all documents from your index because you said so.
All documents removed.
Indexing 1 News.
Failed to add documents to Solr: [Reason: None]
<response><lst name="responseHeader"><int name="status">400</int><int name="QTime">4</int></lst><lst name="error"><str name="msg">ERROR: [doc=news.news.2] unknown field 'django_id'</str><int name="code">400</int></lst></response>
Indexing 1 entries.
Failed to add documents to Solr: [Reason: None]
<response><lst name="responseHeader"><int name="status">400</int><int name="QTime">17</int></lst><lst name="error"><str name="msg">ERROR: [doc=zinnia.entry.2] unknown field 'django_id'</str><int name="code">400</int></lst></response>
I verified my schema.xml ,and I do have :
<field name="django_ct" type="string" indexed="true" stored="true" multiValued="false" />
<field name="django_id" type="string" indexed="true" stored="true" multiValued="false" />
P.S.
I'm using Django 1.2 and Haystack 1.2.7
The solr server needs to have a copy of your schema.xml not django. I usually keep a copy of the schema.xml in my django project for version control, but solr can't find it there.
Is you solr server local? Are you using a hosted or remote Solr service? I develop locally then use websolr b/c i dont want to configure solr for production.
For local dev on OSX
I'm assuming this is local development on OSX and that you have homebrew installed (assumptions - give me more info if this isnt the case):
brew install solr
This is going to install Solr at someplace like: /usr/local/Cellar/solr/...
Note: When im developing locally, I like to use fabric for running deployment and some startup tasks.
So in my fabfile.py I have a fabric command to copy my schema.xml into the proper file and start the solr server (I just run fab solr at the cmd line)
def solr() :
# build a new updated schema.xml (changes to indexes/models may require this so always do it for local testing)
local('python manage.py build_solr_schema > schema.xml')
# copy the schema.xml into the proper directory
local('cp schema.xml /usr/local/Cellar/solr/3.6.0/libexec/example/solr/conf/schema.xml')
# start the solr server
local('cd /usr/local/Cellar/solr/3.6.0/libexec/example && java -jar start.jar')
Note: you can run these commands on the command line if you dont use fabric
I had the same issue, the rebuild task failed. For me the solution was:
Build a new schema.xml and place it in the corresponding folder
Restart Solr
rebuild the index without problems

Deploying a Django App

I'm very sorry for such a simple question-- I'm new at WSGI development, and I'm grateful for any patience you can afford.
I made a Django app; it works great in development mode. I run:
python manage.py runserver
and then direct my browser to 127.0.0.1:8000, and voila, there is my app.
From here I absolutely cannot figure out how to run my app in production mode. I've read several pages like this and this and several others on StackOverflow. But I have no idea of where to even direct my browser to see if my page is working.
I've installed apache2, mod_python, etc., but I think the problem is that my misunderstanding is at such a more basic level. When I've done CGI programs in the past, I direct my browser to webroot/file.html with a form that calls cgi-bin/file.cgi, which generates html output. I don't know if I am supposed to navigate to a .wsgi page, etc.
Under the assumption that I'm supposed to navigate to a .wsgi file, I've also tried making a file containing:
import os
import sys
os.environ['DJANGO_SETTINGS_MODULE'] = 'mysite.settings'
import django.core.handlers.wsgi
application = django.core.handlers.wsgi.WSGIHandler()
path = '/home/orserang/nonparametric-protein/src/www/mysite$'
if path not in sys.path:
sys.path.append(path)
and added
WSGIScriptAlias / /path/to/mysite/apache/django.wsgi
to my apache2/httpd.conf file, so that its contents are:
<Location "/mysite/">
SetHandler python-program
PythonHandler django.core.handlers.modpython
SetEnv DJANGO_SETTINGS_MODULE mysite.settings
PythonOption django.root /mysite
PythonDebug On
PythonPath "['/home/orserang/nonparametric-protein/src/www/mysite'] + ['/home/orserang/nonparametric-protein/src/'] + sys.path"
WSGIScriptAlias /mysite /home/orserang/nonparametric-protein/src/www/mysite/django.wsgi
</Location>
But when I restart apache, it says:
Syntax error on line 8 of /etc/apache2/httpd.conf:
WSGIScriptAlias not allowed here
Given that I don't even know where I should point my browser to get to a Django wsgi page, I figured there is something easy that I'm doing quite wrong.
Perhaps Django WSGI apps require something to run in the background, which will listen for requests (rather than go through apache)?
The online Django documentation on views and databases alone are substantial compared to the documentation for deployment; therefore, my best guess is that this is such a simple thing to do.
Thanks a lot for your help!
The Django Book 2.0 has an overview about this. It's not typically linked to in the Django docs:
Chapter 12: Deploying Django
Look at the "Using Django with Apache and mod_python" section.
You're mixing up mod_python and mod_wsgi deployment methods. Get rid of everything inside the Location directive except for the WSGIScriptAlias line.
I wrote shell script that deploys a django project on apache for linux,
https://github.com/mukulu/shell-scripts/blob/master/deploy-django.sh
You only need to configure couple of variables in first lines of the code,
and it'll figure out the rest.
It pretty much checks and install dependencies for django, writes apache configurations that deploys your project and restart the server.
I'm planning to re-write it in python(I wrote it in a hurry)
Feel free to re-use.
Variables are:
SITE_PREFIX="/djangoproject"
MEDIA_URL="/media"
ADMIN_MEDIA_PREFIX="/static/admin/"
MEDIA_ROOT=""
DJANGO_VERSION="1.3.1"
APACHE2_CONFIG="/etc/apache2/conf.d" #Apache configurations directory in yoru system.
This might be not for a novice, but you can take a look - http://packages.python.org/django-fab-deploy/
It's the library for automating the deploying process. It supports servers based primarily on Debian Lenny or Squeeze.