Can manage.py runserver execute npm scripts? - django

I am developing a web application with React for frontend and Django for backend. I use Webpack to watch for changes and bundle code for React apps.
The problem is that I have to run two commands concurrently, one for React and the other one for Django:
webpack --config webpack.config.js --watch
./manage.py runserver
Is there any way to customize runserver command to execute the npm script, like npm run start:dev? When you use Node.js as a backend platform, you can do the similar job like npm run build:client && npm run start:server.

If you are already using webpack and django, probably you can be interested in using webpack-bundle-tracker and django-webpack-loader.
Basically webpack-bundle-tracker will create an stats.json file each time the bundle is build, and django-webpack-loader will watch for those stats.json file to relaunch the dev server. This stack allows to separate the concerns between the server and the client.
There are a couple of posts out there explaining this pipeline.

I'm two and a half years late, but here's a management command that implements the solution that OP wanted, rather than a redirection to another solution. It inherits from the staticfiles runserver and runs webpack concurrently in a thread.
Create this management command at <some_app>/management/commands/my_runserver.py:
import os
import subprocess
import threading
from django.contrib.staticfiles.management.commands.runserver import (
Command as StaticFilesRunserverCommand,
)
from django.utils.autoreload import DJANGO_AUTORELOAD_ENV
class Command(StaticFilesRunserverCommand):
"""This command removes the need for two terminal windows when running runserver."""
help = (
"Starts a lightweight Web server for development and also serves static files. "
"Also runs a webpack build worker in another thread."
)
def add_arguments(self, parser):
super().add_arguments(parser)
parser.add_argument(
"--webpack-command",
dest="wp_command",
default="webpack --config webpack.config.js --watch",
help="This webpack build command will be run in another thread (should probably have --watch).",
)
parser.add_argument(
"--webpack-quiet",
action="store_true",
dest="wp_quiet",
default=False,
help="Suppress the output of the webpack build command.",
)
def run(self, **options):
"""Run the server with webpack in the background."""
if os.environ.get(DJANGO_AUTORELOAD_ENV) != "true":
self.stdout.write("Starting webpack build thread.")
quiet = options["wp_quiet"]
command = options["wp_command"]
kwargs = {"shell": True}
if quiet:
# if --quiet, suppress webpack command's output:
kwargs.update({"stdin": subprocess.PIPE, "stdout": subprocess.PIPE})
wp_thread = threading.Thread(
target=subprocess.run, args=(command,), kwargs=kwargs
)
wp_thread.start()
super(Command, self).run(**options)
For anyone else trying to write a command that inherits from runserver, note that you need to check for the DJANGO_AUTORELOAD_ENV variable to make sure you don't create a new thread every time django notices a .py file change. Webpack should be doing it's own auto-reloading anyway.
Use the --webpack-command argument to change the webpack command that runs (for example, I use --webpack-command 'vue-cli-service build --watch'
Use --webpack-quiet to disable the command's output, as it can get messy.
If you really want to override the default runserver, rename the file to runserver.py and make sure the app it lives in comes before django.contrib.static in your settings module's INSTALLED_APPS.

You shouldn't mess with the built-in management commands but you can make your own: https://docs.djangoproject.com/en/1.10/howto/custom-management-commands/.
On your place I'd leave runserver in place and create one to run your custom (npm in this case) script, i.e. with os.execvp.
In theory you could run two parallel subprocesses one that would execute for example django.core.management.execute_from_command_line and second to run your script. But it would make using tools like pbd impossible (which makes work very hard).
The way I do it is that I leverage Docker and Docker compose. Then when I use docker-compose up -d my database service, npm scripts, redis, etc run in the background (running runserver separately but that's another topic).

Related

How to have a Django app running on Heroku doing a scheduled job using Heroku Scheduler

I am developing a Django app running on Heroku.
In order to update my database with some data coming from a certain API service, I need to periodically run a certain script (let's say myscript).
How can I use Heroku Scheduler to do it?
As already explained here, quick and simple way to answer this question is asking yourself how would you do to run that script periodically, as if you were the scheduler yourself.
Now, the best way to run a script in your Django app at any moment, it is to create a custom management command and to run it from your command prompt when you need it, like this:
python manage.py some_custom_command
Then, if you were the scheduler, you would run that command from your command prompt at every time written in the schedule.
So, a good idea would be to make Heroku Scheduler behave the same. Thus, the aim here is to have Heroku Scheduler run python manage.py some_custom_command at scheduled times.
Here is how you can do it:
In your_app directory, create a folder management and then inside it create another folder commands and finally, inside it, create a file some_custom_command.py
So, just to be clear
your_app/management/commands/some_custom_command.py
Then, inside some_custom_command.py insert:
from django.core.management.base import BaseCommand
from your_app.path_to_myscript_file import myscript
class Command(BaseCommand):
def handle(self, *args, **options):
# Put here some script to get the data from api service and store it into your models.
myscript()
Then go on Heroku > your_app > resources
In add-ons section select Heroku Scheduler, click on it so that its window opens, then click on add job, select the time you want, insert the command python manage.py some_custom_command and save.

Django Deployment Process to Webfaction.com

Trying to streamline a deployment process to webfaction.com for my django application, I have a master (working copy) and a development branch.
currently I'm doing the following:
Make changes to my development branch in my local dev environment
When changes are working, test with run local server, then merge with my master branch
git push so the code is in my remote repo (this has other issues such as passwords, keys etc which I've not quite solved yet) (also i dont believe its possible to scp code to webfaction and I'm not really a fan of any of the FTP services I've used so far)
SSH into my webfaction server and do a git pull and git merge
Test to see if everything is still working (it never is)
Make anychanges required to get everything working again
commit any changes I've had to do to fix everything then push back to the remote repo
Go back to my development environment and sync the code up with the production code
Rinse Repeat for the next feature
obviously I've missed the efficient development train, for the record I've only been working with django for a couple of months as a hobby project.
Can anyone suggest a django deployment process that would be more conducive to sane development?
I would strongly suggest Fabric to handle your deployments to WebFaction:
http://docs.fabfile.org/en/1.11/tutorial.html
By using Fabric you can deploy code and do other server side operations from your local terminal with no need to manually ssh to the server. First install Fabric:
pip install Fabric
Create fabfile.py in your project root folder. Here is an example fabfile that can get you started:
from fabric.api import task, env, run, cd
from fabric.context_managers import prefix
env.hosts = ('wf_username#wf_username.webfactional.com',)
env.forward_agent = True
MANAGEPY = '~/webapps/my_project/code/my_project/manage.py'
PY = '~/webapps/my_project/env/bin/python2.7'
#task
def deploy():
with cd('~/webapps/my_project/code/'):
with prefix('source production'):
run('git pull --rebase origin master')
run('pip install -r requirements.txt')
run('{} {} migrate'.format(PY, MANAGEPY))
run('{} {} collectstatic --noinput'.format(PY, MANAGEPY))
run('touch my_project/my_project/wsgi.py')
You can run fab task from your terminal with:
fab deploy
In my opinion, making code changes directly on server is a bad practice. Fabric can improve your development flow so that you make code edits only locally, quickly deploy them and test them.
The best and shortest way
In settings.py:
try:
from production_settings import *
except ImportError as e:
pass
You can override what needed in production_settings.py; it should stay out of your version control and you can use git resourcefully.

create dockerfile to build new images

FROM denmarkcontrevida/base:15.05
MAINTAINER Denmark Contrevida<denmarkcontrevida#esutek.com>
# Config files
# Config pyenv
# Config Nginx
# Config PostgreSQL
# Create DB & Restore database
This image will install to the newest version.
PostgreSQL
Nginx
Pyenv
Django
Python 3
IF are about to install that many different services, make sure to start from a base image made to manage them.
Use phusion/baseimage-docker (which always starts the my_init script, to take care of the zombie processes)
In that image, you can define multiple program (daemon) to run:
You only have to write a small shell script which runs your daemon, and runit will keep it up and running for you, restarting it when it crashes, etc.
The shell script must be called run, must be executable, and is to be placed in the directory /etc/service/<NAME>.
If your base image has a /etc/service/helper/run script, then any image based on it would run helper, plus any other /etc/service/xxx/run script of your own: replace xxx by the running services like nginx, django, postgresSQL.
You wouldn't need it for python3 (which is simply called, but doesn't run in the background)

jenkins start django server after successful build

We use jenkins as continious integration system. We have two django servers validated by jenkins.
jenkins validates successully the first server. The second server depends on the first one. Thus we would like to launch at the end of the first server validation the first server itself.
We are using python, virtualenv and django and defined the Virtualenv Builder as follow:
pip install -r requirements.txt
rm -f .coverage
fab localhost test
coverage xml
nohup python manage.py runserver 9090 &
The issue is that the build never ends due to the nohup.
How can I launch the server after a successful build?
I had the same problem.
Ken,
I tried using fabric, but again python manage.py runserver - runs continuosly, so the next command is not starting.
And just few mins ago my collegue showed me how to use nohup and with variable BUILD_ID of Jenkins it would be like this to get Success from the build and leave the Django server running:
BUILD_ID=dontKillMe nohup python manage.py runserver host_server &
This worked for our Django project testing.
Since you are using fabric to test, I would recommend defining another fabric task, say, deploy, which you could call assuming the build succeeds.
Much like the call to fab completes for a successful build such that you get to the nohup line, I would expect the deploy task to return also.
You may also want to consider making the server a service (either via an /etc/init.d style script, or upstart if Ubuntu), and have the fabric task stop the currently running one, copy over whatever new files it needs (or similar process), and then restart it.
Assuming what you have above is a bash script or similar, you may want to also define set -e so that, in case any of the commands returns a non-success code, the script will fail, and in turn, fail the build.

Shell script to update DB via flask

I got started with flask and I tried out the Flaskr example. On the execution of a certain python script, I would like to update one row of my database.
I am a newbie here and would like to understand: am I going to update the DB from inside that python script or I am going to wait for a signal from the flask WSGI script:
I have referred to this thread but am not sure how I am going to interact with the external script. Any help or hints are appreciated.
WSGI handles HTTP requests/responses. A script won't be issuing those. Instead, import your Flask app in the script and make an application context:
from my_project import my_app
ctx = my_app.app_context()
ctx.push()
# ... my code
db.session.commit()
ctx.pop()
Relevant docs: http://flask.pocoo.org/docs/appcontext/, http://flask.pocoo.org/docs/shell/
Or consider using Flask-Script to add command line functions to your application, if the function doesn't need to be a separate script.