Django automatically performing the collectstatic command - django

In my project, I have a main static folder and a sub folder named static. When I make changes in my sub folder named static (which I specified in COLLECTSTATIC_DIRS within the settings file), I save the file and run the collectstatic command.
This successfully saves the changes, however is really inefficient as I am constantly changing css and Javascript files inside my project, which I store as static files.
I browsed the web, and came across a solution named whitenoise, which is a pip package. But this package only works for a short period of time, and after a few times of closing and opening my project folder, it completely stopped working.
Does anybody have another solution to deal with this problem? Thank you.

You can use python-watchdog and write your own Django command:
import time
from django.conf import settings
from django.core.management import call_command
from django.core.management.base import BaseCommand
from watchdog.events import FileSystemEventHandler
from watchdog.observers import Observer
class Command(BaseCommand):
help = "Automatically calls collectstatic when the staticfiles get modified."
def handle(self, *args, **options):
event_handler = CollectstaticEventHandler()
observer = Observer()
for path in settings.STATICFILES_DIRS:
observer.schedule(event_handler, path, recursive=True)
observer.start()
try:
while True:
time.sleep(1)
finally:
observer.stop()
observer.join()
class CollectstaticEventHandler(FileSystemEventHandler):
def on_moved(self, event):
super().on_moved(event)
self._collectstatic()
def on_created(self, event):
super().on_created(event)
self._collectstatic()
def on_deleted(self, event):
super().on_deleted(event)
self._collectstatic()
def on_modified(self, event):
super().on_modified(event)
self._collectstatic()
def _collectstatic(self):
call_command("collectstatic", interactive=False)

You can use 3rd party solution which doesn't belong to Django to monitor your files and run commands on the files changes.
Take a look at fswatch utility Bash Script - fswatch trigger bash function
How to manage static files "Django-way"
Please check details on https://docs.djangoproject.com/en/3.0/howto/static-files/ what is a Django-way to do it correct :)

First of all set DEBUG = True while working on development.
Then add these lines to your project's urls.py:
from django.conf import settings
from django.views.decorators.cache import cache_control
from django.contrib.staticfiles.views import serve
from django.conf.urls.static import static
if settings.DEBUG:
urlpatterns += static(settings.STATIC_URL,
view=cache_control(no_cache=True, must_revalidate=True)(serve))

Here's an example using Python watchfiles and a Django management command.
# Lives in /my_app/manangement/commands/watch_files.py
import time
from django.conf import settings
from django.core.management import call_command
from django.core.management.base import BaseCommand
from watchfiles import watch
class Command(BaseCommand):
help = "Automatically calls collectstatic when the staticfiles get modified."
def handle(self, *args, **options):
print('WATCH_STATIC: Static file watchdog started.')
#for changes in watch([str(x) for x in settings.STATICFILES_DIRS]):
for changes in watch(*settings.STATICFILES_DIRS):
print(f'WATCH_STATIC: {changes}', end='')
call_command("collectstatic", interactive=False)
You can then run the Django management command in the background in whatever script you use to start Django.
python manage.py watch_static &
python manage.py runserver 0.0.0.0:8000

Related

Not able to inspect Django migrations from test but working from Django command

I want to be able to inspect Django migrations within a test, so I can check basic sanity before running more expansive tests.
I was able to come up with the following in a Django command:
from django.core.management.base import BaseCommand
from django.db.migrations.loader import MigrationLoader
class Command(BaseCommand):
help = "Check migrations"
def handle(self, *args, **options):
self.migration_loader = MigrationLoader(None)
self.migration_loader.build_graph()
for root_node in self.migration_loader.graph.root_nodes():
app_name = root_node[0]
print(app_name, root_node[0])
But when I translate that into a test (using pytest):
from django.db.migrations.loader import MigrationLoader
def test_multiple_leaf_nodes_in_migration_graph():
migration_loader = MigrationLoader(None)
migration_loader.build_graph()
for root_node in migration_loader.graph.root_nodes():
app_name = root_node[0]
print(app_name, root_node[0])
Then the graph (root nodes) returns an empty list.
The project is structured as follows:
django_project/
settings.py
... # other Django core files
tests/
test_above.py
django_app/
models.py
... # other files from this app
management/commands/
command_above.py
pytest.ini
pytest.ini:
[pytest]
DJANGO_SETTINGS_MODULE = django_project.settings
python_files = tests.py test_*.py
The command:
pytest --no-migrations
PS: the whole point of this is to be able to detect migration errors without running them.
Is there anything I need to do so I can "see" migrations within the test?
pytest --no-migrations
Since you disabled migrations, the MigrationLoader won't load the migrations.
To load the migrations, you can override settings for MIGRATION_MODULES just for that test:
from django.test import override_settings
#override_settings(MIGRATION_MODULES={}) # Add this
def test_multiple_leaf_nodes_in_migration_graph():
...

Django-Celery: importing from another app into project's celery.py file

I am following the tutorial on here to get periodic tasks defined in my django project working.
The article suggests having a celery.py file of the form:
from celery import Celery
from celery.schedules import crontab
app = Celery()
#app.on_after_configure.connect
def setup_periodic_tasks(sender, **kwargs):
# Calls test('hello') every 10 seconds.
sender.add_periodic_task(10.0, my_task.s('hello'), name='add every 10')
)
#app.task
def my_task(arg):
print(arg)
which works. Now this is good but I don't want to define my tasks locally. my question is, how can I add tasks from other apps?
I have created a blank project called my_proj and it has two apps: my_proj and app_with_tasks. the celery.py file above is at the root level in my_proj app's directory and I want to add periodic tasks from app_with_tasks 's tasks.py file.
I do have app_with_tasks listed in Installed-apps for my_proj settings file but I still can't import anything from an app to anther.
my understanding is that I should use:
from app_with_tasks.tasks import task1
but my_proj will then show as unresolved reference in PyCharm.
I'll tell you what I'm using. Maybe it helps you
my_proj/celery.py
import os
import celery
# set the default Django settings module for the 'celery' program.
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'my_proj.settings')
app = celery.Celery('app_django')
app.config_from_object('django.conf.settings')
app.autodiscover_tasks(lambda: settings.INSTALLED_APPS)
then in app_with_tasks, add file tasks.py
from my_proj.celery import app
from django.apps import apps
#app.task(bind=False)
def your_task(some_arg):
A_Model = apps.get_model('my_proj', 'A_Model')
....
command to start celery server (restart this every time you change a task to reload tasks.py files)
/path/to/virtualenv/bin/celery --app=my_proj.celery:app --loglevel=INFO --concurrency=4 -n default_worker worker
To call the task (here you should use your add_periodic_task code)
from app_with_tasks.tasks import your_task
your_task.apply_async(args=[123], kwargs=None)

Django - Error importing storages.backends

I have created a custom storage backend, the file is called storages.py and is placed in an app called core:
from django.conf import settings
from storages.backends.s3boto import S3BotoStorage
class S3StaticBucket(S3BotoStorage):
def __init__(self, *args, **kwargs):
kwargs['bucket_name'] = getattr(settings, 'static.mysite.com')
super(S3BotoStorage, self).__init__(*args, **kwargs)
In settings.py, I have the follwing:
STATICFILES_STORAGE = 'core.storages.S3StaticBucket'
DEFAULT_FILE_STORAGE = 'storages.backends.s3boto.S3BotoStorage'
When I try to do python manage.py collectstatic it shows the following error:
django.core.exceptions.ImproperlyConfigured: Error importing storage module core.storages: "No module named backends.s3boto"
And when I run python manage.py shell and try to import the same:
>>>
>>> from django.conf import settings
>>> from storages.backends.s3boto import S3BotoStorage
>>>
Any idea what I'm doing wrong?
There is a namespace conflict; the storage absolute name clashes with a storage local name. It may be unintuitive, but you can import from module in itself:
// file my_module/clash.py
import clash
print clash.__file__
Now we run python shell in a dir containing a my_module:
$ python
>>> import my_module.clash
my_module.clash.py
In short, your module tries to import a backend from itself.
You need an absolute import - Trying to import module with the same name as a built-in module causes an import error.
I had this same issue, but for me it turns out that despite django-storages being installed, boto was not. A simple pip install boto fixed the error in my scenario.
I had another type of issue that can help others, I used to have another file named storages.py but I deleted that file days ago, and still getting the Exception... the thing is I didn't had deleted the file storages.pyc !
Typo error.
Change:
DEFAULT_FILE_STORAGE = 'storages.backends.s3boto.S3BotoStorage'
TO:
DEFAULT_FILE_STORAGE = 'storages.backends.s3boto.S3Boto3Storage'

Could not import/No module named Django Error with Apache

I had a small proof of concept set up on a development server on a local machine. I'm now trying to move it over to django on a production server, which I'm using webfaction for. However, now that I'm switched over to apache from the built in django server I get the following:
ViewDoesNotExist: Could not import orgDisplay.views. Error was: No module named orgDisplay.views
But when check my orgDisplay apps folder there is a view.py in it. What am I doing wrong? I've tried adding the following to my settings.py by suggestion of the django IRC room.
import sys
sys.path.append(r"/home/user/webapps/django_project/myproject/orgDisplay")
which is the path to my app.
any ideas on how to even begin to trouble shoot this?
Thanks in advance.
I assume you're using mod_wsgi (which is recommended by Django authors), not mod_python. This is the way you should use your sys.path:
django.wsgi:
import os, sys
sys.path.append(r"/home/user/webapps/django_project/myproject/")
os.environ["DJANGO_SETTINGS_MODULE"] = "settings"
sys.stdout = sys.stderr # Prevent crashes upon print
import django.core.handlers.wsgi
application = django.core.handlers.wsgi.WSGIHandler()
urls.py:
from django.conf.urls.defaults import *
urlpatterns = (
("", include("orgDisplay.urls")),
# ...
)
orgDisplay/urls.py:
import views
urlpatterns = (
(r'^some_view/$', views.some_view), # It is actually orgDisplay.views.some_view
# many more records ...
)
It is a bad idea to add project dir itself to path since you're be getting name conflicts between multiple projects.
I think you're appending the wrong directory to sys.path. I think Python is looking in the .../myproject/orgDisplay folder for the orgDisplay package. Try removing the orgDisplay from your string, like this:
import sys
sys.path.append(r"/home/user/webapps/django_project/myproject")
The other option would be to simply add myproject (or whatever your project is actually called) in the import statement.
# instead of "from orgDisplay import views"
from myproject.orgDisplay import views
Also, make sure to restart Apache after every edit.
looking at manage.py, it does it like so:
import sys
from os.path import abspath, dirname, join
from django.core.management import setup_environ
# setup the environment before we start accessing things in the settings.
setup_environ(settings_mod)
sys.path.insert(0, join(PINAX_ROOT, "apps"))
sys.path.insert(0, join(PROJECT_ROOT, "apps"))
Provided your WSGI file is in your project directory, a slightly more flexible way is this:
import os, sys
sys.path.append(os.path.dirname(__file__))
This will enable you to change your project location later without having to modify your WSGI file.

How to Unit test with different settings in Django?

Is there any simple mechanism for overriding Django settings for a unit test? I have a manager on one of my models that returns a specific number of the latest objects. The number of objects it returns is defined by a NUM_LATEST setting.
This has the potential to make my tests fail if someone were to change the setting. How can I override the settings on setUp() and subsequently restore them on tearDown()? If that isn't possible, is there some way I can monkey patch the method or mock the settings?
EDIT: Here is my manager code:
class LatestManager(models.Manager):
"""
Returns a specific number of the most recent public Articles as defined by
the NEWS_LATEST_MAX setting.
"""
def get_query_set(self):
num_latest = getattr(settings, 'NEWS_NUM_LATEST', 10)
return super(LatestManager, self).get_query_set().filter(is_public=True)[:num_latest]
The manager uses settings.NEWS_LATEST_MAX to slice the queryset. The getattr() is simply used to provide a default should the setting not exist.
EDIT: This answer applies if you want to change settings for a small number of specific tests.
Since Django 1.4, there are ways to override settings during tests:
https://docs.djangoproject.com/en/stable/topics/testing/tools/#overriding-settings
TestCase will have a self.settings context manager, and there will also be an #override_settings decorator that can be applied to either a test method or a whole TestCase subclass.
These features did not exist yet in Django 1.3.
If you want to change settings for all your tests, you'll want to create a separate settings file for test, which can load and override settings from your main settings file. There are several good approaches to this in the other answers; I have seen successful variations on both hspander's and dmitrii's approaches.
You can do anything you like to the UnitTest subclass, including setting and reading instance properties:
from django.conf import settings
class MyTest(unittest.TestCase):
def setUp(self):
self.old_setting = settings.NUM_LATEST
settings.NUM_LATEST = 5 # value tested against in the TestCase
def tearDown(self):
settings.NUM_LATEST = self.old_setting
Since the django test cases run single-threaded, however, I'm curious about what else may be modifying the NUM_LATEST value? If that "something else" is triggered by your test routine, then I'm not sure any amount of monkey patching will save the test without invalidating the veracity of the tests itself.
You can pass --settings option when running tests
python manage.py test --settings=mysite.settings_local
Although overriding settings configuration on runtime might help, in my opinion you should create a separate file for testing. This saves lot of configuration for testing and this would ensure that you never end up doing something irreversible (like cleaning staging database).
Say your testing file exists in 'my_project/test_settings.py', add
settings = 'my_project.test_settings' if 'test' in sys.argv else 'my_project.settings'
in your manage.py. This will ensure that when you run python manage.py test you use test_settings only. If you are using some other testing client like pytest, you could as easily add this to pytest.ini
Update: the solution below is only needed on Django 1.3.x and earlier. For >1.4 see slinkp's answer.
If you change settings frequently in your tests and use Python ≥2.5, this is also handy:
from contextlib import contextmanager
class SettingDoesNotExist:
pass
#contextmanager
def patch_settings(**kwargs):
from django.conf import settings
old_settings = []
for key, new_value in kwargs.items():
old_value = getattr(settings, key, SettingDoesNotExist)
old_settings.append((key, old_value))
setattr(settings, key, new_value)
yield
for key, old_value in old_settings:
if old_value is SettingDoesNotExist:
delattr(settings, key)
else:
setattr(settings, key, old_value)
Then you can do:
with patch_settings(MY_SETTING='my value', OTHER_SETTING='other value'):
do_my_tests()
You can override setting even for a single test function.
from django.test import TestCase, override_settings
class SomeTestCase(TestCase):
#override_settings(SOME_SETTING="some_value")
def test_some_function():
or you can override setting for each function in class.
#override_settings(SOME_SETTING="some_value")
class SomeTestCase(TestCase):
def test_some_function():
#override_settings is great if you don't have many differences between your production and testing environment configurations.
In other case you'd better just have different settings files. In this case your project will look like this:
your_project
your_app
...
settings
__init__.py
base.py
dev.py
test.py
production.py
manage.py
So you need to have your most of your settings in base.py and then in other files you need to import all everything from there, and override some options. Here's what your test.py file will look like:
from .base import *
DEBUG = False
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.sqlite3',
'NAME': 'app_db_test'
}
}
PASSWORD_HASHERS = (
'django.contrib.auth.hashers.MD5PasswordHasher',
)
LOGGING = {}
And then you either need to specify --settings option as in #MicroPyramid answer, or specify DJANGO_SETTINGS_MODULE environment variable and then you can run your tests:
export DJANGO_SETTINGS_MODULE=settings.test
python manage.py test
For pytest users.
The biggest issue is:
override_settings doesn't work with pytest.
Subclassing Django's TestCase will make it work but then you can't use pytest fixtures.
The solution is to use the settings fixture documented here.
Example
def test_with_specific_settings(settings):
settings.DEBUG = False
settings.MIDDLEWARE = []
..
And in case you need to update multiple fields
def override_settings(settings, kwargs):
for k, v in kwargs.items():
setattr(settings, k, v)
new_settings = dict(
DEBUG=True,
INSTALLED_APPS=[],
)
def test_with_specific_settings(settings):
override_settings(settings, new_settings)
I created a new settings_test.py file which would import everything from settings.py file and modify whatever is different for testing purpose.
In my case I wanted to use a different cloud storage bucket when testing.
settings_test.py:
from project1.settings import *
import os
CLOUD_STORAGE_BUCKET = 'bucket_name_for_testing'
manage.py:
def main():
# use seperate settings.py for tests
if 'test' in sys.argv:
print('using settings_test.py')
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'project1.settings_test')
else:
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'project1.settings')
try:
from django.core.management import execute_from_command_line
except ImportError as exc:
raise ImportError(
"Couldn't import Django. Are you sure it's installed and "
"available on your PYTHONPATH environment variable? Did you "
"forget to activate a virtual environment?"
) from exc
execute_from_command_line(sys.argv)
Found this while trying to fix some doctests... For completeness I want to mention that if you're going to modify the settings when using doctests, you should do it before importing anything else...
>>> from django.conf import settings
>>> settings.SOME_SETTING = 20
>>> # Your other imports
>>> from django.core.paginator import Paginator
>>> # etc
I'm using pytest.
I managed to solve this the following way:
import django
import app.setting
import modules.that.use.setting
# do some stuff with default setting
setting.VALUE = "some value"
django.setup()
import importlib
importlib.reload(app.settings)
importlib.reload(modules.that.use.setting)
# do some stuff with settings new value
You can override settings in test in this way:
from django.test import TestCase, override_settings
test_settings = override_settings(
DEFAULT_FILE_STORAGE='django.core.files.storage.FileSystemStorage',
PASSWORD_HASHERS=(
'django.contrib.auth.hashers.UnsaltedMD5PasswordHasher',
)
)
#test_settings
class SomeTestCase(TestCase):
"""Your test cases in this class"""
And if you need these same settings in another file you can just directly import test_settings.
If you have multiple test files placed in a subdirectory (python package), you can override settings for all these files based on condition of presence of 'test' string in sys.argv
app
tests
__init__.py
test_forms.py
test_models.py
__init__.py:
import sys
from project import settings
if 'test' in sys.argv:
NEW_SETTINGS = {
'setting_name': value,
'another_setting_name': another_value
}
settings.__dict__.update(NEW_SETTINGS)
Not the best approach. Used it to change Celery broker from Redis to Memory.
One setting for all tests in a testCase
class TestSomthing(TestCase):
def setUp(self, **kwargs):
with self.settings(SETTING_BAR={ALLOW_FOO=True})
yield
override one setting in the testCase
from django.test import override_settings
#override_settings(SETTING_BAR={ALLOW_FOO=False})
def i_need_other_setting(self):
...
Important
Even though you are overriding these settings this will not apply to settings that your server initialize stuff with because it is already initialized, to do that you will need to start django with another setting module.