I have a pattern that comes up fairly often in a Django app with some task management lib (say celery or dramatiq) that looks as follows:
my_app/
executors/
my_executor.py
tasks/
my_task.py
The circular dependency comes from the fact that the executor may schedule the task, while the task is instantiating an executor.
Executor module:
# my_executor.py
import my_app.tasks.my_task
class MyExecutor():
def some_method(self):
my_app.tasks.my_task.some_actual_task.send()
Task module:
# my_task.py
import my_app.executors.my_executor
#decorating_as_task
def some_actual_task():
executor = my_app.executors.my_executor.MyExecutor()
executor.execute()
There is no issue actually running the app (or tests), but pylint complains about a R0401: Cyclic import between the modules.
But it is not clear to me whether this type of dependency is acceptable or not (Python does not seem to be complaining) and whether there are way of actually improving the code structure (having a higher level module my_app.task_management importing both executors and tasks?) while pleasing pylint on this.
I have seen many similar questions but could not find a satisfactory solution from them.
In the end, through inheritance, I isolated the circular code as follows:
my_app/
abstract_executors/
my_abstract_executor.py
tasks/
my_task.py
# my_abtract_executor.py
class MyAbstractExecutor():
def some_method(self):
self._task_to_schedule.send()
#property
def _task_to_schedule(self):
raise NotImplementedError('_task_to_schedule')
# my_task.py
import my_app.abstract_executors.my_abstract_executor
#decorating_as_task
def some_actual_task():
executor = MyExecutor()
executor.execute()
class MyExecutor(MyAbstractExecutor):
#property
def _task_to_schedule(self):
return some_actual_task
This avoids circular imports while allowing to define most of the execution logic in its own module. Maybe there is something better out there, but this fits the bill (avoid circular imports, please pylint, keep things isolated).
Related
When I run tests in parallel, I get random failures because one test interferes with the cache of another test.
I can work around the problem with
#override_settings(
CACHES={
"default": {
"BACKEND": "django.core.cache.backends.locmem.LocMemCache",
"LOCATION": "[random_string]",
}
},
)
Actually to make that smaller I created a #isolate_cache decorator that is a wrapper around override_settings.
But still I need to go and decorate a large number of test cases. This is error prone, because, as I said, the failures are random. I might run the test suite 100 times without error and think that everything is OK, but I still might have forgotten to decorate a test case, and at some point it will fail randomly.
I've also thought about creating my own TestCase subclass and use only that for all my test cases. This presents a similar problem: at some point someone would inherit from django.test.TestCase out of habit, and it might not fail for a long time. Besides, some of my tests inherit from rest_framework.test.APITestCase (or other classes), so there isn't a single test case subclass.
Is there any way to tell Django to run each test case in an isolated section of the cache once and for all?
You don't need "an isolated section of the cache", just to clear cache between tests.
Here are a few ways.
1. Subclass TestCase
The question mentions this is not desired, but I should still mention this proper way.
from django.core.cache import cache
from django.test import TestCase
class CacheClearTestCase(TestCase):
def tearDown(self):
# super().tearDown()
cache.clear()
2. Patch TestCase.tearDown
Assuming subclasses that override tearDown call super().tearDown(), you could do this.
Add this in manage.py before execute_from_command_line(sys.argv):
if sys.argv[1] == 'test':
from django.test import TestCase
from django.core.cache import cache
TestCase.tearDown = cache.clear
3. Subclass TestSuite
You can clear the cache after each test by subclassing TestSuite to override _removeTestAtIndex and setting DiscoverRunner.test_suite to that subclass.
Add this in manage.py before execute_from_command_line(sys.argv):
if sys.argv[1] == 'test':
from unittest import TestSuite
from django.core.cache import cache
from django.test.runner import DiscoverRunner
class CacheClearTestSuite(TestSuite):
def _removeTestAtIndex(self, index):
super()._removeTestAtIndex(index)
cache.clear()
DiscoverRunner.test_suite = CacheClearTestSuite
Why you don't need an isolated section of the cache
To be clear, this is not a problem caused by running tests in parallel.
From https://docs.djangoproject.com/en/4.0/ref/django-admin/#cmdoption-test-parallel:
--parallel [N]
Runs tests in separate parallel processes.
From https://docs.djangoproject.com/en/4.0/topics/cache/#local-memory-caching-1:
Note that each process will have its own private cache instance, which means no cross-process caching is possible.
The easiest solution would be to have a separate settings file for tests that you can load in manage.py. This can also import all of your default settings.
manage.py
settings = 'my_project.test_settings' if 'test' in sys.argv else 'my_project.settings'
os.environ.setdefault("DJANGO_SETTINGS_MODULE", settings)
test_settings.py
from .settings import * # import default settings
# setting overrides here
CACHES = {
"default": {
"BACKEND": "django.core.cache.backends.locmem.LocMemCache",
"LOCATION": "[random_string]",
}
}
If you need to do more settings overrides, especially for multiple environments, I would recommend using something like django-configurations.
I'm currently developping for python 2, and I'm trying to use abstract base classes to simulate interfaces. I have an interface, a base implementation of that interface and many subclasses that extend the base implementation. It looks like this:
class Interface(object):
__metaclass__ = ABCMeta
class IAudit(Interface):
#abstractproperty
def timestamp(self):
raise NotImplementedError()
#abstractproperty
def audit_type(self):
raise NotImplementedError()
class BaseAudit(IAudit):
def __init__(self, audit_type):
# init logic
pass
#property
def timestamp(self):
return self._timestamp
#property
def audit_type(self):
return self._audit_type
class ConcreteAudit(BaseAudit):
def __init__(self, audit_type):
# init logic
super(ConcreteAudit, self).__init__(audit_type)
pass
However PyCharm notifies me that ConcreteAudit should implement all abstract methods. However, BaseAudit (which is not specified as an abc) already implements those methods, and ConcreteAudit is a subclass of BaseAudit. Why is PyCharm warning me? Shouldn't it detect that IAudit's contract is already implemented through BaseAudit?
Why is PyCharm warning you?
Because all Python IDEs suck, that's why.
Whenever an intern/junior programmer/peer tells me that something that I wrote isn't working for him, I tell him that I'm not discussing it until he tries it by executing a Python script from the command line or from the stock interpreter. 99% of the time, the problem disappears.
Why do they suck? Beats me. But they all sometimes hide exceptions, sometimes make it possible to have stuff imported you didn't know about, and all sometimes decide (as in this case) that something is problematic that a real program running on the stock interpreter simply won't have a problem with.
I tried you code in both Python 2.7 and Python 3.4, and as long as I add from abc import ABCMeta, abstractproperty at the top, it runs peachy-keen fine.
So just ditch PyCharm, or update the tags to show that's where the error is.
I have read Is Rails shared-nothing or can separate requests access the same runtime variables? and they explain my problem:
class variable are maybe share between two request to my rails srver, but where is the solution!?
How can I implement a safe singleton between request?
class Foo
##instances = []
end
How can I be sure instances will be reset for each request HTTP?!
EDIT:
I find "config.reload_classes_only_on_change = false" solution but i'm not sure it's the best for performance.
What are the consequences to this option?
I have an exemple to test safe classes variables :
class Test
def self.log
#test ||= false
puts #test
#test = true
end
end
class ApplicationController < ActionController::Base
def index
Test.log
Test.log
end
end
if I start this code with reloading action (F5), I want to read "false" each time in log of rails server. But, by default it's "false" only the first time.
EDIT 2:
In fact this option reload class, but not resolve the concurency problem in thread.
Classes variables are reset but they can be modify by other thread.
How threadsafe classes variables?
I use the request_store gem, it works great.
My use case is for adding methods to the user model class, like the current_user, their locale, location etc as often my other models need this information.
I just setup the current user from my application controller:
User.current = the_authenticated_user
User.request = request
And in my user model class:
class User
def self.current
RequestStore.store[:current_user]
end
def self.current=(user)
RequestStore.store[:current_user] = user
end
def self.request
RequestStore.store[:current_request]
end
def self.request=(request)
# stash the request so things like IP address and GEO-IP based location is available to other models
RequestStore.store[:current_request] = request
end
def self.location
# resolve the location just once per request
RequestStore.store[:current_location] ||= self.request.try(:location)
end
end
I don't enable the reload classes option as it causes too many problems, I've witnessed multiple versions of classes hanging around. If you use model inheritance (ie STI) lazy loading and/or dynamic class loading will often break how model classes are resolved. You need to use require_dependency in your base and intermediate model classes to ensure downstream classes also get loaded.
My development settings mirror my production settings wrt class handling which is not convenient (requires server restart after a change) but more convenient than chasing non-existent bugs. The rerun gem can monitor file-system changes and restart your server for you so that you get reliable change handling in development, albeit slower than rails broken class reloading.
config/environment/development.rb:
# Rails class reloading is broken, anytime a class references another you get multiple
# class instances for the same named class and that breaks everything. This is especially
# important in Sequel as models resolve classes once.
# So always cache classes (true)
config.cache_classes = true
# Always eager load so that all model classes are known and STI works
config.eager_load = true
Q: How threadsafe classes variables?
A: No variables are thread safe unless protected by synchronize.
From an architectural perspective threading in Rails is a waste of time. The only way I have been able to get true parallel performance/concurrency is multiple processes. It also avoids locking and threading related overheads that just don't exist with long running processes. I tested parallel CPU intensive code using threads with Ruby 2.x and got no parallelism at all. With 1 ruby process per core I got real parallelism.
I would seriously consider Thin with multiple processes and then decide if you want to use Thin+EventMachine to increase overall throughput per process.
I have problems unit testing views of an application that uses django-pipeline? Whenever I perform a client.get() on any URL, it produces the following exception:
ValueError: The file 'css/bootstrap.css' could not be found with <pipeline.storage.PipelineCachedStorage object at 0x10d544950>.
The fact that it is bootstrap.css is of course not important, but that I'm unable to execute view rendering due to this exception.
Any guide / tips are welcome!
I ran into a similar problem. However, setting
STATICFILES_STORAGE='pipeline.storage.NonPackagingPipelineStorage'
when running the tests only partly solved my issue. I also had to disable the pipeline completely if you want to run LiverServerTestCase tests without having to calling 'collecstatic' before running the tests:
PIPELINE_ENABLED=False
Since django 1.4 it's fairly easy to modify settings for tests - there is a handy decorator that works for methods or TestCase classes:
https://docs.djangoproject.com/en/1.6/topics/testing/tools/#overriding-settings
e.g.
from django.test.utils import override_settings
#override_settings(STATICFILES_STORAGE='pipeline.storage.NonPackagingPipelineStorage', PIPELINE_ENABLED=False)
class BaseTestCase(LiveServerTestCase):
"""
A base test case for Selenium
"""
def setUp(self):
...
However this produced inconsistent results as #jrothenbuhler describes in his answer. Regardless, this is less than ideal if you are running integration tests - you should mimic production as much as possible to catch any potential issues. It appears django 1.7 has a solution for this in the form of a new test case "StaticLiveServerTestCase". From the docs:
https://docs.djangoproject.com/en/dev/ref/contrib/staticfiles/#django.contrib.staticfiles.testing.StaticLiveServerCase
This unittest TestCase subclass extends
django.test.LiveServerTestCase.
Just like its parent, you can use it to write tests that involve
running the code under test and consuming it with testing tools
through HTTP (e.g. Selenium, PhantomJS, etc.), because of which it’s
needed that the static assets are also published.
I haven't tested this, but sounds promising. For now I'm doing what #jrothenbuhler in his solution using a custom test runner, which doesn't require you to run collectstatic. If you really, really wanted it to run collectstatic you could do something like this:
from django.conf import settings
from django.test.simple import DjangoTestSuiteRunner
from django.core.management import call_command
class CustomTestRunner(DjangoTestSuiteRunner):
"""
Custom test runner to get around pipeline and static file issues
"""
def setup_test_environment(self):
super(CustomTestRunner, self).setup_test_environment()
settings.STATICFILES_STORAGE = 'pipeline.storage.NonPackagingPipelineStorage'
call_command('collectstatic', interactive=False)
In settings.py
TEST_RUNNER = 'path.to.CustomTestRunner'
I've been running into the same problem. I tackled it using a custom test runner:
from django.conf import settings
from django.test.simple import DjangoTestSuiteRunner
from pipeline.conf import settings as pipeline_settings
class PipelineOverrideRunner(DjangoTestSuiteRunner):
def setup_test_environment(self):
'''Override STATICFILES_STORAGE and pipeline DEBUG.'''
super(PipelineOverrideRunner, self).setup_test_environment()
settings.STATICFILES_STORAGE = 'pipeline.storage.PipelineFinderStorage'
pipeline_settings.DEBUG = True
Then in your settings.py:
TEST_RUNNER = 'path.to.PipelineOverrideRunner'
Setting pipeline's DEBUG setting to True ensures that the static files are not packaged. This prevents the need to run collectstatic before running the tests. Note that it's pipeline's DEBUG setting, not Django's, which is overridden here. The reason for this is that you want Django's DEBUG to be False when testing to best simulate the production environment.
Setting STATICFILES_STORAGE to PipelineFinderStorage makes it so that the static files are found when Django's DEBUG setting is set to False, as it is when running tests.
The reason I decided to override these settings in a custom test runner instead of in a custom TestCase is because certain things, such as the django.contrib.staticfiles.storage.staticfiles_storage object, get set up once based on these and other settings. When using a custom TestCase, I was running into problems where tests would pass and fail inconsistently depending on whether the override happened to be in effect when modules such as django.contrib.staticfiles.storage were loaded.
I ran into the same problem. I managed to get around it by using a different STATIC_FILES_STORAGE when I'm testing:
STATICFILES_STORAGE = 'pipeline.storage.NonPackagingPipelineStorage'
I have separate settings files for production and testing, so I just put it in my test version, but if you don't, you could probably wrap it in if DEBUG.
--EDIT
It took a little more effort, because this can only present during the unittesting. To address that, I used the snippet at http://djangosnippets.org/snippets/1011/ and created a UITestCase class:
class UITestCase(SettingsTestCase):
'''
UITestCase handles setting the Pipeline settings correctly.
'''
def __init__(self, *args, **kwargs):
super(UITestCase, self).__init__(*args, **kwargs)
def setUp(self):
self.settings_manager.set(
STATICFILES_STORAGE='pipeline.storage.NonPackagingPipelineStorage')
Now all of my tests that need to render UI that incude compressed_css tags use UITestCase instead of django.test.TestCase.
I ran into the same problem, and it turned out to be because I had
TEST_RUNNER = 'djcelery.contrib.test_runner.CeleryTestSuiteRunner'
I don't understand how, but it must somehow have interacted with Pipeline. Once I removed that setting, the problem went away.
I still needed to force Celery to be eager during testing, so I used override_settings for the tests that needed it:
from django.test.utils import override_settings
…
class RegisterTestCase(TestCase):
#override_settings(CELERY_EAGER_PROPAGATES_EXCEPTIONS=True,
CELERY_ALWAYS_EAGER=True,
BROKER_BACKEND='memory')
def test_new(self):
…
Same here. Refers to this issues: https://github.com/cyberdelia/django-pipeline/issues/277
As I use py.test, I put this in conftest.py as a workaround:
import pytest
from django.conf import settings
def pytest_configure():
# workaround to avoid django pipeline issue
# refers to
settings.STATICFILES_STORAGE = 'pipeline.storage.PipelineStorage'
i've tried #jrothenbuhler workaround and it helps at first..
but then, for some reason it starts fail again with same error
after hours of debugging i've figured out that the only things that helps is to set
STATICFILES_STORAGE = 'pipeline.storage.NonPackagingPipelineStorage'
directly in settings...
dunno why, but it works.
I want to perform some exhaustive testing against one of my test-cases (say, create a document, to debug some weird things I am encountering..)
My brutal force was to fire python manage.py test myapp in a loop either using Popen or os.system, but now I am back to pure way ?.....
def SimpleTest(unittest.TestCase):
def setUp(self):
def test_01(self):
def tearDown(self):
def suite():
suite = unittest.TestCase()
suite.add(SimpleTest("setUp"))
suite.add(SimpleTest("test_01"))
suite.add(SimpleTest("tearDown"))
return suite
def main():
for i in range(n):
suite().run("runTest")
I ran python manage.py test myapp and I got
File "/var/lib/system-webclient/webclient/apps/myapps/tests.py", line 46, in suite
suite = unittest.TestCase()
File "/usr/lib/python2.6/unittest.py", line 216, in __init__
(self.__class__, methodName)
ValueError: no such test method in <class 'unittest.TestCase'>: runTest
I've googled the error, but I still clueless (I was told to add an empty runTest method, but that doesn't sound right at all...)
Well, according to python's unittest.TestCase:
The simplest TestCase subclass will simply override the runTest()
method in order to perform specific testing code
As you can see, my whole goal is to run my SimpleTest N times. I need to keep track of pass, failure against N.
What option do I have?
Thanks.
Tracking race conditions via unit tests is tricky. Sometimes you're better off hitting your frontend with automated testing tool like Selenium -- unlike unit test, environment is the same and there's no need for extra work to ensure concurrency. Here's one way to run concurrent code in tests when there're no better option: http://www.caktusgroup.com/blog/2009/05/26/testing-django-views-for-concurrency-issues/
Just keep in mind that concurrent test is no definite proof you're free from race conditions -- there's no guarantee it'll recreate all possible combinations of execution order among processes.