Problem:
I want to figure out a way to disable throttling when running my tests with pytest -vv
Details:
I have this default throttling policy in my settings.py file:
'DEFAULT_THROTTLE_RATES': {
'anon': '100/day',
'user': '1000/day'
}
I also have this fixture in my confest.py which returns an error whenever I exceed the limit of requests:
def get_token(user, client):
response = client.post(
"/email-login",
{"email":user.email, "password": "B9vX95phJDi3C4"},
)
return {
"HTTP_AUTHORIZATION": f"Bearer {response.json()['token']['access']}"
}
What I have tried:
I have attempted to use the solution in this GitHub Issue: https://github.com/encode/django-rest-framework/issues/1336, but it doesn't work in my case.
First you need to create a way to differentiate between test env and otherwise. Like we do for PROD and DEV using settings.DEBUG config.
My recommendation is to create an env variable test=Trueand then in your settings.py write -
if os.environ.get("test", False):
REST_FRAMEWORK = {
'DEFAULT_THROTTLE_CLASSES': [
'rest_framework.throttling.AnonRateThrottle',
'rest_framework.throttling.UserRateThrottle'
],
'DEFAULT_THROTTLE_RATES': {
'anon': '100/day',
'user': '1000/day'
}
}
else it does nothing, and drf will not throttle.
I was able to resolve this problem with the following steps:
I created a new settings file which inherited from the base settings file. i.e from settings import *
Then I deleted the DEFAULT_THROTTLE_RATES key i.e del REST_FRAMEWORK["DEFAULT_THROTTLE_RATES"]
Next thing I did was to point to the new settings file in pytest.ini i.e DJANGO_SETTINGS_MODULE="new_settings.py"
Now the tests will use the new settings file
#ra123 has the right idea in general. As another approach, with all Django projects I add something like this to my settings/__init__.py (or just settings.py if you do a one file thing). It looks at argv to see if its in test mode
IS_TESTING = bool(set(sys.argv[:2]) & {"pytest", "test", "jenkins"})
REST_FRAMEWORK = { "YOUR_CONFIG": "..." }
# at the very very end, AFTER your settings are loaded:
if IS_TESTING:
# override your rest framework settings in test mode
REST_FRAMEWORK["DEFAULT_THROTTLE_CLASSES"] = []
# some other handy things, for making tests faster/easier
PASSWORD_HASHERS = ("django.contrib.auth.hashers.MD5PasswordHasher",)
EMAIL_BACKEND = "django.core.mail.backends.locmem.EmailBackend"
DEFAULT_FILE_STORAGE = "inmemorystorage.InMemoryStorage"
I ended up with it this way so we don't have to worry about it ever getting the wrong settings. It also helps keep things centralized, so (for example) you don't call sentry.init in testing mode, even if there is a sentry_url in the environment.
Related
So I've got a django project with several applications.
Each application use its own database, and they share a common database containing django tables (such as auth, sessions).
For this I've got several database routers, so my settings looks like this :
DATABASES = {
'default': {
..
},
'app1_db': {
..
},
'app2_db':{
..
}
}
DATABASE_ROUTERS = ["site.db_router.App1Router", "site.db_router.App2Router"]
# no router for default database
Each application also got its unit tests. To troubleshoot my problem I extracted one of the application. It contains a tests module with four test files. Test file number one looks like below:
class ExcelTestCase(TransactionTestCase):
databases = ["app1_db"]
# some tests
Test case 1
python manage.py test app1.tests.testfile1 raises this error django.core.exceptions.ImproperlyConfigured: Circular dependency in TEST[DEPENDENCIES]
Test case 2
I comment the databases section : only default test database is created (not the application one) and this error is raised:
AssertionError: Database queries to 'app1_db' are not allowed in this test. Add 'app1_db' to app1.tests.testfile1.ExcelTestCase.databases to ensure proper test isolation and silence this fai
lure.
Test case 3
I uncomment databases setting and run python manage.py test app1.tests so every test are runned.
Three test files out of four are TransactionTestCase and therefore have databases settings.
I need to comment at least the first or third (second one is not TransactionTestCase) test file databases or I get Circulary dependency error, and by doing so I got AssertionError because I can't use the app database.
I'm using Django 2.2 and have been struggling on this issue for a while, any help will be appreciated !
Maybe setting your 'DEPENDECIES'.
controlling creation order for test databases
Check that they receive a list of dependencies and the databases without dependencies receive an EMPTY list.
Basically you have to add something like:
'default': {
..
'TEST': {
'DEPENDENCIES': ['app1_db'],
},
},
'app1_db': {
# ... db settings
'TEST': {
'DEPENDENCIES': [],
},
}
We had our Airflow custom UI based on this link and it was working fine with Airflow 1.9.0.
Following this we upgraded to 1.10.1 and also enabled RBAC. Our custom UI stopped coming after this.
We followed this explanation note-on-role-based-views and tried to use our old UI templates with appbuilder_views. On the using the TestAppBuilderBaseView from /tests/plugins/test_plugin.py,
class TestAppBuilderBaseView(AppBuilderBaseView):
#expose("/")
def test(self):
return self.render("test_plugin/test.html", content="Hello galaxy!")
we get the menu and the link, but on clicking we get the error
object has no attribute 'render'
On changing this to
return self.render_template("test_plugin/test.html",content="Hello galaxy!")
we get the error
jinja2.exceptions.TemplateNotFound: test_plugin/test.html
I have tried all possible combination placing the templates folder and the html file, but still its the same error.
I do find some forums telling to enable debug on Blueprint. but I am not aware on how you can do that with Airflow
Any guidance on this please?.
Thanks in Advance
Jeenson
The version 1.10.0 when released had a bug that was not installing the plugins correctly in the new UI. This was fixed in the version 1.10.1, but the code example for plugins in Airflow documentation is broken.
I wrote a sample project to make the integration work, you can check it here: https://github.com/felipegasparini/airflow_plugin_rbac_test
But in short, you need to:
Import the BaseView from appbuilder correctly using:
from flask_appbuilder import BaseView as AppBuilderBaseView
Change the name of the method 'test' to 'list'
Set the template_folder property to point to where your templates are.
Something like this:
from airflow.plugins_manager import AirflowPlugin
from flask_appbuilder import BaseView as AppBuilderBaseView
class TestAppBuilderBaseView(AppBuilderBaseView):
template_folder = '/root/airflow/plugins/test_plugin/templates'
#expose("/")
def list(self):
return self.render_template("test.html", content="Hello galaxy!")
v_appbuilder_view = TestAppBuilderBaseView()
v_appbuilder_package = {"name": "Test View",
"category": "Test Plugin",
"view": v_appbuilder_view}
# Defining the plugin class
class AirflowTestPlugin(AirflowPlugin):
name = "test_plugin"
# operators = [PluginOperator]
# sensors = [PluginSensorOperator]
# hooks = [PluginHook]
# executors = [PluginExecutor]
# macros = [plugin_macro]
# admin_views = [v]
# flask_blueprints = [bp]
# menu_links = [ml]
appbuilder_views = [v_appbuilder_package]
# appbuilder_menu_items = [appbuilder_mitem]
I am also faced the same issue.
After including template folder in blueprint its picking up the correct folder and here is my working example.
Please keep the folder structure like below
Plugin
|_test_plugin
|_templates
|_test.html
test_plugin.py
test_plugin.py
from airflow.plugins_manager import AirflowPlugin
from flask import Blueprint
from flask_admin import BaseView, expose
from flask_admin.base import MenuLink
class TestView(BaseView):
#expose('/')
def test(self):
return self.render("test.html", content="Hello galaxy!")
v = TestView(category="Test Plugin", name="Test View")
blue_print_ = Blueprint("test_plugin",
__name__,
template_folder='templates')
class AirflowTestPlugin(AirflowPlugin):
name = "MenuLinks"
# operators = []
flask_blueprints = [blue_print_]
# hooks = []
# executors = []
admin_views = [v]
#appbuilder_views = [v_appbuilder_package]
fgasparini's answer is correct, but I also need to enable the RBAC setting
rbac = True
in airflow.cfg in order for flask_appbuilder to work with airflow, otherwise the menu won't show up.
I am trying to apply cache in my Django project using memcache.
CACHES = {
'default': {
'BACKEND': 'django.core.cache.backends.memcached.MemcachedCache',
'LOCATION': '127.0.0.1:11211',
}
}
I am applying per view cache as below:
from django.views.decorators.cache import cache_page
url(r'^(?P<slug>[\w-]+)/default/$', cache_page(60 * 15)(default_view), name='default_view')
I am trying to dedug the code in my IDE.
I found the following function at lib/python3.6/site-packages/django/views/decorators/cache.py
def cache_page(timeout, *, cache=None, key_prefix=None):
return decorator_from_middleware_with_args(CacheMiddleware)(
cache_timeout=timeout, cache_alias=cache, key_prefix=key_prefix
)
I tried to create a stop at return line. But I found when i refresh the url the code does not stop at this line.
As already #Alasdair mentioned in the comment if you set a breakpoint in cache_page body your it will be executed only once.
If you want to follow the request you should set your breakpoint in CacheMiddleware methods either in process_response or process_request
Well, I have a django project which works fine now.
I'd like to add a new app to it, in which I need to access multiple databases.
I know Django support multiple databases settings and know how to configure it. This is not the problem.
The issue is that, in the 90% of my project components, I don't need to maintain a multiple databases settings. The only usage for the second databases is in the new added app.
So I tried to alter the settings by calling:
django.conf.settings.configure(DATABASES = {....})
in the new app. And django said:
RuntimeError: Settings already configured.
Which makes sense, since I have the origin settings file and set the DJANGO_SETTINGS_MODULE up.
So I question is that what should be a good approach in this case.
I don't want to discard DJANGO_SETTINGS_MODULE variable.
I try to not include second database in the setting file initially, since the new app is also an independent module which should work independently outside the django project. So I want to have the similar config in new app to setup the database config.
Does anyone has any idea about this?
Thanks in advance!
Actually, I have the same issue in a current project. As you I have a totally independent app which uses an another database, and I could have other apps which could have the same behaviour.
The thing that I have done is to create a dir apps where I store my apps and then I add this at the end of my settings.py file :
DATABASE_ROUTERS = ['myproject.routers.MultiDBRouter']
import os
APPS_DIR = os.path.join(PROJECT_ROOT, 'apps')
for app_name in os.listdir(APPS_DIR):
print '\nLooking for settings in apps/%s :' % app_name
if os.path.exists(os.path.join(APPS_DIR, app_name, 'settings.py')):
print ' Settings file found...'
app = __import__('%s.settings' % app_name)
content = dir(app.settings)
if 'DATABASES' in content:
print ' Adding databases :'
for key, value in app.settings.DATABASES.iteritems():
if DATABASES.has_key(key):
print ' Can not add %s database config, because it already exists' % key
else:
DATABASES[key] = value
DATABASES[key]['APPS'] = [app_name]
print ' Added %s database config' % key
It will automatically look after settings.py file in all the apps/myapp/ directories. If it finds a new DATABASES variable in a app/myapp/settings.py file, it will add the other database configurations to your DATABASES variable (the true one).
I have also created a router to do not have to use the using command (the MultiDBRouter).
And then I add a settings.py file in all the app which requires another database :
DATABASES = {
'db': {
'ENGINE': 'django.db.backends.mysql',
'NAME': 'database',
'USER': 'username',
'PASSWORD': 'mysecretpassword',
}
}
I've been evaluating django and wondered if the following is possible. I've already looked at the regular multiple database docs so please don't point me to that because this use case isn't mentioned as far as i can make out. If i'm wrong i take it back :)
I want one main database in which most of my app's models will reside, however one of the app's will need to dynamically create databases, these will be customer specific databases.
The database path (i plan to use sqlite) will be stored in primary database and so the cursor would need to be changed but the model will remain the same.
I would welcome any thoughts on ways to achieve this?
I will open with "You should not edit settings at runtime".
Having said that, I have exactly this same issue, where I want to create a unique database for each user. The reason for doing this is I am offering the ability for the user to save/access to/from a database not stored on my server, which entails having multiple databases, and thus one for each user.
This answer is NOT the recommended way to achieve the desired goal. I would love to hear from a django-guru how to best approach this problem. However, this is a solution I have been using and it has worked well so far. I am using sqlite however it can be easily modified for any of the databases.
In summary, this is the process:
Add the new database to settings (at runtime)
Create a file to store these settings for reloading when the server is restarted (at runtime)
Run a script which loads the saved settings files (whenever the server is restarted)
Now, how to achieve this:
1) Firstly, when a new user is created, I create a new database in the settings. This code lives in my view where new users are created.
from YOUR_PROJECT_NAME import settings
database_id = user.username #just something unique
newDatabase = {}
newDatabase["id"] = database_id
newDatabase['ENGINE'] = 'django.db.backends.sqlite3'
newDatabase['NAME'] = '/path/to/db_%s.sql' % database_id
newDatabase['USER'] = ''
newDatabase['PASSWORD'] = ''
newDatabase['HOST'] = ''
newDatabase['PORT'] = ''
settings.DATABASES[database_id] = newDatabase
save_db_settings_to_file(newDatabase) #this is for step 2)
This script loads the database settings 'at runtime' into the django project settings. However if the server is restarted, this database will no longer be in settings.
2) To facilitate reloading these settings automatically whenever the server is restarted, I create a file for each database which will be loaded whenever the server is started. Creating this file is performed by the function save_db_settings_to_file:
def save_db_settings_to_file(db_settings):
path_to_store_settings = "/path/to/your/project/YOUR_PROJECT_NAME/database_settings/"
newDbString = """
DATABASES['%(id)s'] = {
'ENGINE': '%(ENGINE)s', # Add 'postgresql_psycopg2', 'mysql', 'sqlite3' or 'oracle'.
'NAME': '%(NAME)s', # Or path to database file if using sqlite3.
'USER': '', # Not used with sqlite3.
'PASSWORD': '', # Not used with sqlite3.
'HOST': '', # Set to empty string for localhost. Not used with sqlite3.
'PORT': '', # Set to empty string for default. Not used with sqlite3.
}
""" % db_settings
file_to_store_settings = os.path.join(path_to_store_settings, db_settings['id'] + ".py")
write_file(file_to_store_settings, newDbString) #psuedocode for compactness
3) To actually load these settings when the server is started, I add a single line to the very bottom of /path/to/your/project/YOUR_PROJECT_NAME/settings.py, which loads each file in the settings folder and runs it, having the effect of loading the database details into the settings.
import settings_manager
Then, import settings_manager will load the file at /path/to/your/project/YOUR_PROJECT_NAME/settings_manager.py, which contains the following code:
from settings import DATABASES
import os
path_to_store_settings = "/path/to/your/project/YOUR_PROJECT_NAME/database_settings/"
for fname in os.listdir(path_to_settings):
full_path = os.path.join(path_to_settings, fname)
f = open(full_path)
content = f.read()
f.close()
exec(content) #you'd better be sure that the file doesn't contain anything malicious
Note that you could put this code directly at the bottom of settings.py instead of the import statement, but using the import statement keeps the abstraction level of settings.py consistent.
This is a convenient way to load each database setting because to remove a database from the settings all you have to do is delete the settings file, and the next time the server restarts it won't load those details into the settings, and the database will not be accessible.
As I said, this works and I have had success using it so far, but this is NOT the ideal solution. I would really appreciate if someone could post a better solution.
What's bad about it:
It explicitly defies advice from django team not to modify settings at runtime. I do not know the reason for why this advice is given.
It uses an exec statement to load the data into settings. This should be OK, but if you get some corrupt or malicious code in one of those files you will be a sad panda.
Note that I still use the default database for auth and sessions data, but all the data from my own apps is stored in the user-specific database.
To augment #thedawnrider's answer, in some cases editing settings.DATABASES may not be enough. It might be more reliable to edit django.db.connections.databases, which serves as a cache and wrapper around settings.DATABASES.
e.g.
from django.db import connections
database_id = user.username #just something unique
newDatabase = {}
newDatabase["id"] = database_id
newDatabase['ENGINE'] = 'django.db.backends.sqlite3'
newDatabase['NAME'] = '/path/to/db_%s.sql' % database_id
newDatabase['USER'] = ''
newDatabase['PASSWORD'] = ''
newDatabase['HOST'] = ''
newDatabase['PORT'] = ''
connections.databases[database_id] = newDatabase
This question is pretty outdated and should be updated.
Django support multiple databases out of the box
Here is many good tutorials [ 1, 2 ] and answers how to manage connections dynamically
Also you may simply use django-dynamic-db-router
Database connection settings may be configured on the fly by using configure method of settings object:
from django.conf import settings
dbs = settings.DATABASES.copy()
dbs['some_new_db'] = {'ENGINE': 'dummy'}
settings.configure(DATABASES=dbs)