With DJANGO app and postgresql db, I use panda read_sql quite a bit for rendering complex queries into dataframe for manipulation and ultimate rendering to JSON. Historically have used django.db dbconnections to map from multi db environment and pass that connection directly into read_sql function. As panda evolves, and becomes less tolerant of non sqlalchemy connections as an argument, I am looking for a simple method to take my existing connection and use it to create_engine.
I've seen some related past comments that suggest
engine = create_engine('postgresql+psycopg2://', creator=con)
but for me that generates an error: TypeError: 'ConnectionProxy' object is not callable
I've played with various attributes of the ConnectionProxy...no success. Would like to avoid writing a separate connection manager or learning too much about sqlalchemy. Django is using psycopg2 to create its connections and I am long time user of same. Some of the add-ins like aldjemy are too intrusive since I neither need or want models mapped. Some have suggested ignoring warning message since dbconnection still works...seems like risk longterm...
Related
Please help, i'm fairly new to Django and not sure what's the best way to proceed with my unit-tests.
So, i have a large django app, and it has dozens of views-methods, and the postgresql schemas get pretty complex. I've read that if I use "from django.test import TestCase" then the test database is flushed after running each unit-test. I wanted to prevent from flushing the db in between unit-tests within the same class, so i started using "from unittest import TestCase". That did the trick and the db is preserved in between unit-tests, but now the statement
self.assertTemplateUsed(response, 'samplepage.html') gives me errors AttributeError: 'TestViews' object has no attribute 'assertTemplateUsed'.
What can I do? Is there an alternative to 'assertTemplateUsed' that can be used with unittest.TestCase? Many thanks in advance!
We need to use real data for one of our many external data sources during Django tests:
data is externally managed and read-only
data is accessed through manage.py inspectdb generated ORM classes
data is highly sensitive and we are not allowed to store fixtures of the actual data
tables are legacy design and will be phased out, hundreds of tables with complex relations, even getting a single record is complex
there is too much to do and I am unwilling to spend the time it would take to generate the fixtures, guarantee they're obfuscated, get approval of the obfuscation and justify keeping it around just to bridge us for a few months
We understand the downsides: This violates test purity and introduces a potential safety risk. We are willing to compromise on both to get us past the next few months when we will phase out this problem data source.
In this case, I need Django to understand that I don't want it to stand up a test database for this source, and just use the actual source so we can run some quick checks and walk away.
What is the simplest way to achieve this, with full understanding and acceptance of the risks and recommendations against?
For us, the solution was a custom test runner.
With help from Django's Advanced Testing Topics documentation, we overrode the default DiscoverRunner like this:
from django.test.runner import DiscoverRunner
def should_create_db(db_name):
# analyse db_name, a key from DATABASES, to determine whether a test
# database should be created
return db_name != 'messy_legacy_database'
class CustomTestRunner(DiscoverRunner):
# override method from superclass to selectively skip database setup
def setup_databases(self, **kwargs):
# 'aliases' is a set of unique keys from settings DATABASES dictionary
aliases = kwargs.get('aliases')
filtered = set([i for i in aliases if should_create_db(i)])
kwargs['aliases'] = filtered
# 'aliases' now contains only keys which trigger test database creation
return super().setup_databases(**kwargs)
# there was no need to override teardown_databases()
Next we update settings.py to use our override instead of the default runner:
TEST_RUNNER = 'path.to.CustomTestRunner'
Finally we tell our test class which databases it can use:
from django.test import TestCase
class OurTest(TestCase):
databases = [
'default',
'messy_legacy_database',
]
def test_messy_legacy_database(self):
# go nuts on your messy legacy database testing calls
pass
In this way our tests now skip test database creation for our messy legacy databases, and the logic we test pulls data from the actual data sources, allowing us to implement quick checks to ensure these code paths work.
There are many times when I would like to apply some custom schema definitions (functions, views, types, triggers, ...)
when a new database is initialized or before tests run. I conjured up the following addition to settings.py, which does work
from django.db.models.signals import post_syncdb
from django.db import connection
# Apply schema changes not defined by Django. Set a module-level
# flag to guard against multiple invocations of syncdb_handler()
def syncdb_handler(**kwargs):
if getattr(connection, 'syncdb_called', None):
return
connection.syncdb_called = True
cursor = connection.cursor()
cursor.execute(open('/path/to/schema.sql')).read())
post_syncdb.connect(syncdb_handler)
The documentation does not seem to suggest a solution like this. Is there a convention for applying custom SQL as part of database creation?
In this example the connect handler is called multiple times and keeping a global variable doesn't seem like a good design pattern.
Using the post_syncdb signal was he preferred way of doing things in the past (though it was most unusual to place this code in settings.py).
But syncdb is no longer being used. Now databases are created and modified with migrations. With that the practice is to use a migrations.RunSQL or migrations.RunPython make customized changes to the database if needed.
I've used Django(python) previously and now I'm learning Kohana (PHP). It seems to be good, but I wonder if there is something like "django-admin.py syncdb" in Kohana, or at least, a tool that makes the same job.
Django Model knows all information about their field before start working with database. Kohana Models, in contrast, knows only table name and on first using ORM authomaticaly calls ORM::list_columns() which executing sql query
SHOW FULL COLUMNS FROM `tablename`
and fill protected $_table_columns variable for this Model
I'm creating a system in django and it'd be really helpful to have a signal that is called every time a SQL "select" query is done on the database. In other words, does anyone know if there is something like a "pre_select" or "post_select" signal method?
I found the signal "connection_created" in the django docs, but couldn't find any clues of how to use it and less about accessing the model that called it. The official documentation just say that it exists but don't give a simple using example... =/
EDIT:
The connection_created just works when the connection is created (how its name says), so, I still without a solution =/.
An example of what I want would be the execution of this queries on distinct objects:
ExampleObject1.objects.filter(attribute=somevalue)
ExampleObject2.objects.filter(attribute=somevalue)
ExampleObject3.objects.filter(attribute=somevalue)
So a function is called receiving the data from each them just before each query being sent to the database in order to threat data, log, etc.
I imagine that exists some functionality like that in django because django log system appears to use something alike.
Any help is welcome. Thanks in advance!
From http://dabapps.com/blog/logging-sql-queries-django-13/
It's not in the form of signal, but it allows you to track all queries. Tracking specific selects should be doable by providing customized log handlers.
import logging
l = logging.getLogger('django.db.backends')
l.setLevel(logging.DEBUG)
l.addHandler(logging.StreamHandler())
#make your queries now...