I have a Flask web app where I need two use two existing databases. The first database (DB-A) is used only for reading data (not writing permissions). The second database (DB-B) I am allowed read/write.
I can read DB-A using automap extension
from sqlalchemy.ext.automap import automap_base
Base = automap_base()
Base.prepare(db.engine, reflect=True)
DB-A = Base.classes.visitors
Reading and writing in DB-B I do ti with sqlalchemy common ORM
SQLALCHEMY_DATABASE_URI = ""
db = SQLAlchemy(app)
According to the documentation SQLALCHEMY_BINDS should do the work, but I cannt find anything about, how can I tell my automap engine to read a specific database.
my question is, how can I use automap for reading DB-A along with NOT-automap for writing in DB-B?
Thanks
Have you seen this part of the documentation in Flask explaining how to bind multiple databases?
Quote from documentation:
SQLALCHEMY_DATABASE_URI = 'postgres://localhost/main'
SQLALCHEMY_BINDS = {
'users': 'mysqldb://localhost/users',
'appmeta': 'sqlite:////path/to/appmeta.db'
}
And when this is setup, you can use the following (also quote from documentation):
>>> db.create_all()
>>> db.create_all(bind=['users'])
>>> db.create_all(bind='appmeta')
>>> db.drop_all(bind=None)
Related
I'm using flask_sqlalchemy in my flask application with a local MySQL (8.0.19) database. I've never got this issue before (started to develop this app months ago). Not sure what've changed, what component of the app got updated but I'm getting this error out of nowhere at the moment. I've searched and found that it might be some character encoding issue, but following the instructions I still get the warning when I open my app:
C:\Users\MyUserName\AppData\Local\Programs\Python\Python37\lib\site packages\pymysql\cursors.py:170:Warning:
(1366, "Incorrect string value: '\\xF6z\\xE9p-e...' for column 'VARIABLE_VALUE' at row 1")
result = self._query(query)
This is my url env variable:
MYSQL_URL = mysql+pymysql://user:passoword#localhost:3306/testdb?charset=utf8mb4
And this is how I create my db session:
db_url = os.getenv('MYSQL_URL')
engine = create_engine(db_url, echo=True)
Session = sessionmaker()
Session.configure(bind=engine)
session = Session()
This is the most simple usage of the session:
def row_count():
return (
session.query(Value.ValueID).count()
)
When I inspect this local database with HeidiSQL it says its collation is utf8mb4_0900_ai_ci. I don't know what those suffix specifics mean and there's a ton of utf8mb4 variant available. This is the default value.
Anyone has any idea how to resolve this warning? What does it mean exactly? As I'm using an ORM I'm not creating any database or running any query by hand, so how should I handle this?
ai : accent insensitive
ci : case insensitive
Did your try the following URL:
MYSQL_URL = mysql+pymysql://user:passoword#localhost:3306/testdb?charset=utf8mb4_ai_ci
I am using:
A Python application in Bluemix
Bluemix cloudant v2.0.0b2 database linked to the Python app
According to https://pypi.python.org/pypi/cloudant/2.0.0b2, everything broke from 0.5 to 2.0, and they are still working on the documentation as everything is Beta. Next to this, I am also new to Python and databases. Documentation can be found here:
http://python-cloudant.readthedocs.io/en/latest/getting_started.html
What I am trying to do is check if a document already exists.
Things that I have tried:
from cloudant.account import Cloudant
import time
import json
# Connect to the database
client = Cloudant(*hidden*)
client.connect()
# The database we work in
db = client['myDatabase']
# The document we work on
doc = db['myDocument']
print doc.exists()
But the code fails before retrieving the document. I checked the source code, and it looks like it is supposed to:
def __getitem__(self, key):
if key in list(self.keys()):
return super(CouchDatabase, self).__getitem__(key)
if key.startswith('_design/'):
doc = DesignDocument(self, key)
else:
doc = Document(self, key)
if doc.exists():
doc.fetch()
super(CouchDatabase, self).__setitem__(key, doc)
return doc
else:
raise KeyError(key)
Source: https://pypi.python.org/pypi/cloudant/2.0.0b2
Is there a way I can check if the document exists before I retrieve it? Or should I retrieve it and catch the error? Or is there a different approach?
The behavior you are describing is the desired behavior for the python-cloudant library database object, so if you intend to use the database object to retrieve your documents and populate your local database cache you should look to except a KeyError in the event of a non-existent document and handle accordingly. However, if are interested in capturing whether a document exists before bringing it into your local database cache then changing your code to something like:
from cloudant.account import Cloudant
from cloudant.document import Document
# Connect to the database
client = Cloudant(*hidden*)
client.connect()
# The database we work in
db = client['myDatabase']
# The document we work on
if Document(db, 'myDocument').exists():
doc = db['myDocument']
would do the trick.
Similarly you could just do:
from cloudant.account import Cloudant
from cloudant.document import Document
# Connect to the database
client = Cloudant(*hidden*)
client.connect()
# The database we work in
db = client['myDatabase']
# The document we work on
doc = Document(db, 'myDocument')
if doc.exists():
doc.fetch()
But this would not populate your local database cache, the db dictionary.
I have some code in a Django app which does the following, to get a Pubmed article by DOI:
def getPubmedByDOI(request,doi):
Entrez.email = 'me#mydomain.com'
handle = Entrez.esearch(db="pubmed", term=doi)
record = Entrez.read(handle)
return getPubmedArticle(request,record["IdList"][0]) // renders the article
This works nicely but for one thing - the Entrez.esearch call insists upon access to /root/.config on the server, specifically to write to the following empty directory:
/root/.config/biopython/Bio/Entrez/DTDs/
It's Apache on Gentoo, running as follows:
User django
Group apache
All the code for the application is in ~django/, so I'd expect any writing to be in ~django/.config rather than /root/.config. I can work around this by changing permissions on /root but a better solution would be to configure Biopython or Apache so as not to write to /root. Does anyone have any suggestions as to how this might be done?
Logged upstream as https://github.com/biopython/biopython/issues/918 which suggests setting:
>>> from Bio.Entrez.Parser import DataHandler
>>> DataHandler.global_dtd_dir
'/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/Bio/Entrez/DTDs'
>>> DataHandler.local_dtd_dir = '...'
Can't find a direct, head on answer to this. Is there a way to access a tempfile in Django across 2 distinct views? Say I have the following code:
view#1(request):
temp = tempfile.NamedTemporaryFile()
write_book.save(temp_file)
temp_file_name = temp_file.name
print temp_file_name
request.session['output_file_name'] = temp_file_name
request.session.modified = True
return #something or other
view#2(request):
temp_file_name = request.session['output_file_name']
temp_file = open(str(temp_file_name))
#do something with 'temp_file' here
My problem comes in specifically on view#2, the 2nd line "open(temp_file_name)". Django complains this file/pathway doesn't exist, which is consistent of my understanding of the tempfile module (that the file is 'hidden' and only available to Django).
Is there a way for me to access this file? In case it matters, I ONLY need to read from it (technically serve it for download).
I'd think of this as how to access a NamedTemporaryFile across different requests, rather than different views. Looking at this documentation on NamedTemporaryFile, it says that the file can be opened across the same process, but not necessarily across multiple processes. Perhaps your other view is being called in a different Django process.
My suggestion would be to abandon the use of NamedTemporaryFile and instead just write it as a permanent file, then delete the file in the other view.
Thanks seddonym for attempting to answer. My partner clarified this for me...seddonym is correct for the Django version of NamedTemporaryFile. By calling the python version (sorry, don't have enough cred to post hyperlinks. Stupid rule) you CAN access across requests.
The trick is setting the delete=False parameter, and closing the file before 'returning' at the end of the request. Then, in the subsequent request, just open(file_name). Psuedo code below:
>>> import tempfile
>>> file = tempfile.NamedTemporaryFile(delete=False)
>>> file.name
'c:\\users\\(blah)\(blah)\(blah)\\temp\\tmp9drcz9'
>>> file.close()
>>> file
<closed file '<fdopen>', mode 'w+b' at 0x00EF5390>
>>> f = open(file.name)
>>> f
<open file 'c:\users\ymalik\appdata\local\temp\tmp9drcz9', mode 'r' at 0x0278C128>
This is, of course, done in the console, but it works in django as well.
I've been evaluating django and wondered if the following is possible. I've already looked at the regular multiple database docs so please don't point me to that because this use case isn't mentioned as far as i can make out. If i'm wrong i take it back :)
I want one main database in which most of my app's models will reside, however one of the app's will need to dynamically create databases, these will be customer specific databases.
The database path (i plan to use sqlite) will be stored in primary database and so the cursor would need to be changed but the model will remain the same.
I would welcome any thoughts on ways to achieve this?
I will open with "You should not edit settings at runtime".
Having said that, I have exactly this same issue, where I want to create a unique database for each user. The reason for doing this is I am offering the ability for the user to save/access to/from a database not stored on my server, which entails having multiple databases, and thus one for each user.
This answer is NOT the recommended way to achieve the desired goal. I would love to hear from a django-guru how to best approach this problem. However, this is a solution I have been using and it has worked well so far. I am using sqlite however it can be easily modified for any of the databases.
In summary, this is the process:
Add the new database to settings (at runtime)
Create a file to store these settings for reloading when the server is restarted (at runtime)
Run a script which loads the saved settings files (whenever the server is restarted)
Now, how to achieve this:
1) Firstly, when a new user is created, I create a new database in the settings. This code lives in my view where new users are created.
from YOUR_PROJECT_NAME import settings
database_id = user.username #just something unique
newDatabase = {}
newDatabase["id"] = database_id
newDatabase['ENGINE'] = 'django.db.backends.sqlite3'
newDatabase['NAME'] = '/path/to/db_%s.sql' % database_id
newDatabase['USER'] = ''
newDatabase['PASSWORD'] = ''
newDatabase['HOST'] = ''
newDatabase['PORT'] = ''
settings.DATABASES[database_id] = newDatabase
save_db_settings_to_file(newDatabase) #this is for step 2)
This script loads the database settings 'at runtime' into the django project settings. However if the server is restarted, this database will no longer be in settings.
2) To facilitate reloading these settings automatically whenever the server is restarted, I create a file for each database which will be loaded whenever the server is started. Creating this file is performed by the function save_db_settings_to_file:
def save_db_settings_to_file(db_settings):
path_to_store_settings = "/path/to/your/project/YOUR_PROJECT_NAME/database_settings/"
newDbString = """
DATABASES['%(id)s'] = {
'ENGINE': '%(ENGINE)s', # Add 'postgresql_psycopg2', 'mysql', 'sqlite3' or 'oracle'.
'NAME': '%(NAME)s', # Or path to database file if using sqlite3.
'USER': '', # Not used with sqlite3.
'PASSWORD': '', # Not used with sqlite3.
'HOST': '', # Set to empty string for localhost. Not used with sqlite3.
'PORT': '', # Set to empty string for default. Not used with sqlite3.
}
""" % db_settings
file_to_store_settings = os.path.join(path_to_store_settings, db_settings['id'] + ".py")
write_file(file_to_store_settings, newDbString) #psuedocode for compactness
3) To actually load these settings when the server is started, I add a single line to the very bottom of /path/to/your/project/YOUR_PROJECT_NAME/settings.py, which loads each file in the settings folder and runs it, having the effect of loading the database details into the settings.
import settings_manager
Then, import settings_manager will load the file at /path/to/your/project/YOUR_PROJECT_NAME/settings_manager.py, which contains the following code:
from settings import DATABASES
import os
path_to_store_settings = "/path/to/your/project/YOUR_PROJECT_NAME/database_settings/"
for fname in os.listdir(path_to_settings):
full_path = os.path.join(path_to_settings, fname)
f = open(full_path)
content = f.read()
f.close()
exec(content) #you'd better be sure that the file doesn't contain anything malicious
Note that you could put this code directly at the bottom of settings.py instead of the import statement, but using the import statement keeps the abstraction level of settings.py consistent.
This is a convenient way to load each database setting because to remove a database from the settings all you have to do is delete the settings file, and the next time the server restarts it won't load those details into the settings, and the database will not be accessible.
As I said, this works and I have had success using it so far, but this is NOT the ideal solution. I would really appreciate if someone could post a better solution.
What's bad about it:
It explicitly defies advice from django team not to modify settings at runtime. I do not know the reason for why this advice is given.
It uses an exec statement to load the data into settings. This should be OK, but if you get some corrupt or malicious code in one of those files you will be a sad panda.
Note that I still use the default database for auth and sessions data, but all the data from my own apps is stored in the user-specific database.
To augment #thedawnrider's answer, in some cases editing settings.DATABASES may not be enough. It might be more reliable to edit django.db.connections.databases, which serves as a cache and wrapper around settings.DATABASES.
e.g.
from django.db import connections
database_id = user.username #just something unique
newDatabase = {}
newDatabase["id"] = database_id
newDatabase['ENGINE'] = 'django.db.backends.sqlite3'
newDatabase['NAME'] = '/path/to/db_%s.sql' % database_id
newDatabase['USER'] = ''
newDatabase['PASSWORD'] = ''
newDatabase['HOST'] = ''
newDatabase['PORT'] = ''
connections.databases[database_id] = newDatabase
This question is pretty outdated and should be updated.
Django support multiple databases out of the box
Here is many good tutorials [ 1, 2 ] and answers how to manage connections dynamically
Also you may simply use django-dynamic-db-router
Database connection settings may be configured on the fly by using configure method of settings object:
from django.conf import settings
dbs = settings.DATABASES.copy()
dbs['some_new_db'] = {'ENGINE': 'dummy'}
settings.configure(DATABASES=dbs)