Flask, managing DB connections(without ORM) - flask

I'm newbie in Flask and web programming on the whole, so I'm trying to understand the principles of contexts and db operations of the framework.
Now I want to make out how to manage DB connections without Sql-Alchemy and other ORMs, just with plain sql.
So the simple example with sqlite3 db.
The structure of my test project:
.
├── application
│   ├── __init__.py
│   ├── routes.py
│   └── templates
│   ├── base.html
│   ├── index.html
├── config.py
├── requirements.txt
├── run.py
└── test.db
init.py
from flask import Flask,g
from config import Config
import sqlite3
app = Flask(__name__)
app.config.from_object(Config)
def get_db():
db = getattr(g, '_database', None)
if db is None:
db = g._database = sqlite3.connect('test.db')
return db
#app.teardown_appcontext
def close_connection(exception):
db = getattr(g, '_database', None)
if db is not None:
db.close()
from application import routes
routes.py
# -*- coding: utf-8 -*-
from flask import render_template,redirect
from application import app, get_db
# import sqlite3
#app.route('/')
def index():
cur = get_db().cursor()
cur.execute('select*from users')
select=cur.fetchall()
return render_template('index.html',title='DBSelectTest', posts=select)
I've copied "get_db" and "close_connection" functions from the official site of Flask framework, but I'm not clearly understand how it works and what will be the difference if I use just simple code for "def get_db" without "g" like this:
con = sqlite3.connect("catalog.db")
cur = con.cursor()
And obviously close connections after query is done like this:
cur.close()
con.close()
I'm just trying to catch how the application and the request contexts work in that case with using g.
The main problem that I don't understand what is a "request" in that case.
My wrong understanding of this case:
So in init.py we create "global variable" g which is in application context and should be accessible for each tread in request context.
In routes.py we get query from client with root route("/") so in that moment request context is created and after that application context as well.
So as I understand "get_db" is accessible from our request context and we're able to connect to the db and use that g value in all functions inside the current tread? What about "close_connection"? Because of decorator "teardown_appcontext" the connections have to been closed after app context is failed or ended? Will it work in my code?
So, could anyone provide a correct case of using plain sql with separate connection function?

Flask application and request contexts behave like global variables for the purposes of your code, but are in fact proxies that will contain different values every time your application receives a new web request. The request context contains data specific to that particular web request such as form data, remote ip address, http headers, etc.
Every time a web request is received, new application and request contexts are created before your view function is called and these are then destroyed after a response has been returned and the processing of the web request has been completed. The g proxy is stored with the application context and it's used to store variables with a lifetime of that particular web request. This is used to pass information between functions called by flask as a result of events and signals.
In your code you use a decorator to register the function close_connection with the teardown_appcontext event, thus guaranteeing it is called after completion of handling of every web request. There are many other events that can be hooked to execute before or after your view function to provide pre- and post-processing of the request or response. The g proxy is merely used to store your database connection, which was created in your view function, and makes it available to the close_connection function where it is guaranteed to be closed. This prevents the database connection being left open if an error occurs in your application code.
The pattern, as shown below, is used to accomplish exactly the same thing as would be achieved by using a context manager (or try, except, finally block) to close the database connection within your function. This pattern is commonly used with an ORM to hide the underlying management of the database connection and reduce boilerplate code. The following code should produce the same effect without the use of events or the g proxy.
# -*- coding: utf-8 -*-
from flask import render_template, redirect
from application import app
from contextlib import closing
import sqlite3
def get_db():
return sqlite3.connect('test.db')
#app.route('/')
def index():
with closing(get_db()) as conn:
cur = conn.cursor()
cur.execute('select * from users')
select = cur.fetchall()
return render_template('index.html', title='DBSelectTest', posts=select)

Related

How to get a handle to the initialized database from a Flask application with SQLAlchemy?

I would like to add data to a database with Flask-SQLAlchemy without the Flask app running.
Is there a way to get db back from the app after the app and the database have been initialized.
My code looks like
db = SQLAlchemy()
def init_app():
app = Flask(__name__)
db = SQLAlchemy(app)
db.init_app(app)
return app
And what I would like to do is something like
from app import init_app
app = init_app() # initialized but not running
# db is used in model.py, but not initialized
# with Flask
# from db = SQLAlchemy()
from model import Machine # class Machine(db.Model)
p = Machine(name='something')
# now I need the initialized db from somewhere
db.session.add(p)
db.session.commit()
Basically I would like to do what's described here:
Another disadvantage is that Flask-SQLAlchemy makes using the database
outside of a Flask context difficult. This is because, with
FLask-SQLAlchemy, the database connection, models, and app are all
located within the app.py file. Having models within the app file, we
have limited ability to interact with the database outside of the app.
This makes loading data outside of your app difficult. Additionally,
this makes it hard to retrieve data outside of the Flask context.
Well, once you initialize the app, Flask spines a server (either development or production, whichever you set), so if you would like to add data to a database with Flask-SQLAlchemy without the Flask app running, you would better use the flask shell command which runs in the context of the current app, then you could add your data.
But first, it would be better is you set up your app as the following so we could directly import stuff like db, auth, etc:
...
from flask_sqlalchemy import SQLAlchemy
db = SQLAlchemy()
def init_app():
app = Flask(__name__)
db.init_app(app)
return app
In the root of your project, type in the terminal the following command
flask shell
Now that you have a shell running in the context of the current app but not the server not running:
from app import db
from model import Machine # class Machine(db.Model)
p = Machine(name='something')
# now I need the initialized db from somewhere
db.session.add(p)
db.session.commit()
From the wonderful tutorial...
https://blog.miguelgrinberg.com/post/the-flask-mega-tutorial-part-iv-database
Define something like this...
from app import app, db
from app.models import User, Post
#app.shell_context_processor
def make_shell_context():
return {'db': db, 'User': User, 'Post': Post}
And then...
(venv) $ flask shell
>>> db
<SQLAlchemy engine=sqlite:////Users/migu7781/Documents/dev/flask/microblog2/app.db>
>>> User
<class 'app.models.User'>
>>> Post
<class 'app.models.Post'>

Structuring your Flask project the right way (no Blueprint)

Intro
I've read like a thousand posts on SO and other sites trying to figure out what's wrong with my Flask structure and why I can't seem to figure out something. As a last resort, I decided to finally ask the question here.
My project is really simple:
I have to fetch some data from some networking devices via API, process the data and store it in a Postgresql DB (most part of the code is in lib/).
This project is going to be deployed on multiple environments (test, dev, staging and prod).
To do the above I'm using the following:
Flask-SQLAlchemy + Flask-Migrate + Flask-Script - all of these for handling migrations and DB related operations;
Python-dotenv - for handling sensitive configuration data
Project details
My project structure looks like this:
my_project/
├── api/
├── app.py
├── config.py
├── __init__.py
├── lib/
│   ├── exceptions.py
│   └── f5_bigip.py
├── log.py
├── logs/
├── manage.py
├── migrations/
├── models/
│   ├── __init__.py
│   ├── model1.py
│   └── model2.py
└── run.py
My app.py looks like this:
import os
import sys
sys.path.append(os.path.dirname(os.path.abspath(__file__)))
from dotenv import load_dotenv
from flask import Flask
from flask_migrate import Migrate
from flask_sqlalchemy import SQLAlchemy
db = SQLAlchemy()
migrate = Migrate()
def create_app():
load_dotenv()
app = Flask(__name__)
environment = app.config['ENV']
if environment == 'production':
app.config.from_object('config.ProductionConfig')
elif environment == 'testing':
app.config.from_object('config.TestingConfig')
else:
app.config.from_object('config.DevelopmentConfig')
db.init_app(app)
migrate.init_app(app, db)
return app
My config.py looks like this:
import os
from sqlalchemy.engine.url import URL
PROJECT_ROOT = os.path.dirname(os.path.realpath(__file__))
class BaseConfig:
DEBUG = False
TESTING = False
DB_DRIVERNAME = os.getenv('DB_DRIVERNAME')
DB_HOST = os.getenv('DB_HOST')
DB_PORT = os.getenv('DB_PORT')
DB_NAME = os.getenv('DB_NAME')
DB_USERNAME = os.getenv('DB_USERNAME')
DB_PASSWORD = os.getenv('DB_PASSWORD')
DB = {
'drivername': DB_DRIVERNAME,
'host': DB_HOST,
'port': DB_PORT,
'database': DB_NAME,
'username': DB_USERNAME,
'password': DB_PASSWORD,
}
SQLALCHEMY_DATABASE_URI = URL(**DB)
SQLALCHEMY_TRACK_MODIFICATIONS = False
class DevelopmentConfig(BaseConfig):
DEVELOPMENT = True
DEBUG = True
class TestingConfig(BaseConfig):
TESTING = True
class StagingConfig(BaseConfig):
DEVELOPMENT = True
DEBUG = True
class ProductionConfig(BaseConfig):
pass
My __init__.py looks like this:
from contextlib import contextmanager
from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker
# here, create_engine needs the SQLALCHEMY_DATABASE_URI
# how do I get it from the proper config?
engine = create_engine()
Session = sessionmaker(bind=engine)
#contextmanager
def session_scope():
"""
Provide a transactional scope around a series of operations.
"""
session = Session()
try:
yield session
session.commit()
except Exception as e:
print(f'Something went wrong here: {str(e)}. rolling back.')
session.rollback()
raise
finally:
session.close()
My manage.py looks like this:
from flask_script import Manager
from flask_migrate import MigrateCommand
from app import create_app
from models import *
manager = Manager(create_app)
manager.add_command('db', MigrateCommand)
if __name__ == '__main__':
manager.run()
My models/model1.py looks like this:
from sqlalchemy.dialects.postgresql import INET
from sqlalchemy.sql import func
from app import db
class Model1(db.Model):
__tablename__ = 'model1'
id = db.Column(db.BigInteger, primary_key=True, autoincrement=True)
ip_address = db.Column(INET, unique=True, nullable=False)
last_update = db.Column(db.DateTime(), server_default=func.now())
def __repr__(self):
return f'<Model1: {self.ip_address}>'
def __init__(self, ip_address):
self.ip_address = ip_address
Questions
Now, I have three main questions:
In my main __init__.py how can I import SQLALCHEMY_DATABASE_URI from the app's config?
Having the Session() object in the __init__.py doesn't seem too intuitive. Should it be placed in other places? For more context, the session is used in lib/f5_bigip.py and is probably going to be used in the api/ as well.
Is the overall project structure ok?
Your questions 1 and 2 are directly related to the part of your project that I found strange, so instead of answering those questions I'll just give you a much simpler and better way.
It seems in __init__.py you are implementing your own database sessions, just so that you can create a scoped session context manager. Maybe you've got that code from another project? It is not nicely integrated with the rest of your project, which uses the Flask-SQLAlchemy extension, you are just ignoring Flask-SQLAlchemy, which manages your database connections and sessions, and basically creating another connection into the database and new sessions.
What you should do instead is leverage the connections and sessions that Flask-SQLAlchemy provides. I would rewrite __init__.py as follows (doing this by memory, so excuse minor mistakes):
from contextlib import contextmanager
from app import db
#contextmanager
def session_scope():
"""
Provide a transactional scope around a series of operations.
"""
try:
yield db.session
session.commit()
except Exception as e:
print(f'Something went wrong here: {str(e)}. rolling back.')
db.session.rollback()
raise
finally:
db.session.close()
With this you are reusing the connection/sessions from Flask-SQLAlchemy. Your question 1 then is not a problem anymore. For question 2, you would use db.session anywhere in your app where you need database sessions.
Regarding question 3 I think you're mostly okay. I would suggest you do not use Flask-Script, which is a fairly old and unmaintained extension. Instead you can move your CLI to Flask's own CLI support.

How to correctly setup Flask + uWSGI + SQLAlchemy to avoid database connection issues

I am using a Flask app with SQLAlchemy and a uWSGI server. It is a known issue that uWSGI's duplicates connection through all the processes during forking. The information about how to fix this issue is a bit scatted in the web, but the two main options seems to be:
Using lazy-apps = true within uWSGI's config (Which is not recommended because it consumes a lot of memory)
Using uWSGI's #postfork decorator to close the connection after forking to start new fresh connections for each new process, but it's unclear for me when and how should be used within the Flask app.
This the sample of my app:
# file: my_app/app.py
db = SQLAlchemy()
def create_app():
app = Flask(__name__)
app.config.from_pyfile(f'../config/settings.py')
db.init_app(app)
db.create_all(app=app)
return app
Sample of the run.py file:
# file: run.py
from my_app/app import create_app
app = create_app()
if "__main__" == __name__:
app.run(debug=app.config["DEBUG"], port=5000)
So the question is where and how should the postfork executed to properly setup uWSGI's server to use isolated connections on each process without using lazy-apps = true?
The SQLAlchemy manual provides two examples how to approach this: Using Connection Pools with Multiprocessing.
The first approach involving Engine.dispose() can be approached using uwsgidecorators.postfork as suggested by Hett - simple example that should work if there's only the default binding involved:
db = SQLAlchemy()
def create_app():
app = Flask(__name__)
app.config["SQLALCHEMY_DATABASE_URI"] = "postgres:///test"
db.init_app(app)
def _dispose_db_pool():
with app.app_context():
db.engine.dispose()
try:
from uwsgidecorators import postfork
postfork(_dispose_db_pool)
except ImportError:
# Implement fallback when running outside of uwsgi...
raise
return app
The second one involves sqlalchemy.event and discarding connections from different PIDs - quoting:
The next approach is to instrument the Pool itself with events so that connections are automatically invalidated in the subprocess. This is a little more magical but probably more foolproof.
If you want the latter magical solution, see the docs.
In general you need to use #postfork decorator provided by uwsgi.
We are not using SQLAlchemy, but I think you can adapt our solution to your case as well.
Here's an excerpt from our uwsgi configuration:
[uwsgi]
strict = true
; process management
master = true
vacuum = true
pidfile = /tmp/uwsgi.pid
enable-threads = true
cpu-affinity = 1
processes = 4
threads = 2
mules = 2
...
And here's the application initialization code using #postfork to create process safe DB connections.
from flask import Flask
from uwsgidecorators import postfork
from app.lib.db import DB # DB object will be replaced by SQLAlchemy on your side
app = Flask(__name__)
# due to uwsgi's copy-on-write semantics workers need to connect to database
# to preserve unique descriptors after fork() is called in master process
#postfork
def db_reconnect():
# Open the DB connection and log the event
app.db = DB(config.get('db', 'host'),
config.get('db', 'user'),
config.get('db', 'pass'),
config.get('db', 'name'))
In our case all initialization code is placed in __init__.py or app folder, but I don't think you'll have any difficulties to migrate it into your app.py file.
Let me know if you'll encounter any other questions on this.

Unable to register a blueprint

CASE UPDATED: 2016-02-10 # 19:45 UTC
I've been researching for several weeks StackOverflow after reading a couple of books related to Flask, I've been testing and trying different configuration options, playing with several codes and finally I'm forced to write asking for help because I'm totally stuck and I can't go ahead alone. I'm sorry about that.
The problem: I'm not able to register a Blueprint correctly in my app. I'm doing something wrong or something is not being done in the right way. Routes that belong to the blueprint are not working.
The error received: Not Found. The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again.
I'm trying to build an API with Flask and Flask-Restful. I'm new to Flask and new to Flask-Restful too, so, you can imagine how big is my actual learning curve right now.
At present time, my project has this aspect:
/
├── PROJECT
│   ├── __init__.py
│   └── api_1_0
│   ├── __init__.py
│   ├── resources
│   │   ├── __init__.py
│   │   └── users.py
│   └── routes.py
├── config.py
└── run.py
The Plan
The PROJECT (named internally SelfCare) folder will have in the future two or three more packages inside (for web modules belonging to the same project). By now the only one of them is api_1_0.
The idea #1 is to build an API with some basic functionalities. Now, the focus is to present plain messages (return 'hi hi') just to confirm all blocks are working alltogether.
The idea #2 is to build the API as a blueprint inside the SelfCare package. Now i can have only 1 API, but later on i migth have two in parallel running (v1 and v2).
Some days ago i managed to get to work idea 1 but idea 2 is what i cannot get to work. Sorry.
My current files have this contents:
File: run.py
from SelfCare import create_app
app = create_app(config_name='development')
if __name__ == '__main__':
app.run()
File: config.py
import os
class Config:
SECRET_KEY = 'whatever'
APP_DIR = os.path.abspath(os.path.realpath(__file__))
STATIC_DIR = os.path.join(APP_DIR, 'static')
IMAGES_DIR = os.path.join(STATIC_DIR, 'images')
STYLES_DIR = os.path.join(STATIC_DIR, 'styles')
TEMPLATES_DIR = os.path.join(STATIC_DIR, 'templates')
# Method with an application instance (for now, just a bypass)
#staticmethod
def init_app(app):
pass
class DevelopmentConfig(Config):
DEBUG = True
class TestingConfig(Config):
DEBUG = True
class ProductionConfig(Config):
DEBUG = False
# Finally, this is a dead simple configuration dictionary
config = {
# Possible profiles to run under
'development': DevelopmentConfig,
'testing': TestingConfig,
'production': ProductionConfig,
# Default application profile
'default': DevelopmentConfig
}
File: PROJECT/init.py
from flask import Flask, render_template
from flask.ext.moment import Moment
from config import config
moment = Moment()
def create_app(config_name):
app = Flask(__name__)
app.config.from_object(config[config_name])
config[config_name].init_app(app)
# attach routes and custom error pages here
#app.route('/')
def index():
return 'Here we will start'
# HERE IT IS SUPPOSED THE BLUEPRINT IS REGISTERED
# AND ROUTES FROM BLUEPRINT ARE INCORPORATED TO MAIN PROGRAM
from .api_1_0 import api as api_blueprint
app.register_blueprint(api_blueprint)
return app
File: PROJECT/api_1_0/__init__.py
from flask import Blueprint
api = Blueprint('api', __name__)
from .resources import users
File: PROJECT/api_1_0/routes.py
from .api_1_0 import resources
#resources.route('/user', methods=['GET'])
def users():
return 'routes.py en blueprint api_1_0'
File: PROJECT/api_1_0/resources/__init__.py
This file is empty (a constructor).
File: PROJECT/api_1_0/resources/user.py
from flask_restful import Resource, reqparse, abort
# With the following dictionrary I do simulate a database connection
USERS = {
'user1': {'task': 'Build an API'},
'user2': {'task': 'Coordinate development'},
'user3': {'task': 'Build core'},
}
def abort_if_user_doesnt_exist(user):
if user not in USERS:
abort(404, message="User {} doesn't exist".format(user))
parser = reqparse.RequestParser()
parser.add_argument('task')
class User(Resource):
def get(self, user):
abort_if_user_doesnt_exist(user)
return USERS[user]
def delete(self, user):
abort_if_user_doesnt_exist(user)
del USERS[callsign]
return '', 204
def put(self, user):
args = parser.parse_args()
task = {'task': args['task']}
USERS[user] = task
return task, 201
So, routes.py seems not to work, so I cannot go further in the API. I'm stuck, sorry... Probably the error is just in from of me as a big stone and I cannot see it. Sorry.
Any help will be much more than appreciated.
Regards.
You're not registering flask-restful.Api and the REST routes correctly.
Install Flask-DebugToolbar - it will help you in showing your registered routes.
Change your code as follows:
In file config.py add the following two lines to the Config class:
DEBUG_TOOLBAR_ENABLED = True
REST_URL_PREFIX = '/api'
In file: PROJECT/init.py change to:
from flask import Flask, render_template
from flask.ext.moment import Moment
from config import config
moment = Moment()
def create_app(config_name):
app = Flask(__name__)
app.config.from_object(config[config_name])
config[config_name].init_app(app)
try:
from flask_debugtoolbar import DebugToolbarExtension
DebugToolbarExtension(app)
except:
pass
# attach routes and custom error pages here
#app.route('/')
def index():
return render_template('index.html')
# HERE IT IS SUPPOSED THE BLUEPRINT IS REGISTERED
# AND ROUTES FROM BLUEPRINT ARE INCORPORATED TO MAIN PROGRAM
from PROJECT.api_1_0 import api_bp, API_VERSION_V1
app.register_blueprint(
api_bp,
url_prefix='{prefix}/v{version}'.format(prefix=app.config['REST_URL_PREFIX'], version=API_VERSION_V1),
)
# This registers the actual routes
import api_1_0.routes
return app
In file: PROJECT/api_1_0/__init__.py change to:
from flask import Blueprint
import flask_restful
API_VERSION_V1 = 1
api_bp = Blueprint('api', __name__)
api_v1 = flask_restful.Api(api_bp)
Map your resources to routes; in file: PROJECT/api_1_0/routes.py change to:
from PROJECT.api_1_0 import api_v1
from PROJECT.api_1_0.resources.user import User
api_v1.add_resource(User, '/user/<string:user>')
Your routes will now be something like :
http://127.0.0.1:5000/api/v1/user/user1 which returns:
{
"task": "Build an API"
}
http://127.0.0.1:5000/api/v1/user/user2 which returns:
{
"task": "Coordinate development"
}
Note how the routes have been built up; the REST_URL_PREFIX from config plus the version number plus the url in the routes.py add_resource.
It works!
I was confused in the way the blueprints have to be registered. Maybe it was because I'm using PyCharm 5 Pro and, even now that the code is running like a charm, the IDE is telling me in the file PROJECT/api_1_0/___init____.py this message:
api_v1 = flask_restful.Api(api_bp)
api_bp --> Expected type 'Flask', got 'Blueprint' instead.
The inspection detects type errors in function call expressions. Due
to dynamic dispatch and duck typing, this is posible in a limite but
useful number of cases. Types of function parameters can be specified
in docstrings or in Python 3 functon annotations.
No I know I don't have to pay too much attention to this message (or that my IDE is wrongly configured). Many thanks for all.
Regards.

Importing Flask app when using app factory and flask script

This is Flask app context
app = Flask(__name__)
with app.app_context():
# insert code here
Most of the use cases of app context involves having 'app' initialized on the same script or importing app from the base.
My application is structured as the following:
# application/__init__.py
def create_app(config):
app = Flask(__name__)
return app
# manage.py
from application import create_app
from flask_script import Manager
manager = Manager(create_app)
manager.add_command("debug", Server(host='0.0.0.0', port=7777))
This might be really trivial issue, but how I should call 'with app.app_context()' if my application is structured like this?
Flask-Script calls everything inside the test context, so you can use current_app and other idioms:
The Manager runs the command inside a Flask test context. This means that you can access request-local proxies where appropriate, such as current_app, which may be used by extensions.
http://flask-script.readthedocs.org/en/latest/#accessing-local-proxies
So you don't need to use with app.app_context() with Manager scripts. If you're trying to do something else, then you'd have to create the app first:
from application import create_app
app = create_app()
with app.app_context():
# stuff here