Structuring your Flask project the right way (no Blueprint) - flask

Question

Intro
I've read like a thousand posts on SO and other sites trying to figure out what's wrong with my Flask structure and why I can't seem to figure out something. As a last resort, I decided to finally ask the question here.
My project is really simple:
I have to fetch some data from some networking devices via API, process the data and store it in a Postgresql DB (most part of the code is in lib/).
This project is going to be deployed on multiple environments (test, dev, staging and prod).
To do the above I'm using the following:
Flask-SQLAlchemy + Flask-Migrate + Flask-Script - all of these for handling migrations and DB related operations;
Python-dotenv - for handling sensitive configuration data
Project details
My project structure looks like this:
my_project/
├── api/
├── app.py
├── config.py
├── __init__.py
├── lib/
│   ├── exceptions.py
│   └── f5_bigip.py
├── log.py
├── logs/
├── manage.py
├── migrations/
├── models/
│   ├── __init__.py
│   ├── model1.py
│   └── model2.py
└── run.py
My app.py looks like this:
import os
import sys
sys.path.append(os.path.dirname(os.path.abspath(__file__)))
from dotenv import load_dotenv
from flask import Flask
from flask_migrate import Migrate
from flask_sqlalchemy import SQLAlchemy
db = SQLAlchemy()
migrate = Migrate()
def create_app():
load_dotenv()
app = Flask(__name__)
environment = app.config['ENV']
if environment == 'production':
app.config.from_object('config.ProductionConfig')
elif environment == 'testing':
app.config.from_object('config.TestingConfig')
else:
app.config.from_object('config.DevelopmentConfig')
db.init_app(app)
migrate.init_app(app, db)
return app
My config.py looks like this:
import os
from sqlalchemy.engine.url import URL
PROJECT_ROOT = os.path.dirname(os.path.realpath(__file__))
class BaseConfig:
DEBUG = False
TESTING = False
DB_DRIVERNAME = os.getenv('DB_DRIVERNAME')
DB_HOST = os.getenv('DB_HOST')
DB_PORT = os.getenv('DB_PORT')
DB_NAME = os.getenv('DB_NAME')
DB_USERNAME = os.getenv('DB_USERNAME')
DB_PASSWORD = os.getenv('DB_PASSWORD')
DB = {
'drivername': DB_DRIVERNAME,
'host': DB_HOST,
'port': DB_PORT,
'database': DB_NAME,
'username': DB_USERNAME,
'password': DB_PASSWORD,
}
SQLALCHEMY_DATABASE_URI = URL(**DB)
SQLALCHEMY_TRACK_MODIFICATIONS = False
class DevelopmentConfig(BaseConfig):
DEVELOPMENT = True
DEBUG = True
class TestingConfig(BaseConfig):
TESTING = True
class StagingConfig(BaseConfig):
DEVELOPMENT = True
DEBUG = True
class ProductionConfig(BaseConfig):
pass
My __init__.py looks like this:
from contextlib import contextmanager
from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker
# here, create_engine needs the SQLALCHEMY_DATABASE_URI
# how do I get it from the proper config?
engine = create_engine()
Session = sessionmaker(bind=engine)
#contextmanager
def session_scope():
"""
Provide a transactional scope around a series of operations.
"""
session = Session()
try:
yield session
session.commit()
except Exception as e:
print(f'Something went wrong here: {str(e)}. rolling back.')
session.rollback()
raise
finally:
session.close()
My manage.py looks like this:
from flask_script import Manager
from flask_migrate import MigrateCommand
from app import create_app
from models import *
manager = Manager(create_app)
manager.add_command('db', MigrateCommand)
if __name__ == '__main__':
manager.run()
My models/model1.py looks like this:
from sqlalchemy.dialects.postgresql import INET
from sqlalchemy.sql import func
from app import db
class Model1(db.Model):
__tablename__ = 'model1'
id = db.Column(db.BigInteger, primary_key=True, autoincrement=True)
ip_address = db.Column(INET, unique=True, nullable=False)
last_update = db.Column(db.DateTime(), server_default=func.now())
def __repr__(self):
return f'<Model1: {self.ip_address}>'
def __init__(self, ip_address):
self.ip_address = ip_address
Questions
Now, I have three main questions:
In my main __init__.py how can I import SQLALCHEMY_DATABASE_URI from the app's config?
Having the Session() object in the __init__.py doesn't seem too intuitive. Should it be placed in other places? For more context, the session is used in lib/f5_bigip.py and is probably going to be used in the api/ as well.
Is the overall project structure ok?

Answer 1

Your questions 1 and 2 are directly related to the part of your project that I found strange, so instead of answering those questions I'll just give you a much simpler and better way.
It seems in __init__.py you are implementing your own database sessions, just so that you can create a scoped session context manager. Maybe you've got that code from another project? It is not nicely integrated with the rest of your project, which uses the Flask-SQLAlchemy extension, you are just ignoring Flask-SQLAlchemy, which manages your database connections and sessions, and basically creating another connection into the database and new sessions.
What you should do instead is leverage the connections and sessions that Flask-SQLAlchemy provides. I would rewrite __init__.py as follows (doing this by memory, so excuse minor mistakes):
from contextlib import contextmanager
from app import db
#contextmanager
def session_scope():
"""
Provide a transactional scope around a series of operations.
"""
try:
yield db.session
session.commit()
except Exception as e:
print(f'Something went wrong here: {str(e)}. rolling back.')
db.session.rollback()
raise
finally:
db.session.close()
With this you are reusing the connection/sessions from Flask-SQLAlchemy. Your question 1 then is not a problem anymore. For question 2, you would use db.session anywhere in your app where you need database sessions.
Regarding question 3 I think you're mostly okay. I would suggest you do not use Flask-Script, which is a fairly old and unmaintained extension. Instead you can move your CLI to Flask's own CLI support.