I'm reading DATABASE_URL from env variable in uwsgi file. My django ORM connection is working with % in password but when I try to connect with sqlalchemy it is not working.
from sqlalchemy import orm, create_engine, MetaData
from app.settings import DATABASES
from urllib import quote_plus
con = "mysql+pymysql://{0}:{1}#{2}:{3}/{4}".format(
DATABASES['default']['USER'],
DATABASES['default']['PASSWORD'],
DATABASES['default']['HOST'],
DATABASES['default']['PORT'],
DATABASES['default']['NAME']
)
engine = create_engine(
con,
echo=False,
pool_recycle=1800,
max_overflow = 250,
pool_size=200
)
I even tried quote_plus but it did not worked.
This does not works
mysql+pymysql://api:#>J.6D%qhZ#localhost:3306/table_tmp
This works
mysql+pymysql://api:#>J.6DqhZ#localhost:3306/table_tmp
When I use same password with % via ipython it works but via env variable in uwsgi.ini file it doesn't
Related
I'm currently migrating my Flask app from flask-sqlalchemy 2.5.1 to 3.0.2. The app includes an endpoint that lets the client create a new database file at a selected path. Before the migration, it was achieved by simply setting the path and creating the tables in the following way:
app.config['SQLALCHEMY_DATABASE_URI'] = filepath
db.create_all()
However, the file is no longer automatically created in 3.0.2.
Question
I have been digging through the flask-sqlalchemy and sqlalchemy documentations for the past two days yet I can't find anything that would mention the changed behaviour. Creating the file before setting the config doesn't work either, as the tables are not created and the file size is 0b after calling the endpoint.
How can I create a new sqlalchemy DB file inside of a Flask endpoint?
Below is a minimal reproducible example:
server.py
from flask import Flask
from flask_sqlalchemy import SQLAlchemy
import os
TEST_DB_PATH = os.path.join(os.getcwd(), "test.db")
# create the extension
db = SQLAlchemy()
class User(db.Model):
id = db.Column(db.Integer, primary_key=True)
# create the app
app = Flask(__name__)
app.config["SQLALCHEMY_TRACK_MODIFICATIONS"] = True
# configure the first SQLite database, in memory
app.config["SQLALCHEMY_DATABASE_URI"] = "sqlite:///"
# initialize the app with the extension
db.init_app(app)
with app.app_context():
db.create_all()
#app.route('/test')
def test():
# configure the second SQLite database, relative to the app instance folder
app.config["SQLALCHEMY_DATABASE_URI"] = "sqlite:///" + TEST_DB_PATH
db.create_all()
return "OK", 200
if __name__ == '__main__':
app.run()
test_server.py
import server
import os
import pytest
#pytest.fixture(scope="session")
def client():
app = server.app
return app.test_client()
def test_create_db_file(client):
# clean local test.db file if it exists
try:
os.remove(server.TEST_DB_PATH)
except OSError:
pass
ans = client.get("test")
assert ans.status_code == 200, ans.data
# this assertion fails
assert os.path.exists(
server.TEST_DB_PATH), f"DB file created at execution cannot be found at " + server.TEST_DB_PATH
assert os.path.getsize(
server.TEST_DB_PATH) > 0, "Created db file size is 0"
Running the test
py -m pytest .\test_server.py
The assertions fail with flask-sqlalchemy 3.0.2 and pass with 2.5.1.
Environment
Windows 11 x64
flask-sqlalchemy 2.2.2
sqlalchemy 1.4.46
pytest 7.2.0
python 3.9.13
I'm very new on GCP and I've tried everything around to achieve this connection, my app connects to any other postgress service but not to Cloud SQL.
I've started to think that is a problem with my code in flask.
So far I have followed this guide:
https://cloud.google.com/sql/docs/postgres/connect-run#public-ip-default
and watched some Youtube Tutorials.
Maybe my problem is in the Cloud Proxy Auth.
MyCode looks like this:
import os
from os.path import join, dirname
from flask import Flask
from flask_cors import CORS
from flask_restful import Api
from flask_jwt_extended import JWTManager
from flask_bcrypt import Bcrypt
from flask_migrate import Migrate
from constants import ACTIVE_SCHEDULER
from db import db
from ma import ma
from resources.routes import initialize_routes, initialize_errors, initialize_cli
from sched import init_scheduler
app = Flask(__name__)
db_user = os.environ.get("DB_USER")
db_pass = os.environ.get("DB_PASS")
db_name = os.environ.get("DB_NAME")
unix_socket_path = os.environ.get("INSTANCE_UNIX_SOCKET")
db_url = 'postgresql+pg8000://{user}:{pw}#/{db}?unix_sock={path}/.s.PGSQL.5432'.format(user=db_user,pw=db_pass,db=db_name, path=unix_socket_path)
app.config["SQLALCHEMY_DATABASE_URI"] = db_url
app.config["SECRET_KEY"] = os.environ.get('JWT_SECRET_KEY', '')
app.config["SQLALCHEMY_TRACK_MODIFICATIONS"] = False
app.config["PROPAGATE_EXCEPTIONS"] = True
and the error that I always get is:
"error": "(pg8000.exceptions.InterfaceError) Can't create a connection to host 172.17.0.1 and port 5432 (timeout is None and source_address is None). (Background on this error at: https://sqlalche.me/e/14/rvf5)"
I'm out of ideas, I hope anyone knows any guide or solution to my problem, Maybe I'm missing some access to the service account, but I think I follow correctly the instructions I even try to connect Privately with the TCP instructions and is almost the same.
I would like to add data to a database with Flask-SQLAlchemy without the Flask app running.
Is there a way to get db back from the app after the app and the database have been initialized.
My code looks like
db = SQLAlchemy()
def init_app():
app = Flask(__name__)
db = SQLAlchemy(app)
db.init_app(app)
return app
And what I would like to do is something like
from app import init_app
app = init_app() # initialized but not running
# db is used in model.py, but not initialized
# with Flask
# from db = SQLAlchemy()
from model import Machine # class Machine(db.Model)
p = Machine(name='something')
# now I need the initialized db from somewhere
db.session.add(p)
db.session.commit()
Basically I would like to do what's described here:
Another disadvantage is that Flask-SQLAlchemy makes using the database
outside of a Flask context difficult. This is because, with
FLask-SQLAlchemy, the database connection, models, and app are all
located within the app.py file. Having models within the app file, we
have limited ability to interact with the database outside of the app.
This makes loading data outside of your app difficult. Additionally,
this makes it hard to retrieve data outside of the Flask context.
Well, once you initialize the app, Flask spines a server (either development or production, whichever you set), so if you would like to add data to a database with Flask-SQLAlchemy without the Flask app running, you would better use the flask shell command which runs in the context of the current app, then you could add your data.
But first, it would be better is you set up your app as the following so we could directly import stuff like db, auth, etc:
...
from flask_sqlalchemy import SQLAlchemy
db = SQLAlchemy()
def init_app():
app = Flask(__name__)
db.init_app(app)
return app
In the root of your project, type in the terminal the following command
flask shell
Now that you have a shell running in the context of the current app but not the server not running:
from app import db
from model import Machine # class Machine(db.Model)
p = Machine(name='something')
# now I need the initialized db from somewhere
db.session.add(p)
db.session.commit()
From the wonderful tutorial...
https://blog.miguelgrinberg.com/post/the-flask-mega-tutorial-part-iv-database
Define something like this...
from app import app, db
from app.models import User, Post
#app.shell_context_processor
def make_shell_context():
return {'db': db, 'User': User, 'Post': Post}
And then...
(venv) $ flask shell
>>> db
<SQLAlchemy engine=sqlite:////Users/migu7781/Documents/dev/flask/microblog2/app.db>
>>> User
<class 'app.models.User'>
>>> Post
<class 'app.models.Post'>
Experts!
I am a beginner in Flask.
I have any project and it includes some models(already defined).
This is one of my models and these are placed as other files, not in app.py.
from sqlalchemy.ext.declarative import declarative_base
Base = declarative_base()
from sqlalchemy import Column, Integer, SmallInteger, String, Text, Date, Boolean, Float
class CategoryTopicLink(Base):
__tablename__ = 'category_topic_link'
id = Column(Integer, primary_key=True)
category_id = Column(Integer)
topic_id = Column(Integer)
And I am going to migrate these by running app.py.
So I inputted some code in app.py and my app file look at following:
from flask import Flask, jsonify, request, make_response
...
from flask_sqlalchemy import SQLAlchemy
from flask_migrate import Migrate
app = Flask(__name__)
CORS(app)
api = Api(app)
db = SQLAlchemy(app)
migrate = Migrate(app, db)
from .models import CategoryTopicLink
........
if __name__=="__main__":
app.run(debug=True,host="127.0.0.1",port="5001")
And To migrate model I commanded
"python3 app.py db init" (but server run)
"python3 app.py db migrate" (also server run, not migrate)
"flask run db init" (error - flask has not db attribute)
"flask run db migrate" (error - flask has not db attribute)
......
Please let me know how to migrate models.
Any help will be appreciated.
Regards, from jis0324!!!
The commands are always in the form
flask db command
Also see the manual
https://flask-migrate.readthedocs.io/en/latest/
I am using django 1.11 on python 3.6.
The easiest way I can explain my problem is that I am creating a test database on the fly in my code. I do this with the following script:
from uuid import uuid4 as uuid
db = '_' + uuid().hex + 'db.sqlite3'
os.environ.setdefault('DJANGO_TEST_DB_NAME', db)
print(db)
import django
from django.apps import apps
from django.conf import settings
from django.core.management import call_command
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "project.settings")
django.setup()
print("Setting up test database\n")
print("Clearing migrations...\n")
_dir = os.getcwd()
_search = ['project', 'email_auth', 'google', 'app2', 'app3']
for _d in _search:
_walk = os.walk(_dir + os.path.sep + _d)
for _walkObj in _walk:
if _walkObj[0].split(os.path.sep)[-1] == 'migrations' or \
(_walkObj[0].split(os.path.sep)[-2] == 'migrations'
and _walkObj[0].split(os.path.sep)[-1] == '__pycache__'):
for _fName in _walkObj[2]:
os.remove(_walkObj[0] + os.path.sep + _fName)
print("Calling manage:makemigrations\n")
call_command('makemigrations', *_search[1:])
apps.clear_cache()
apps.populate(settings.INSTALLED_APPS)
print("Calling manage:migrate...\n")
call_command('migrate')
print("Creating objects...\n")
call_command('create_objects')
print("Starting server...\n")
call_command('runserver')
No matter what I do to the apps object (I tried some hacky things to clear out everything inside of it to reload, but no dice) or anything else I cannot get django to realize that there are migrations created for any of my apps when calling migrate.
I have even attempted just calling into migrate, email_auth and it states that django.core.management.base.CommandError: App 'email_auth' does not have migrations.
I can call migrate outside of this script if I cancel it after the makemigrations portion and it migrates just fine.
I strongly suspect it's not working because somewhere django has the old references to the modules and I have no idea how to update them.
Edit:
Just so there is proof that it is indeed migrating, I uploaded a paste bin of console output: https://pastebin.com/aSructTW
I finally figured out how to do this properly. Please note that there is more to getting the test database to be created/used, however this question's scope is for reloading the apps. If anyone wants code for that let me know. Here is the function to clear out apps:
def resetApps(search=[]):
from collections import OrderedDict, defaultdict
import os
import sys
from django.apps import apps
from django.conf import settings
apps.clear_cache()
apps.all_models = defaultdict(OrderedDict)
apps.app_configs = OrderedDict()
apps.stored_app_configs = []
apps.apps_ready = apps.models_ready = apps.ready = False
apps._pending_operations = defaultdict(list)
# clear out py caches of apps as we need to reload
# modify this to allow removal of .pyc files that aren't in pycache, btw
for _d in search:
_walk = os.walk(_dir + os.path.sep + _d)
for _walkObj in _walk:
if _walkObj[0].split(os.path.sep)[-1] == '__pycache__':
for _fName in _walkObj[2]:
os.remove(_walkObj[0] + os.path.sep + _fName)
# clear out sys modules of required things
_toClear = []
for module in sys.modules:
if module.split('.')[0] in search:
_toClear.append(module)
for module in _toClear:
del sys.modules[module]
apps.populate(settings.INSTALLED_APPS)