I am trying to replace
jaydebeapi with the presto-python-client by facebook
the question is how to replace the authentication bit
db = jaydebeapi.connect(connection['jclass'], connection['host'],[ connection['user'], connection['pass']], connection['jar'])
while with presto-python-client
import prestodb
conn= prestodb.dbapi.connect(
host='localhost',
port=8080,
user='the-user',
catalog='the-catalog',
schema='the-schema',
isolation_level=transaction.IsolationLevel.REPEATABLE_READ,
)
and I couldn't find anywhere to specify how to pass a password.
The reason for this change is that I am getting ambigous errors when trying to pass long queries (18k characters) with jaydebeapi
we ended up using SQLAlchemy so the solution is
from sqlalchemy.engine import create_engine
engine = create_engine('presto://{0}:{1}#{2}:{3}/hive'.format(connection['user'],connection['pass'],connection['host'],int(connection['port'])), connect_args={'protocol': 'https', 'requests_kwargs': {'verify': False}})
db = engine.raw_connection()
Now the old method are deprecated, so here is the new one to connect to presto
cursor = presto.connect(presto_host,
presto_port,
requests_kwargs={'auth': HTTPBasicAuth(presto_username, presto_password)},
poll_interval=1,
protocol='https',
source='pyhive').cursor()
Related
I'm using flask_sqlalchemy in my flask application with a local MySQL (8.0.19) database. I've never got this issue before (started to develop this app months ago). Not sure what've changed, what component of the app got updated but I'm getting this error out of nowhere at the moment. I've searched and found that it might be some character encoding issue, but following the instructions I still get the warning when I open my app:
C:\Users\MyUserName\AppData\Local\Programs\Python\Python37\lib\site packages\pymysql\cursors.py:170:Warning:
(1366, "Incorrect string value: '\\xF6z\\xE9p-e...' for column 'VARIABLE_VALUE' at row 1")
result = self._query(query)
This is my url env variable:
MYSQL_URL = mysql+pymysql://user:passoword#localhost:3306/testdb?charset=utf8mb4
And this is how I create my db session:
db_url = os.getenv('MYSQL_URL')
engine = create_engine(db_url, echo=True)
Session = sessionmaker()
Session.configure(bind=engine)
session = Session()
This is the most simple usage of the session:
def row_count():
return (
session.query(Value.ValueID).count()
)
When I inspect this local database with HeidiSQL it says its collation is utf8mb4_0900_ai_ci. I don't know what those suffix specifics mean and there's a ton of utf8mb4 variant available. This is the default value.
Anyone has any idea how to resolve this warning? What does it mean exactly? As I'm using an ORM I'm not creating any database or running any query by hand, so how should I handle this?
ai : accent insensitive
ci : case insensitive
Did your try the following URL:
MYSQL_URL = mysql+pymysql://user:passoword#localhost:3306/testdb?charset=utf8mb4_ai_ci
I am trying to get the list of all the timezones supported by PSQL database in my Django project, so I can validate timestamps with timezones before sending them to the the database. I asked another question and got an answer regarding the PSQL query here:
How to get the list of timezones supported by PostgreSQL?
Using that, I am trying to do the following:
from django.db.models.expressions import RawSQL
RawSQL("SELECT name, abbrev, utc_offset, is_dst FROM pg_timezone_names;", [])
However, it does not seem to work. I saw the docs for RawSQL, and it usually has a model attached to it, which I can't really have. How to do solve this issue? Thanks.
Following should work for you
from django.db import connection
with connection.cursor() as cursor:
cursor.execute("SELECT name, abbrev, utc_offset, is_dst FROM pg_timezone_names")
zones = cursor.fetchall()
I'm currently trying to migrate my Flask Users over to a Django Backend.
However, when I'm using passlib to verify the hash, I can't figure out why it won't verify.
Our flask app settings
SECURITY_PASSWORD_HASH = "pbkdf2_sha512"
SECURITY_PASSWORD_SALT = "stackoverflow" # this is an example
An example of a hash I pulled from a database
flask_hash =
"$pbkdf2sha512$12000$ZQyhNEbIOSfk/J/T2vs/Bw$j.yxtixV.DqAcpsY9XTnJZZb3lCkR2fMWmV329Uc7Y/vz5Z0yMshEkYlUsE2Y9xm8TICwYkG55RgAplzZzLl7g"
So I created a custom pbkdf2_sha512 with the the rounds and salt
from passlib.hash import pbkdf2_sha512
rounds = 12000
salt = "stackoverflow".encode() # assume I swapped this out with the right salt
custom_pbkdf2 = pbkdf2_sha512.using(rounds=rounds, salt=salt)
verify_result = custom_pbkdf2.verify(hash=flask_hash, secret=password)
print (verify_result) # false
But if I create a new hash ... it does work
test_hash = custom_pbkdf2.hash('testing-if-this-works')
test_hash_confirm = custom_pbkdf2.verify('testing-if-this-works', hash=test_hash)
Is there something I'm missing? Thank you so much for any help here ... I know the password to this -- it's a dummy account I used for testing.
I was struck in exactly the same situation, luckily found this reddit thread, which had the explanation.
Basically, what you have to do verify the user is:
from flask_security.utils import verify_password
verify_password(<plain text password>, <password hash>)
More details here
Does anyone know how to access airflow environment variable using AIRFLOW_CONN_ and use in the python code. I know we can use hook to get the password, but have been trying to use AIRFLOW_CONN in my python to connect to the database. I have saved the connection in Airflow UI and in the docs, they mentioned to use AIRFLOW_CONN_ prefix to the conn_id to use. I used it in my python code using os.environ['AIRFLOW_CONN_REDSHIFT'], but it does not identify the environment variable. Please help.
Saving the connection to database and setting an AIRFLOW_CONN_ environment variable are two different ways to add a connection. You should only choose one way, unless you want them stored under connection ids.
Assuming you are running your python code through an operator like PythonOperator, you should be able to fetch your connection just like the BaseHook does.
Stored in database:
#classmethod
def _get_connections_from_db(cls, conn_id):
session = settings.Session()
db = (
session.query(Connection)
.filter(Connection.conn_id == conn_id)
.all()
)
session.expunge_all()
session.close()
if not db:
raise AirflowException(
"The conn_id `{0}` isn't defined".format(conn_id))
return db
Stored in environment variable:
#classmethod
def _get_connection_from_env(cls, conn_id):
environment_uri = os.environ.get(CONN_ENV_PREFIX + conn_id.upper())
conn = None
if environment_uri:
conn = Connection(conn_id=conn_id, uri=environment_uri)
return conn
Although I would recommend fetching it via a hook to avoid duplicating this code!
I am using:
A Python application in Bluemix
Bluemix cloudant v2.0.0b2 database linked to the Python app
According to https://pypi.python.org/pypi/cloudant/2.0.0b2, everything broke from 0.5 to 2.0, and they are still working on the documentation as everything is Beta. Next to this, I am also new to Python and databases. Documentation can be found here:
http://python-cloudant.readthedocs.io/en/latest/getting_started.html
What I am trying to do is check if a document already exists.
Things that I have tried:
from cloudant.account import Cloudant
import time
import json
# Connect to the database
client = Cloudant(*hidden*)
client.connect()
# The database we work in
db = client['myDatabase']
# The document we work on
doc = db['myDocument']
print doc.exists()
But the code fails before retrieving the document. I checked the source code, and it looks like it is supposed to:
def __getitem__(self, key):
if key in list(self.keys()):
return super(CouchDatabase, self).__getitem__(key)
if key.startswith('_design/'):
doc = DesignDocument(self, key)
else:
doc = Document(self, key)
if doc.exists():
doc.fetch()
super(CouchDatabase, self).__setitem__(key, doc)
return doc
else:
raise KeyError(key)
Source: https://pypi.python.org/pypi/cloudant/2.0.0b2
Is there a way I can check if the document exists before I retrieve it? Or should I retrieve it and catch the error? Or is there a different approach?
The behavior you are describing is the desired behavior for the python-cloudant library database object, so if you intend to use the database object to retrieve your documents and populate your local database cache you should look to except a KeyError in the event of a non-existent document and handle accordingly. However, if are interested in capturing whether a document exists before bringing it into your local database cache then changing your code to something like:
from cloudant.account import Cloudant
from cloudant.document import Document
# Connect to the database
client = Cloudant(*hidden*)
client.connect()
# The database we work in
db = client['myDatabase']
# The document we work on
if Document(db, 'myDocument').exists():
doc = db['myDocument']
would do the trick.
Similarly you could just do:
from cloudant.account import Cloudant
from cloudant.document import Document
# Connect to the database
client = Cloudant(*hidden*)
client.connect()
# The database we work in
db = client['myDatabase']
# The document we work on
doc = Document(db, 'myDocument')
if doc.exists():
doc.fetch()
But this would not populate your local database cache, the db dictionary.