I'm using flask_sqlalchemy in my flask application with a local MySQL (8.0.19) database. I've never got this issue before (started to develop this app months ago). Not sure what've changed, what component of the app got updated but I'm getting this error out of nowhere at the moment. I've searched and found that it might be some character encoding issue, but following the instructions I still get the warning when I open my app:
C:\Users\MyUserName\AppData\Local\Programs\Python\Python37\lib\site packages\pymysql\cursors.py:170:Warning:
(1366, "Incorrect string value: '\\xF6z\\xE9p-e...' for column 'VARIABLE_VALUE' at row 1")
result = self._query(query)
This is my url env variable:
MYSQL_URL = mysql+pymysql://user:passoword#localhost:3306/testdb?charset=utf8mb4
And this is how I create my db session:
db_url = os.getenv('MYSQL_URL')
engine = create_engine(db_url, echo=True)
Session = sessionmaker()
Session.configure(bind=engine)
session = Session()
This is the most simple usage of the session:
def row_count():
return (
session.query(Value.ValueID).count()
)
When I inspect this local database with HeidiSQL it says its collation is utf8mb4_0900_ai_ci. I don't know what those suffix specifics mean and there's a ton of utf8mb4 variant available. This is the default value.
Anyone has any idea how to resolve this warning? What does it mean exactly? As I'm using an ORM I'm not creating any database or running any query by hand, so how should I handle this?
ai : accent insensitive
ci : case insensitive
Did your try the following URL:
MYSQL_URL = mysql+pymysql://user:passoword#localhost:3306/testdb?charset=utf8mb4_ai_ci
Related
I am trying to make a simple DNS update using Python's dns.update. However, every-time i run the script i get "rcode REFUSED". I tried a series of different permutations but cant seem to figure where i am going wrong. I am able to directly use this key with nsupdates and make changes.
I am running this on Python 2.7
My key looks like this
key test.testdomain.com. {
algorithm HMAC-MD5;
secret "5MbEv7VrELN7ztkNMGSUvfimpoLAEzdmDzAHE9X4ax0ZDxiYnz1rkIx29SQru2AHQ3XbRBHmY7EQ/xD/2FocCA==";
};
Here is my code, I have hard-coded it all for the purpose of troubleshooting.
import sys
import dns.update
import dns.query
import dns.tsigkeyring
import dns.resolver
def main():
UpdateDNS()
####################################################################################################################
def UpdateDNS():
# set zone and dnsserver
zone = 'testdomain.com'
dnshostname = 'dns-test.testdomain.com'
keyring = dns.tsigkeyring.from_text ({'test.testdomain.com.' : '5MbEv7VrELN7ztkNMGSUvfimpoLAEzdmDzAHE9X4ax0ZDxiYnz1rkIx29SQru2AHQ3XbRBHmY7EQ/xD/2FocCA=='})
update = dns.update.Update(zone, keyring = keyring, keyalgorithm = 'hmac-md5.sig-alg.reg.int')
update.add('foo.testdomain.com', 8600, 'A', '179.33.72.36')
response = dns.query.tcp(update, 'dns-test.testdomain.com')
print response
#########################################################
# Main
#########################################################
if __name__ == '__main__':
main()
Here is my response
x1c\x08'}
id 45721
opcode UPDATE
rcode REFUSED
flags QR RA
;ZONE
testdomain.com. IN SOA
;PREREQ
;UPDATE
;ADDITIONAL
Generally your code looks OK to me. I just tested essentially the same code on my name server and it works like a charm.
Did you allow updates for the TSIG key to the zone you're trying to update? There should be something like this in your bind config (probably there as you wrote you can use the key manually, but just to make sure):
zone "testdomain.com" IN {
type master;
[...]
allow-update {
key "test.testdomain.com.";
};
};
What do the name server logs say when you run your update script? Normally there should be a reason for rejecting the update:
view internal: signer "test-key" denied
view internal: request has invalid signature: TSIG test-key: tsig verify failure (BADKEY)
The former would indicate that the key is not allowed to update the zone, the latter that the key itself wasn't accepted (though that would also have resulted in an exception when running the code).
I am trying to replace
jaydebeapi with the presto-python-client by facebook
the question is how to replace the authentication bit
db = jaydebeapi.connect(connection['jclass'], connection['host'],[ connection['user'], connection['pass']], connection['jar'])
while with presto-python-client
import prestodb
conn= prestodb.dbapi.connect(
host='localhost',
port=8080,
user='the-user',
catalog='the-catalog',
schema='the-schema',
isolation_level=transaction.IsolationLevel.REPEATABLE_READ,
)
and I couldn't find anywhere to specify how to pass a password.
The reason for this change is that I am getting ambigous errors when trying to pass long queries (18k characters) with jaydebeapi
we ended up using SQLAlchemy so the solution is
from sqlalchemy.engine import create_engine
engine = create_engine('presto://{0}:{1}#{2}:{3}/hive'.format(connection['user'],connection['pass'],connection['host'],int(connection['port'])), connect_args={'protocol': 'https', 'requests_kwargs': {'verify': False}})
db = engine.raw_connection()
Now the old method are deprecated, so here is the new one to connect to presto
cursor = presto.connect(presto_host,
presto_port,
requests_kwargs={'auth': HTTPBasicAuth(presto_username, presto_password)},
poll_interval=1,
protocol='https',
source='pyhive').cursor()
Does anyone know how to access airflow environment variable using AIRFLOW_CONN_ and use in the python code. I know we can use hook to get the password, but have been trying to use AIRFLOW_CONN in my python to connect to the database. I have saved the connection in Airflow UI and in the docs, they mentioned to use AIRFLOW_CONN_ prefix to the conn_id to use. I used it in my python code using os.environ['AIRFLOW_CONN_REDSHIFT'], but it does not identify the environment variable. Please help.
Saving the connection to database and setting an AIRFLOW_CONN_ environment variable are two different ways to add a connection. You should only choose one way, unless you want them stored under connection ids.
Assuming you are running your python code through an operator like PythonOperator, you should be able to fetch your connection just like the BaseHook does.
Stored in database:
#classmethod
def _get_connections_from_db(cls, conn_id):
session = settings.Session()
db = (
session.query(Connection)
.filter(Connection.conn_id == conn_id)
.all()
)
session.expunge_all()
session.close()
if not db:
raise AirflowException(
"The conn_id `{0}` isn't defined".format(conn_id))
return db
Stored in environment variable:
#classmethod
def _get_connection_from_env(cls, conn_id):
environment_uri = os.environ.get(CONN_ENV_PREFIX + conn_id.upper())
conn = None
if environment_uri:
conn = Connection(conn_id=conn_id, uri=environment_uri)
return conn
Although I would recommend fetching it via a hook to avoid duplicating this code!
I am using:
A Python application in Bluemix
Bluemix cloudant v2.0.0b2 database linked to the Python app
According to https://pypi.python.org/pypi/cloudant/2.0.0b2, everything broke from 0.5 to 2.0, and they are still working on the documentation as everything is Beta. Next to this, I am also new to Python and databases. Documentation can be found here:
http://python-cloudant.readthedocs.io/en/latest/getting_started.html
What I am trying to do is check if a document already exists.
Things that I have tried:
from cloudant.account import Cloudant
import time
import json
# Connect to the database
client = Cloudant(*hidden*)
client.connect()
# The database we work in
db = client['myDatabase']
# The document we work on
doc = db['myDocument']
print doc.exists()
But the code fails before retrieving the document. I checked the source code, and it looks like it is supposed to:
def __getitem__(self, key):
if key in list(self.keys()):
return super(CouchDatabase, self).__getitem__(key)
if key.startswith('_design/'):
doc = DesignDocument(self, key)
else:
doc = Document(self, key)
if doc.exists():
doc.fetch()
super(CouchDatabase, self).__setitem__(key, doc)
return doc
else:
raise KeyError(key)
Source: https://pypi.python.org/pypi/cloudant/2.0.0b2
Is there a way I can check if the document exists before I retrieve it? Or should I retrieve it and catch the error? Or is there a different approach?
The behavior you are describing is the desired behavior for the python-cloudant library database object, so if you intend to use the database object to retrieve your documents and populate your local database cache you should look to except a KeyError in the event of a non-existent document and handle accordingly. However, if are interested in capturing whether a document exists before bringing it into your local database cache then changing your code to something like:
from cloudant.account import Cloudant
from cloudant.document import Document
# Connect to the database
client = Cloudant(*hidden*)
client.connect()
# The database we work in
db = client['myDatabase']
# The document we work on
if Document(db, 'myDocument').exists():
doc = db['myDocument']
would do the trick.
Similarly you could just do:
from cloudant.account import Cloudant
from cloudant.document import Document
# Connect to the database
client = Cloudant(*hidden*)
client.connect()
# The database we work in
db = client['myDatabase']
# The document we work on
doc = Document(db, 'myDocument')
if doc.exists():
doc.fetch()
But this would not populate your local database cache, the db dictionary.
I have some code in a Django app which does the following, to get a Pubmed article by DOI:
def getPubmedByDOI(request,doi):
Entrez.email = 'me#mydomain.com'
handle = Entrez.esearch(db="pubmed", term=doi)
record = Entrez.read(handle)
return getPubmedArticle(request,record["IdList"][0]) // renders the article
This works nicely but for one thing - the Entrez.esearch call insists upon access to /root/.config on the server, specifically to write to the following empty directory:
/root/.config/biopython/Bio/Entrez/DTDs/
It's Apache on Gentoo, running as follows:
User django
Group apache
All the code for the application is in ~django/, so I'd expect any writing to be in ~django/.config rather than /root/.config. I can work around this by changing permissions on /root but a better solution would be to configure Biopython or Apache so as not to write to /root. Does anyone have any suggestions as to how this might be done?
Logged upstream as https://github.com/biopython/biopython/issues/918 which suggests setting:
>>> from Bio.Entrez.Parser import DataHandler
>>> DataHandler.global_dtd_dir
'/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/Bio/Entrez/DTDs'
>>> DataHandler.local_dtd_dir = '...'