I am trying to store u'\U0001f381' string in Azure sql server from python 2.7.11 in ubuntu 14.04 LTS. I have set the column type as nvarchar(MAX) so that it will accept unicode strings.
following is the python script:
import pymssql
from creds import *
conn = pymssql.connect(host=HOST, user=USER, password=PASSWORD, database=DATABASE)
cursor = conn.cursor()
lst = [u'2017-07-04', u'\U0001f3e8', 1.0, 0.0, 0.0, 9.0]
print lst
placeholder = '%s,' * len(lst)
query = 'INSERT INTO Example_SearchAnalytics VALUES ( '+placeholder.rstrip(',')+ ')'
cursor.execute(query,tuple(lst))
conn.commit()
conn.close()
But I am getting following error when I execute above script from ubuntu environment.
pymssql.OperationalError: (105, "Unclosed quotation mark after the
character string ''.DB-Lib error message 20018, severity 15:\nGeneral
SQL Server error: Check messages from the SQL Server\nDB-Lib error
message 20018, severity 15:\nGeneral SQL Server error: Check messages
from the SQL Server\n")
I don't get any error when I execute same script from windows environment. I think I need to escape any character from unicode string but I am not sure which.
Please help.
Repeat quotation marks or use CHAR(39) as explained on below thread:
Escaping single quote in SQL Server
Hope this helps.
Regards,
Alberto Morillo
Related
I have a problem getting UTF-8 strings from MySQL database. I use C++ connector 1.1 and connect with following code:
sql::ConnectOptionsMap connection_properties;
connection_properties["hostName"] = server;
connection_properties["userName"] = user;
connection_properties["password"] = password;
connection_properties["schema"] = database;
connection_properties["port"] = 3306;
connection_properties["OPT_CHARSET_NAME"] = "utf8";
connection_properties["characterSetResults"] = "utf8";
connection_properties["preInit"] = "SET NAMES utf8";
driver = get_driver_instance();
con = driver->connect(connection_properties);
con->setSchema(database);
I already tried different utf8 options as you see....
If a statement should return database strings like "アフガニスタン" I only see chars like this "アフガニスタン" when I use Visual Studio debugger. The observed code:
std::string name = res->getString(2);
After Json encode it prints "ÒéóÒâòÒé¼ÒâïÒé╣Òé┐Òâ│" into command line.
Other utf8 columns with normal latin characters are returned as expected. It only affects translation columns with non latin chars.
Same database call from PHP with same logic (db connection and json encode) on same PC prints out following chars "\u30a2\u30d5\u30ac\u30cb\u30b9\u30bf\u30f3".
Any ideas about that?
Actually there is no problem. I wrote returned data into a file and all UTF-8 characters are correct. Debugger and CMD are not able to display UTF-8 data as expected...
I have a fasta file with sequence that I want to blast locally to 'nt' database dowloaded on my computer from ncbi website
I dowloaded blast 2.6.0.
In order to access blast from anywhere, I did:
gedit ~/.bashrc
export PATH=/usr/local/ncbi-blast-2.6.0+/bin:$PATH
then I did:
source ~/.bashrc
Then I downloaded 'nt' database (155.6GB) and stored it in /usr/local/blastdb
I want to run in python script this command:
from Bio.Blast.Applications import NcbiblastnCommandline
cline = NcbiblastnCommandline(query="/home/proprietaire/Desktop/JADE/stage_scripts/seq_error_fasta.fasta", db="/usr/local/blastdb/nt", evalue=0.001, out="blast_result_local.xml", outfmt=5)
But it is not working for a reason. Please help me figure out what I'm doing wrong. Thank you for your help.
EDIT:
'seq_error_fasta.fasta' : is my fasta file with 64 sequences that I want to blast to 'nt' database.
My 'seq_error_fasta.fasta' contains sequence loaded with error like S, J, X so I want to blast them to 'nt' db in order to get the closest better sequence
I found out that I need to format the nt database dowloaded from ncbi so I did this:
makeblastdb -dbtype nucl -in nt
Then I added this after my cline variable in my python script:
stdout, stderr = cline()
The script is running but unfortunately I'm getting this error now:
Bus error (core dumped)
I think it's a ram memory problem so I thought that I need to shorten 'nt' db by taking only the bacteria sequence. I looked on NCBI for a whole bacteria only database but there is multiple database of different species like more then a thousand.
I also tried blast online using this script:
f = open('output_blast.xml','w')
for rec in SeqIO.parse(open("seq_error_fasta.fasta"), 'fasta):
result_handle = NCBIWWW.qblast("blastn", "nt", rec.format("fasta"), format_type="XML", alignments=1, perc_ident=95, expect= 0.001)
f.write(result_handle.read())
f.close()
but this only doing one query sequence and returning all hits, althought I specified 1 hit and 95% of identity.
This is driving me crazy lollll Please help
Download NCBI nr database:
$ mkdir db
$ cd db
$ wget ftp://ftp.ncbi.nlm.nih.gov/blast/db/nr*
Run blast:
import io
import shlex
import subprocess
BLAST_OUTFMT6 = """\
'6 qacc sacc pident length mismatch gapopen qstart qend sstart send evalue bitscore qseq sseq'\
"""
BLAST_OUTFMT6_COLUMN_NAMES = [
'query_id', 'subject_id', 'pc_identity', 'alignment_length', 'mismatches', 'gap_opens',
'q_start', 'q_end', 's_start', 's_end', 'evalue', 'bitscore', 'qseq', 'sseq',
]
def blastp(sequence, db, evalue=0.001, max_target_seqs=100000):
system_command = (
'blastp -db {db} -outfmt {outfmt} -evalue {evalue} -max_target_seqs {max_target_seqs}'
.format(db=db, outfmt=BLAST_OUTFMT6, evalue=evalue, max_target_seqs=max_target_seqs)
)
cp = subprocess.Popen(
shlex.split(system_command),
stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE,
universal_newlines=True)
result, error_message = cp.communicate(sequence)
if error_message.strip():
print("Error: {}".format(error_message))
return result
if __name__ == '__main__':
result = blastp('AAAAAAAAAAAAAA', db='/path/to/db/nr')
print(result)
I want to upload a string in unicode to my sql server. I'm using python 2.7.6, sqlalchemy-migrate 0.7.2, pymssql 2.1.2.
But when I saved my object I got an OperationalError from sqlalchemy
OperationalError: (OperationalError) (105, "Unclosed quotation mark
after the character string '\xd8\xa3\xd8\xb3\xd8\xb1\xd8\xa7\xd8\xb1
\xd8\xaa\xd8\xad\xd8\xaf\xd9\x8a\xd8\xaf\xd8\xa7\xd9\x84\xd9\x88\xd8
\xac\xd9\x87 - \xd9\x81\xd9\x82\xd8\xb7 \xd9\x84\xd8\xaf\xd9\x89\xd9
\x85\xd8\xad\xd9\x84\xd8\xa7\xd8\xaa \xd9\x88\xd8\xac\xd9\x88\xd9\x87
\xe2\x9c\x8e '.DB-Lib error message 105, severity 15:\nGeneral SQL
Server error: Check messages from the SQL Server\n") 'INSERT INTO...
With more detail I'm guesssing is from my Description value :
...\u0648\u062c\u0648\u0647 \u270e \U0001f38'}
I saw a big U and not u, the character is the gift unicode just before, the "\u270e" works well and show a pencil. I strongly thing is because of the 8 values versus 4 for others.
But how to prevent this error ?
The column description inside of my DB is a nvarchar(2000)
I'm using Using flask restful reqparse to parse the argument create sync the object from the DB and save it :
parser_edit.add_argument('Name',
type=unicode,
required=True,
location='json')
parser_edit.add_argument('Description',
type=unicode,
required=False,
location='json')
I collected user data using a wx python gui and than I used uno to fill this data into an openoffice document under ubuntu 10.xx
user + my-script ( +empty document ) --> prefilled document
After upgrading to ubuntu 14.04 uno doesn't work with python 2.7 anymore and now we have libreoffice instead of openoffice in ubuntu. when I try to run my python2.7 code, it says:
ImportError: No module named uno
How could I bring it back to work?
what I tried:
installed https://pypi.python.org/pypi/unotools v0.3.3
sudo apt-get install libreoffice-script-provider-python
converted the code to python3 and got uno importable, but wx is not importable in python3 :-/
ImportError: No module named 'wx'
googled and read python3 only works with wx phoenix
so tried to install: http://wxpython.org/Phoenix/snapshot-builds/
but wasn't able to get it to run with python3
is there a way to get the uno bridge to work with py2.7 under ubuntu 14.04?
Or how to get wx to run with py3?
what else could I try?
Create a python macro in LibreOffice that will do the work of inserting the data into LibreOffice and then in your python 2.7 code envoke the macro.
As the macro is running from with LibreOffice it will use python3.
Here is an example of how to envoke a LibreOffice macro from the command line:
#!/usr/bin/python3
# -*- coding: utf-8 -*-
##
# a python script to run a libreoffice python macro externally
# NOTE: for this to run start libreoffice in the following manner
# soffice "--accept=socket,host=127.0.0.1,port=2002,tcpNoDelay=1;urp;" --writer --norestore
# OR
# nohup soffice "--accept=socket,host=127.0.0.1,port=2002,tcpNoDelay=1;urp;" --writer --norestore &
#
import uno
from com.sun.star.connection import NoConnectException
from com.sun.star.uno import RuntimeException
from com.sun.star.uno import Exception
from com.sun.star.lang import IllegalArgumentException
def uno_directmacro(*args):
localContext = uno.getComponentContext()
localsmgr = localContext.ServiceManager
resolver = localsmgr.createInstanceWithContext("com.sun.star.bridge.UnoUrlResolver", localContext )
try:
ctx = resolver.resolve("uno:socket,host=localhost,port=2002;urp;StarOffice.ComponentContext")
except NoConnectException as e:
print ("LibreOffice is not running or not listening on the port given - ("+e.Message+")")
return
msp = ctx.getValueByName("/singletons/com.sun.star.script.provider.theMasterScriptProviderFactory")
sp = msp.createScriptProvider("")
scriptx = sp.getScript('vnd.sun.star.script:directmacro.py$directmacro?language=Python&location=user')
try:
scriptx.invoke((), (), ())
except IllegalArgumentException as e:
print ("The command given is invalid ( "+ e.Message+ ")")
return
except RuntimeException as e:
print("An unknown error occurred: " + e.Message)
return
except Exception as e:
print ("Script error ( "+ e.Message+ ")")
print(e)
return
return(None)
uno_directmacro()
And this is the corresponding macro code within LibreOffice called "directmacro.py" and stored in the User area for libreOffice macros (which would normally be $HOME/.config/libreoffice/4/user/Scripts/python :
#!/usr/bin/python
from com.sun.star.awt.MessageBoxButtons import BUTTONS_OK, BUTTONS_OK_CANCEL, BUTTONS_YES_NO, BUTTONS_YES_NO_CANCEL, BUTTONS_RETRY_CANCEL, BUTTONS_ABORT_IGNORE_RETRY
from com.sun.star.awt.MessageBoxButtons import DEFAULT_BUTTON_OK, DEFAULT_BUTTON_CANCEL, DEFAULT_BUTTON_RETRY, DEFAULT_BUTTON_YES, DEFAULT_BUTTON_NO, DEFAULT_BUTTON_IGNORE
from com.sun.star.awt.MessageBoxType import MESSAGEBOX, INFOBOX, WARNINGBOX, ERRORBOX, QUERYBOX
def directmacro(*args):
import socket, time
class FontSlant():
from com.sun.star.awt.FontSlant import (NONE, ITALIC,)
#get the doc from the scripting context which is made available to all scripts
desktop = XSCRIPTCONTEXT.getDesktop()
model = desktop.getCurrentComponent()
text = model.Text
tRange = text.End
cursor = desktop.getCurrentComponent().getCurrentController().getViewCursor()
doc = XSCRIPTCONTEXT.getDocument()
parentwindow = doc.CurrentController.Frame.ContainerWindow
# your cannot insert simple text and text into a table with the same method
# so we have to know if we are in a table or not.
# oTable and oCurCell will be null if we are not in a table
oTable = cursor.TextTable
oCurCell = cursor.Cell
insert_text = "This is text inserted into a LibreOffice Document\ndirectly from a macro called externally"
Text_Italic = FontSlant.ITALIC
Text_None = FontSlant.NONE
cursor.CharPosture=Text_Italic
if oCurCell == None: # Are we inserting into a table or not?
text.insertString(cursor, insert_text, 0)
else:
cell = oTable.getCellByName(oCurCell.CellName)
cell.insertString(cursor, insert_text, False)
cursor.CharPosture=Text_None
return None
You will of course need to adapt the code to either accept data as arguments, read it from a file or whatever.
Ideally I would say use python 3, because python 2 is becoming outdated. The switch requires quite a bit of new coding changes, but better sooner than later. So I tried:
sudo pip3 install -U --pre \
-f http://wxpython.org/Phoenix/snapshot-builds/ \
wxPython_Phoenix
However this gave me errors, and I didn't want to spend the next couple of days working through them. Probably the pre-release versions are not ready for prime time yet.
So instead, what I recommend is to switch to AOO for now. See https://stackoverflow.com/a/27980255/5100564 for instructions. AOO does not have all the latest features that LO has, but it is a good solid Office product.
Apparently it is also possible to rebuild LibreOffice with python 2 using this script: https://gist.github.com/hbrunn/6f4a007a6ff7f75c0f8b
I'm having trouble passing a command with a backslash in it to django.connection.cursor (with a Postgres backend):
from django.db import connection
cursor = connection.cursor()
copy_str = "\\copy my_table FROM 'my_table.csv' DELIMITER ',' CSV";
cursor.execute(copy_str)
But this gives me:
django.db.utils.ProgrammingError: syntax error at or near "\"
Using a single slash gives me the same error.
Can I use backslashes with django.connection.cursor, or should I fall back to psycopg2?
The only reason I'm not using psycopg2 in the first place is that I am using the Django ORM elsewhere in the same script, so it would be convenient to use it here too.
Try to use the raw string with the r prefix:
copy_str = r"\\copy my_table FROM 'my_table.csv' DELIMITER ',' CSV"
But are you sure that this is valid SQL syntax?