error when pipe output to python script on windows cmd - python-2.7

I generate xml content from input file using an xml_gen.bat file (I cannot modify it), and I use it like this on Windows cmd to generate xml:
xml_gen --options [inputname] [optional outputname]
I also have a python script xml_parse.py that parses xml input:
import sys
import lxml.etree as ET
def xml_parse(xml_input):
for event, elem in ET.iterparse(xml_input, events=('end', ), tag='TAG'):
# do something
# write parsed content to file
elem.clear()
def main():
xml_parse(sys.stdin)
if __name__ == '__main__':
main()
Both xml_gen and xml_parse.py work when tested separately, i.e., xm_gen can take an input file and generate a correct xml file; xml_parse.py (ran in PyCharm) can take an input xml file from disk and do the parsing correctly.
Since my actual experiment will generate gigabytes of xml from xml_gen, I want to pipe the output of xml_gen directly to the python script to process them.
When I tried:
dir_to_xml_gen\xml_gen --option dir_to_input\input | python xml_parse.py
It produced such error:
Traceback (most recent call last): File "xml_parse.py", line 77, in
main() File "xml_parse.py", line 71, in main xml_parse(sys.stdin) File "xml_parse.py", line 50, in xml_parse for event, elem in ET.iterparse(xml_parse, events=('end', ), tag='TAG'):File "iterparse.pxi", line 208, in lxml.etree.iterparse.next (src\lxml\lxml.etree.c:131498) lxml.etree.XMLSyntaxError: Document is empty, line 2, column 1 ! System error ! 'SPIO_E_END_OF_FILE'
when I tried: dir_to_xml_gen\xml_gen --option dir_to_input\input | python xml_parse.py
it produced similar error: System error 'SPIO_E_END_OF_FILE'
I'm aware of the issues in Cannot redirect output when I run Python script on Windows using just script's name, and followed directions here STDIN/STDOUT Redirection May Not Work If Started from a File Association, but didn't solve my problem. Thank you for your help.

Related

cx_Freeze: 'The system cannot find the file specified' error during build [win10] [PyQt4] [python2.7]

I'm trying to create a .exe file from a python script (which use PyQt4 GUI and matplotlib). I'm using cx_Freeze version 5.1.1 for 64-bit windows with the following setup.py:
import cx_Freeze
import sys
import matplotlib
base = "Win32GUI"
includes = ["atexit"]
buildOptions = dict(
#create_shared_zip=False,
#append_script_to_exe=True,
includes=includes
)
executables = [cx_Freeze.Executable(script = "main.py", base = base)] # icon = "chart32.jpg")]
cx_Freeze.setup(
name= "1ChPlotGUI",
options = dict(build_exe=buildOptions), # {"build_exe": {"packages": ["matplotlib"], "include_files":["chart32.jpg"]}},
version = "0.01",
description = "1 Channel Plotting app with GUI",
executables = executables
)
after running
python setup.py build
in the cmd from the position of
C:\Users\Us.Er\Pyth-examples\Qt\UI-examples\ChannelplotGUI-to-exe
I have something like this:
running build
running build_exe
copying c:\users\Us.Er\appdata\local\enthought\canopy\user\lib\site-
packages\cx_Freeze\bases\Win32GUI.exe -> build\exe.win-amd64-2.7\main.exe
copying
c:\users\Us.Er\appdata\local\enthought\canopy\user\scripts\python27.dll ->
build\exe.win-amd64-2.7\python27.dll
Traceback (most recent call last):
File "setup.py", line 23, in <module>
executables = executables
File "c:\users\Us.Er\appdata\local\enthought\canopy\user\lib\site-packages\cx_Freeze\dist.py", line 349, in setup
distutils.core.setup(**attrs)
File "C:\Users\Us.Er\AppData\Local\Enthought\Canopy\App\appdata\canopy-1.7.4.3348.win-x86_64\lib\distutils\core.py", line 151, in setup
dist.run_commands()
File "C:\Users\Us.Er\AppData\Local\Enthought\Canopy\App\appdata\canopy-1.7.4.3348.win-x86_64\lib\distutils\dist.py", line 953, in run_commands
self.run_command(cmd)
File "C:\Users\Us.Er\AppData\Local\Enthought\Canopy\App\appdata\canopy-1.7.4.3348.win-x86_64\lib\distutils\dist.py", line 972, in run_command
cmd_obj.run()
File "C:\Users\Us.Er\AppData\Local\Enthought\Canopy\App\appdata\canopy-1.7.4.3348.win-x86_64\lib\distutils\command\build.py", line 127, in run
self.run_command(cmd_name)
File "C:\Users\Us.Er\AppData\Local\Enthought\Canopy\App\appdata\canopy-1.7.4.3348.win-x86_64\lib\distutils\cmd.py", line 326, in run_command
self.distribution.run_command(command)
File "C:\Users\Us.Er\AppData\Local\Enthought\Canopy\App\appdata\canopy-1.7.4.3348.win-x86_64\lib\distutils\dist.py", line 972, in run_command
cmd_obj.run()
File "c:\users\Us.Er\appdata\local\enthought\canopy\user\lib\site-packages\cx_Freeze\dist.py", line 219, in run
freezer.Freeze()
File "c:\users\Us.Er\appdata\local\enthought\canopy\user\lib\site-packages\cx_Freeze\freezer.py", line 626, in Freeze
self._FreezeExecutable(executable)
File "c:\users\Us.Er\appdata\local\enthought\canopy\user\lib\site-packages\cx_Freeze\freezer.py", line 232, in _FreezeExecutable
self._AddVersionResource(exe)
File "c:\users\Us.Er\appdata\local\enthought\canopy\user\lib\site-packages\cx_Freeze\freezer.py", line 172, in _AddVersionResource
stamp(fileName, versionInfo)
File "c:\users\Us.Er\appdata\local\enthought\canopy\user\lib\site-packages\win32\lib\win32verstamp.py", line 159, in stamp
h = BeginUpdateResource(pathname, 0)
pywintypes.error: (2, 'BeginUpdateResource', 'The system cannot find the file specified.')
What is the possible solution for above problem?
EDIT
To be clear: I don't want to add any icon for now. I am happy with just a simple, working .exe
I have added the target in this way:
executables = [cx_Freeze.Executable(script = "main.py", base = base, targetName="main.exe")]
I have tried to add
targetDir = "C:\Users\Us.Er\Pyth-examples\Qt\UI-examples\ChannelplotGUI-to-exe"
anyway, the return is:
TypeError: __init__() got an unexpected keyword argument 'targetDir'
The main problem is as before - the error with last lines as:
h = BeginUpdateResource(pathname, 0)
pywintypes.error: (2, 'BeginUpdateResource', 'The system cannot find the file specified.')
In my case the issue was solved by changing
'build_exe': 'build_folder'
to
'build_exe': './/build_folder'
This is because open(pathname) would work on the pathname='build_folder\\executable.exe', allowing the code to advance during the check at the begginning of the stamp method, but BeginUpdateResource(pathname, 0) would only work on './/build_folder\\executable.exe'
Solution
options = {
'build_exe': {
'build_exe': './/build'
}
}
Note: .//build here, build is the folder name where you want to build your project.
Thanks to help of user jpeg I managed to successfully freeze the script.
To eliminate the problem with path I have added a line print(pathname) in win32verstamp.py before line 159.
The print statement worked fine showing the relative path to the newly made .exe file.
Despite showing the correct path, the error was still present. I went to the stamp() definition in win32verstamp.py and found a try - except block, and have inserted print(pathname) over there.
The part is:
def stamp(pathname, options):
# For some reason, the API functions report success if the file is open
# but doesnt work! Try and open the file for writing, just to see if it is
# likely the stamp will work!
#print("Current path is " + pathname)
try:
f = open(pathname, "a+b")
f.close()
print("Possible to open" + pathname) #<---line added
except IOError, why:
print "WARNING: File %s could not be opened - %s" % (pathname, why)r code here
....
Since that the freeze was possible. (Not sure why tho, I'm happy to add some info if explanation is known)
I had the same issue tying to freeze my PyQt5 app (using python 3.7 under windows 10) with the following message :
h = BeginUpdateResource(pathname, 0)
pywintypes.error: (2, 'BeginUpdateResource', 'Le fichier spécifié est introuvable.')
So I choose to use cx_freeze Sample provided as a first step.
(This samples are availables after cx_freeze installation under MyVENVpath\Lib\site-packages\cx_Freeze\samples)
The very basic PyQt5 exemple was working, but my case was way more complex as I used several sub Python class I have developed.
I honestly don't really understand why but I was able to make it work So here is all modifications I have performed :
In setup.py identify sub python files
add my personnal packages and class like this :
include = ["PyQt5", "PySerial", "myclass1", "myclass2", "myclass3", "PandasModel"]
packages = ["PySerial", "myclass1", "myclass2", "myclass3", "PandasModel", "pandas","math"]
Use relative path in setup.py
executables = [cx_Freeze.Executable(os.getcwd() + r'\..\..\MyFileName.py', base=base)]
include_files = [(os.getcwd() + r'\..\..\CalibrationTool.ui', r'Inputs\GUI\CalibrationTool.ui')
The reason for "...." is to jump two directories higher level, as my python.exe is in sub folders VENV\Script\
Add python Files containing "Myclass1" at same level as setup.py.
Even if in my application, class files are in a subfolder Input\Packages\Myclass1File.py, I had to copy-paste it at the same level as setup.py. This way the include and package line of setup.py work and was able to generate the executable.
Last but not least,the terminal command
I have no idea why but the only command who did suceed was:
to use full path to the python.exe and full path to the setup.py with the "build" at the end.
cmd:>C:\Folder1\Folder2\MyVENV\Scripts\python.exe D:\Folder1\Folder2\setup.py build
This finaly created the executable in a build folder under C:\Folder1\Folder2\MyVENV\Scripts\

Error translating Tornado template with gettext

I've got this site running on top with Tornado and its template engine that I want to Internationalize, so I thought on using gettext to help me with that.
Since my site is already in Portuguese, my message.po (template) file has all msgid's in portuguese as well (example):
#: base.html:30 base.html:51
msgid "Início"
msgstr ""
It was generated with xgettext:
xgettext -i *.html -L Python --from-code UTF-8
Later I used Poedit to generate the translation file en_US.po and later compile it as en_US.mo.
Stored in my translation folder:
translation/en_US/LC_MESSAGES/site.mo
So far, so good.
I've created a really simple RequestHandler that would render and return the translated site.
import os
import logging
from tornado.web import RequestHandler
import tornado.locale as locale
LOG = logging.getLogger(__name__)
class SiteHandler(RequestHandler):
def initialize(self):
locale.load_gettext_translations(os.path.join(os.path.dirname(__file__), '../translations'), "site")
def get(self, page):
LOG.debug("PAGE REQUESTED: %s", page)
self.render("site/%s.html" %page)
As far as I know that should work perfectly, but somehow I've encountered some issues:
1 - How do I tell Tornado that my template has its text in Portuguese so it won't go looking for a pt locale which I don't have?
2 - When asking for the site with en_US locale, it loads ok but when Tornado is going to translate, it throws an encoding exception.
TypeError: not all arguments converted during string formatting
ERROR:views.site:Could not load template
Traceback (most recent call last):
File "/Users/ademarizu/Dev/git/new_plugin/site/src/main/py/views/site.py", line 20, in get
self.render("site/%s.html" %page)
File "/Users/ademarizu/Dev/virtualEnvs/execute/lib/python2.7/site-packages/tornado/web.py", line 664, in render
html = self.render_string(template_name, **kwargs)
File "/Users/ademarizu/Dev/virtualEnvs/execute/lib/python2.7/site-packages/tornado/web.py", line 771, in render_string
return t.generate(**namespace)
File "/Users/ademarizu/Dev/virtualEnvs/execute/lib/python2.7/site-packages/tornado/template.py", line 278, in generate
return execute()
File "site/home_html.generated.py", line 11, in _tt_execute
_tt_tmp = _("Início") # site/base.html:30
File "/Users/ademarizu/Dev/virtualEnvs/execute/lib/python2.7/site-packages/tornado/locale.py", line 446, in translate
return self.gettext(message)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/gettext.py", line 406, in ugettext
return self._fallback.ugettext(message)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/gettext.py", line 407, in ugettext
return unicode(message)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 2: ordinal not in range(128)
Any help?
Ah, I'm running python 2.7 btw!
1 - How do I tell Tornado that my template has its text in Portuguese so it won't go looking for a pt locale which I don't have?
This is what the set_default_locale method is for. Call tornado.locale.set_default_locale('pt') (or pt_BR, etc) once at startup to tell tornado that your template source is in Portuguese.
2 - When asking for the site with en_US locale, it loads ok but when Tornado is going to translate, it throws an encoding exception.
Remember that in Python 2, strings containing non-ascii characters need to be marked as unicode. Instead of _("Início"), use _(u"Início").

Web2py application: How to reference .yaml file in controller?

I have an app running online under web2py. Now, i am adding names.yml file which i need to call in my controller file (default.py) on web2py server. where should I keep the .yml/.yaml files. Currently I have kept them in views with default/names.yml but when I call it in default.py like:
dicttagger = DictionaryTagger([ 'default/names.yml', 'default/surname.yml'])
i get no such file error.
Also tried below:
dicttagger = DictionaryTagger([ 'views/default/names.yml', 'views/default/surname.yml'])
same error
class snapshot as under:
class DictionaryTagger(object):
def __init__(self, dictionary_paths):
files = [open(path, 'r') for path in dictionary_paths]
dictionaries = [yaml.load(dict_file) for dict_file in files]
map(lambda x: x.close(), files)
Any suggestions as how to do this or am I making mistake of using yaml/yml file in we2py and it doesn't work in web2py app hosted online?
question 2
thank you. it resolved an error but I am not sure how to add nltk.download() into my hosted app. I keep getting the below error. Can you pls have a look:
Traceback
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
Traceback (most recent call last):
File "/home/prakashsukhwal/web2py/gluon/restricted.py", line 220, in restricted
exec ccode in environment
File "/home/prakashsukhwal/web2py/applications/Sensiva/controllers/default.py", line 4, in
nltk.download()
File "/usr/local/lib/python2.7/dist-packages/nltk/downloader.py", line 644, in download
self._interactive_download()
File "/usr/local/lib/python2.7/dist-packages/nltk/downloader.py", line 958, in _interactive_download
DownloaderShell(self).run()
File "/usr/local/lib/python2.7/dist-packages/nltk/downloader.py", line 981, in run
user_input = raw_input('Downloader> ').strip()
EOFError: EOF when reading a line
Error snapshot help
(EOF when reading a line)
inspect attributes
Frames
File /home/prakashsukhwal/web2py/gluon/restricted.py in restricted at line 220 code arguments variables
File /home/prakashsukhwal/web2py/applications/Sensiva/controllers/default.py in at line 4 code arguments variables
File /usr/local/lib/python2.7/dist-packages/nltk/downloader.py in download at line 644 code arguments variables
File /usr/local/lib/python2.7/dist-packages/nltk/downloader.py in _interactive_download at line 958 code arguments variables
File /usr/local/lib/python2.7/dist-packages/nltk/downloader.py in run at line 981 code arguments variables
Function argument list
(self=)
Code listing
def run(self):
print 'NLTK Downloader'
while True:
self._simple_interactive_menu(
'd) Download', 'l) List', ' u) Update', 'c) Config', 'h) Help', 'q) Quit')
user_input = raw_input('Downloader> ').strip()
if not user_input: print; continue
command = user_input.lower().split()[0]
args = user_input.split()[1:]
try:
Variables
user_input undefined
builtinraw_input
).strip undefined
Context
You can store the files wherever you want, but if you're using the Python open function, you'll need to give it full paths, not paths relative to the web2py application folder. Instead, try:
import os
dicttagger = DictionaryTagger([os.path.join(request.folder, 'views',
'default', 'names.yml'),
...])

During migrating tool from windows to linux lxml error

I have developed a tool in python 2.7 that take xsd file as input ,
and give the process data into a test file
During processing the xsd file I used lxml, I am unable to resolve this sort of error.
AttributeError: 'Element' object has no attribute 'iterdescendants'
I don`t know what wrong with the lxml lib.
I want to know is there any lxml Linux compatible version for python 2.7
I have imported in the file like below:
try:
from lxml import etree
except ImportError:
import xml.etree.ElementTree as etree
I have imported only in file , and sending the element tree pointer to process the the element into another file ,
it is OK in the declared file , giving error in another file only.
the code throw the error is :
for tdocNode in lincFileRootNode:
rootNode = tdocNode.getroot()
lchildren = rootNode.getchildren()
for elt in lchildren:
if 'complex' == elt.tag:
if 'name' in elt.attrib:
if 'element' == item.tag:
if 'type' in item.attrib:
if elt.attrib['name'] == item.attrib['type']:
for key in elt.iterdescendants(tag='element'):
bIsElemTypeSimple = false
bIsElemTypeSimple = process_elementtype(key, lincFileRootNode)
where :
lincFileRootNode --> is list that containe the xsd file pointer to be processed
the error thrown is :
Traceback (most recent call last):
File "run.py", line 1210, in <module>
iret = xsd2dic_main()
File "run.py", line 71, in xsd2dic_main
iRet = yxsdtodic()
File "run.py", line 352, in yxsdtodic
iret = process_xsdfile(sXsdPath)
File "run.py", line 485, in xsdfile
sRet = process_dic_elementtype(item,lincFileRootNode)
File "run.py", line 817, in process_dic_elementtype
for key in elt.iterdescendants(tag='element'):
AttributeError: 'Element' object has no attribute 'iterdescendants'
i tired in the both the cases :
1:writing all code in a same file
2:writing different files
still i am getting the same error
This is mostly a guess, but look into it.
You appear to be calling iterdescendants from lxml's implementation of the Element type. However, if lxml fails to import, you fall back on Python's built in xml library instead. But it's implementation of Element doesn't have an iterdescendants methods of any kind. In other words, the two implementations have different public APIs. Add some print statements to see which library you're importing and do some additionally checking to see exactly what type elt is. If you want to be able to fall back on Python's built in xml, you'll need to structure your code to accommodate the different APIs.

PYPDF watermarking returns error

hi im trying to watermark a pdf fileusing pypdf2 though i get this error i cant figure out what goes wrong.
i get the following error:
Traceback (most recent call last): File "test.py", line 13, in <module>
page.mergePage(watermark.getPage(0)) File "C:\Python27\site-packages\PyPDF2\pdf.py", line 1594, in mergePage
self._mergePage(page2) File "C:\Python27\site-packages\PyPDF2\pdf.py", line 1651, in _mergePage
page2Content, rename, self.pdf) File "C:Python27\site-packages\PyPDF2\pdf.py", line 1547, in
_contentStreamRename
op = operands[i] KeyError: 0
using python 2.7.6 with pypdf2 1.19 on windows 32bit.
hopefully someone can tell me what i do wrong.
my python file:
from PyPDF2 import PdfFileWriter, PdfFileReader
output = PdfFileWriter()
input = PdfFileReader(open("test.pdf", "rb"))
watermark = PdfFileReader(open("watermark.pdf", "rb"))
# print how many pages input1 has:
print("test.pdf has %d pages." % input.getNumPages())
print("watermark.pdf has %d pages." % watermark.getNumPages())
# add page 0 from input, but first add a watermark from another PDF:
page = input.getPage(0)
page.mergePage(watermark.getPage(0))
output.addPage(page)
# finally, write "output" to document-output.pdf
outputStream = file("outputs.pdf", "wb")
output.write(outputStream)
outputStream.close()
Try writing to a StringIO object instead of a disk file. So, replace this:
outputStream = file("outputs.pdf", "wb")
output.write(outputStream)
outputStream.close()
with this:
outputStream = StringIO.StringIO()
output.write(outputStream) #write merged output to the StringIO object
outputStream.close()
If above code works, then you might be having file writing permission issues. For reference, look at the PyPDF working example in my article.
I encountered this error when attempting to use PyPDF2 to merge in a page which had been generated by reportlab, which used an inline image canvas.drawInlineImage(...), which stores the image in the object stream of the PDF. Other PDFs that use a similar technique for images might be affected in the same way -- effectively, the content stream of the PDF has a data object thrown into it where PyPDF2 doesn't expect it.
If you're able to, a solution can be to re-generate the source pdf, but to not use inline content-stream-stored images -- e.g. generate with canvas.drawImage(...) in reportlab.
Here's an issue about this on PyPDF2.