I tried to run the script in linux termianl using python 2.7.
I'm getting the following error:
Traceback (most recent call last):
File "main1.py", line 50, in <module>
main()
File "main1.py", line 25, in main
ver = zstd.__version__;
NameError: global name 'zstd' is not defined
please check code and suggest the changes or any other way to run the script successfully.
# import zstd library for zstandard simple api access and Compress
import csv
import os
import time
import sys
import io
import random
import zlib
#import zstandard as zstd
from zstd import *
def main():
#set path of INPUT FILE here
path = 'sai.txt'
fh_input = open(path, "rb")
#determine size of input file
sizeinfo_if = os.stat('sai.txt')
print('Size of input file is :',sizeinfo_if.st_size,'Bytes')
#make sure zstd is installed and running
ver = zstd.__version__;
print('Zstd version : ',ver)
#File for already existing(or newly created ) output file where the compressed data needs to be written
fh_output = open('output.txt','wb')
#Zstd compressor object creation
cctx = zstd.ZstdCompressor()
initial_timestamp = time.time()
#Get byte sized chucks from in File, compress and write to out file
with open('sai.txt', 'rb') as fh_input:
byte_chunk = fh_input.read(1)
while byte_chunk:
#bdata=bytes(str.encode(byte_chunk))
compressed = cctx.compress(byte_chunk)
with cctx.write_to(fh_output) as compressor:
compressor.write(byte_chunk)
byte_chunk = fh_input.read(1)
end_timestamp = time.time()
print('Time taken to compress:',end_timestamp - initial_timestamp)
sizeinfo_of = os.stat('output.txt')
print('Size of output File is:',sizeinfo_of.st_size,'Bytes')
main()
It worked for me importing in the next way (python 3.8)
import zstd
print('Zstd version : ', zstd.version())
Related
The following piece of code comes from Google's Vision API Documentation, the only modification I've made is adding the argument parser for the function at the bottom.
import argparse
import os
from google.cloud import vision
import io
def detect_text(path):
"""Detects text in the file."""
client = vision.ImageAnnotatorClient()
with io.open(path, 'rb') as image_file:
content = image_file.read()
image = vision.types.Image(content=content)
response = client.text_detection(image=image)
texts = response.text_annotations
print('Texts:')
for text in texts:
print('\n"{}"'.format(text.description))
vertices = (['({},{})'.format(vertex.x, vertex.y)
for vertex in text.bounding_poly.vertices])
print('bounds: {}'.format(','.join(vertices)))
ap = argparse.ArgumentParser()
ap.add_argument("-i", "--image", type=str,
help="path to input image")
args = vars(ap.parse_args())
detect_text(args)
If I run it from a terminal like below, I get this invalid file error:
PS C:\VisionTest> python visionTest.py --image C:\VisionTest\test.png
Traceback (most recent call last):
File "visionTest.py", line 31, in <module>
detect_text(args)
File "visionTest.py", line 10, in detect_text
with io.open(path, 'rb') as image_file:
TypeError: invalid file: {'image': 'C:\\VisionTest\\test.png'}
I've tried with various images and image types as well as running the code from different locations with no success.
Seems like either the file doesn't exist or is corrupt since it isn't even read. Can you try another image and validate it is in the location you expect?
I have to monitor an XML file being written by a tool running all the day. But the XML file is properly completed and closed only at the end of the day.
Same constraints as XML stream processing:
Parse an incomplete XML file on-the-fly and trigger actions
Keep track of the last position within the file to avoid processing it again from the beginning
On answer of Need to read XML files as a stream using BeautifulSoup in Python, slezica suggests xml.sax, xml.etree.ElementTree and cElementTree. But no success with my attempts to use xml.etree.ElementTree and cElementTree. There are also xml.dom, xml.parsers.expat and lxml but I do not see support for "on-the-fly parsing".
I need more obvious examples...
I am currently using Python 2.7 on Linux, but I will migrate to Python 3.x => please also provide tips on new Python 3.x features. I also use watchdog to detect XML file modifications => Optionally, reuse the watchdog mechanism. Optionally support also Windows.
Please provide easy to understand/maintain solutions. If it is too complex, I may just use tell()/seek() to move within the file, use stupid text search in the raw XML and finally extract the values using basic regex.
XML sample:
<dfxml xmloutputversion='1.0'>
<creator version='1.0'>
<program>TCPFLOW</program>
<version>1.4.6</version>
</creator>
<configuration>
<fileobject>
<filename>file1</filename>
<filesize>288</filesize>
<tcpflow packets='12' srcport='1111' dstport='2222' family='2' />
</fileobject>
<fileobject>
<filename>file2</filename>
<filesize>352</filesize>
<tcpflow packets='12' srcport='3333' dstport='4444' family='2' />
</fileobject>
<fileobject>
<filename>file3</filename>
<filesize>456</filesize>
...
...
First test using SAX failed:
import xml.sax
class StreamHandler(xml.sax.handler.ContentHandler):
def startElement(self, name, attrs):
print 'start: name=', name
def endElement(self, name):
print 'end: name=', name
if name == 'root':
raise StopIteration
if __name__ == '__main__':
parser = xml.sax.make_parser()
parser.setContentHandler(StreamHandler())
with open('f.xml') as f:
parser.parse(f)
Shell:
$ while read line; do echo $line; sleep 1; done <i.xml >f.xml &
...
$ ./test-using-sax.py
start: name= dfxml
start: name= creator
start: name= program
end: name= program
start: name= version
end: name= version
Traceback (most recent call last):
File "./test-using-sax.py", line 17, in <module>
parser.parse(f)
File "/usr/lib64/python2.7/xml/sax/expatreader.py", line 107, in parse
xmlreader.IncrementalParser.parse(self, source)
File "/usr/lib64/python2.7/xml/sax/xmlreader.py", line 125, in parse
self.close()
File "/usr/lib64/python2.7/xml/sax/expatreader.py", line 220, in close
self.feed("", isFinal = 1)
File "/usr/lib64/python2.7/xml/sax/expatreader.py", line 214, in feed
self._err_handler.fatalError(exc)
File "/usr/lib64/python2.7/xml/sax/handler.py", line 38, in fatalError
raise exception
xml.sax._exceptions.SAXParseException: report.xml:15:0: no element found
Since yesterday I found the Peter Gibson's answer about the undocumented xml.etree.ElementTree.XMLTreeBuilder._parser.EndElementHandler.
This example is similar to the other one but uses xml.etree.ElementTree (and watchdog).
It does not work when ElementTree is replaced by cElementTree :-/
import time
import watchdog.events
import watchdog.observers
import xml.etree.ElementTree
class XmlFileEventHandler(watchdog.events.PatternMatchingEventHandler):
def __init__(self):
watchdog.events.PatternMatchingEventHandler.__init__(self, patterns=['*.xml'])
self.xml_file = None
self.parser = xml.etree.ElementTree.XMLTreeBuilder()
def end_tag_event(tag):
node = self.parser._end(tag)
print 'tag=', tag, 'node=', node
self.parser._parser.EndElementHandler = end_tag_event
def on_modified(self, event):
if not self.xml_file:
self.xml_file = open(event.src_path)
buffer = self.xml_file.read()
if buffer:
self.parser.feed(buffer)
if __name__ == '__main__':
observer = watchdog.observers.Observer()
event_handler = XmlFileEventHandler()
observer.schedule(event_handler, path='.')
try:
observer.start()
while True:
time.sleep(10)
finally:
observer.stop()
observer.join()
While the script is running, do not forget to touch one XML file, or simulate the on-the-fly writing using this one line script:
while read line; do echo $line; sleep 1; done <in.xml >out.xml &
For information, the xml.etree.ElementTree.iterparse does not seem to support a file being written. My test code:
from __future__ import print_function, division
import xml.etree.ElementTree
if __name__ == '__main__':
context = xml.etree.ElementTree.iterparse('f.xml', events=('end',))
for action, elem in context:
print(action, elem.tag)
My output:
end program
end version
end creator
end filename
end filesize
end tcpflow
end fileobject
end filename
end filesize
end tcpflow
end fileobject
end filename
end filesize
Traceback (most recent call last):
File "./iter.py", line 9, in <module>
for action, elem in context:
File "/usr/lib64/python2.7/xml/etree/ElementTree.py", line 1281, in next
self._root = self._parser.close()
File "/usr/lib64/python2.7/xml/etree/ElementTree.py", line 1654, in close
self._raiseerror(v)
File "/usr/lib64/python2.7/xml/etree/ElementTree.py", line 1506, in _raiseerror
raise err
xml.etree.ElementTree.ParseError: no element found: line 20, column 0
Three hours after posting my question, no answer received. But I have finally implemented the simple example I was looking for.
My inspiration is from saaj's answer and is based on xml.sax and watchdog.
from __future__ import print_function, division
import time
import watchdog.events
import watchdog.observers
import xml.sax
class XmlStreamHandler(xml.sax.handler.ContentHandler):
def startElement(self, tag, attributes):
print(tag, 'attributes=', attributes.items())
self.tag = tag
def characters(self, content):
print(self.tag, 'content=', content)
class XmlFileEventHandler(watchdog.events.PatternMatchingEventHandler):
def __init__(self):
watchdog.events.PatternMatchingEventHandler.__init__(self, patterns=['*.xml'])
self.file = None
self.parser = xml.sax.make_parser()
self.parser.setContentHandler(XmlStreamHandler())
def on_modified(self, event):
if not self.file:
self.file = open(event.src_path)
self.parser.feed(self.file.read())
if __name__ == '__main__':
observer = watchdog.observers.Observer()
event_handler = XmlFileEventHandler()
observer.schedule(event_handler, path='.')
try:
observer.start()
while True:
time.sleep(10)
finally:
observer.stop()
observer.join()
While the script is running, do not forget to touch one XML file, or simulate the on-the-fly writing using the following command:
while read line; do echo $line; sleep 1; done <in.xml >out.xml &
Feel like a dunce. I'm trying to interact with a zip file and can't seem to use the zipfile library. Fairly new to python
from zipfile import *
#set filename
fpath = '{}_{}_{}.zip'.format(strDate, day, week)
#use zipfile to get info about ftp file
zip = zipfile.Zipfile(fpath, mode='r')
# doesn't matter if I use
#zip = zipfile.Zipfile(fpath, mode='w')
#or zip = zipfile.Zipfile(fpath, 'wb')
I'm getting this error
zip = zipfile.Zipfile(fpath, mode='r')
NameError: name 'zipfile' is not defined
if I just use import zipfile I get this error:
TypeError: 'module' object is not callable
Two ways to fix it:
1) use from, and in that case drop the zipfile namespace:
from zipfile import *
#set filename
fpath = '{}_{}_{}.zip'.format(strDate, day, week)
#use zipfile to get info about ftp file
zip = ZipFile(fpath, mode='r')
2) use direct import, and in that case use full path like you did:
import zipfile
#set filename
fpath = '{}_{}_{}.zip'.format(strDate, day, week)
#use zipfile to get info about ftp file
zip = zipfile.ZipFile(fpath, mode='r')
and there's a sneaky typo in your code: Zipfile should be ZipFile (capital F, so I feel slightly bad for answering...
So the lesson learnt is:
avoid from x import y because editors have a harder time to complete words
with a proper import zipfile and an editor which proposes completion, you would never have had this problem in the first place.
Easiest way to zip a file using Python:
import zipfile
zf = zipfile.ZipFile("targetZipFileName.zip",'w', compression=zipfile.ZIP_DEFLATED)
zf.write("FileTobeZipped.txt")
zf.close()
I've exhausted what I'm capable of currently regarding the script that I have posted below. I am unable to properly call gdal_calc.py with this script. I have received various error message regarding a lack of .dll files to a win32 error.
I have installed and reinstalled GDAL, however I continue to receive an error message that suggests I don't have osgeo, and I cannot install GDAL properly within my Python27 pathway because of an issue with Visual C++ for Python because I either don't have the proper .bat file OR I get this message.
fatal error C1083: Cannot open include file: 'cpl_port.h': No such file or directory
I would appreciate even a direction to be point in, because I am a novice and this is over my head.
Thanks,
import os
import sys
from osgeo import gdal
from gdalconst import *
import numpy as np
import math
import subprocess
from subprocess import call
import math
import PIL
from PIL import Image
# constants
gdalTranslate = r'C:\\Program Files (x86)\\GDAL\\gdal_translate.exe'
src = r"C:\Users\jack\Desktop\RUA_FIL\IMG-HV-ALOS2110871010-160611-HBQR1.5RUA.img"
srcVH = r"C:\Users\jack\Desktop\RUA_FIL\IMG-VH-ALOS2110871010-160611-HBQR1.5RUA.img"
dstVH = r"C:\Users\jack\Desktop\New_Trash\vhFloat32-3.img"
dst = r"C:\\Users\jack\Desktop\\New_Trash\hvFloat32-3.img"
cmd = "-ot Float32 -of HFA" # hopefully this works
#gdalCalc = r'C:\Python27\ArcGIS10.4\Scripts\dist\gdal_calc.exe'
#hvFloat32 = r"C:\Users\jack\Desktop\New_Trash\hvFloat32-1.img"
#vhFloat32 = r"C:\Users\jack\Desktop\New_Trash\hvFloat32-1.img"
#prevName = r"C:\Users\jack\Desktop\RUA_FIL\IMG-HV-ALOS2110871010-160611-HBQR1.5RUA"
#newName = r"C:\Users\jack\Desktop\RUA_FIL\IMG-HV-ALOS2110871010-160611-HBQR1.5RUA.img"
#os.rename(prevName,newName)
# setting def
#def gdal_Translate (item):
#return item
hvfullCmd = ' '.join([gdalTranslate, cmd, gdal_Translate(src), gdal_Translate(dst)])
subprocess.Popen(hvfullCmd)
vhfullCmd = ' '.join([gdalTranslate,cmd,gdal_Translate(srcVH),gdal_Translate(dstVH)])
subprocess.Popen(vhfullCmd)
# Run gdal_calc.py for HV
try:
os.system('C:\Users\jack\Python\QGIS Python Scripts\ridofZs.py')
except:
print ("This isn't working")
sys.exit()
subprocess.call([sys.executable,'C:\Program Files (x86)\GDAL\gdal_calc.py', '-A', 'C:\Users\jack\Desktop\New_Trash\hvFloat32.img', '--outfile=C:\Users\jack\Desktop\New_Trash\NEWCALCHV-2.img','--calc=A*(A>=1)'])
# Run gdal_calc.py for VH
#subprocess.call([sys.executable, 'C:\OSGeo4W64\bin\gdal_calc.py', '-A', 'C:\Users\jack\Desktop\New_Trash\vhFloat32.img', '--outfile=C:\Users\jack\Desktop\New_Trash\NEWCALCVH-2.img','--calc=A*(A>=1)'])
# Run gdal_calc.py to dB for HV
#subprocess.call([sys.executable, 'C:\OSGeo4W64\bin\gdal_calc.py', '-A', 'C:\Users\jack\Desktop\New_Trash\NEWCALCHV-2.img', '--outfile=C:\Users\jack\Desktop\New_Trash\HVindB.img','--calc=10*log10(power(A,2))-83'])
# Run gdal_calc.py to dB for HV
#subprocess.call([sys.executable, 'C:\OSGeo4W64\bin\gdal_calc.py', '-A', 'C:\Users\jack\Desktop\New_Trash\NEWCALCVH-2.img', '--outfile=C:\Users\jack\Desktop\New_Trash\VHindB.img','--calc=10*log10(power(A,2))-83'])
# Open Rasters using GDAL
hvRaster = gdal.Open("C:\\Users\\jack\\Desktop\\New_Trash\\hvFloat32.img")
vhRaster = gdal.Open("C:\\Users\\jack\\Desktop\\New_Trash\\vhFloat32.img")
# Get RasterBand
hvRasterBand = hvRaster.GetRasterBand(1)
vhRasterBand = vhRaster.GetRasterBand(1)
# Get Raster Mean
hvMean = hvRasterBand.GetStatistics(0,1)[2]
vhMean = vhRasterBand.GetStatistics(0,1)[2]
#### Maybe not needed
print hvMean
print vhMean
# calculate stDev
rasterList = (hvMean,vhMean)
stDev = np.std(rasterList)
#print stDev
if stDev >= 0.06:
print "The imagery isn't acceptable"
if stDev <= 0.06:
print "The imagery is acceptable"
I'm trying to have a script that will automatically download files and install them in the correct directories. However When i try to delete the downloaded file after everything has been extracted correctly, I get an OS error that its being used by another process. I use with so the file should close correctly, but even if i put manual closes for source and zip_file it still gives this error. Does anyone know why?
It is most definitely being used by the python interpreter and not any other program, i tested this using "Unlocker". So my only conclusion is the file isn't closing after i open it.
# Downloads, updates, and installs the most recent version of flippy. Only tested on Windows operating systems.
import os
import shutil
import zipfile
import urllib
from os.path import expanduser
import maya.cmds as mc
import maya.mel as mel
download_dir = expanduser("~")
scripts_path = mc.internalVar(usd=True)
prefs_path = mc.internalVar(upd=True)
urllib.urlretrieve(r"https://drive.google.com/uc?export=download&id=0BwF-kFX824PuXzZVQmJja3JzNkE", r"%s/flippy.zip" %(download_dir))
with zipfile.ZipFile("%s/flippy.zip" %(download_dir)) as zip_file:
for member in zip_file.namelist():
with zip_file.open(member) as source:
filename = os.path.basename(member)
if filename == 'flippy.py' or filename == 'flippy_invert_attrs.txt':
target = file(os.path.join(scripts_path, filename), "wb")
copy = True
elif filename == 'flippy_icon.png':
target = file(os.path.join("%s/icons" %(prefs_path), filename), "wb")
copy = True
elif filename == 'shelf_NeoTools.mel':
target = file(os.path.join("%s/shelves" %(prefs_path), filename), "wb")
copy = True
elif filename == 'CHANGELOG.txt':
copy = False
for line in source:
print line
else:
copy = False
# copy file (taken from zipfile's extract)
if copy:
with source, target:
shutil.copyfileobj(source, target)
os.remove("%s/flippy.zip" %(download_dir))
This code gives the following error:
# Error: 32
# Traceback (most recent call last):
# File "<maya console>", line 40, in <module>
# WindowsError: [Error 32] The process cannot access the file because it is being used by another process: 'F:/My Documents/flippy.zip' #