python 2.7
fr = open(r'test_audio/16k.wav', 'rb')
audio_data27 = fr.read()
python 3.6
fr = open(r'test_audio/16k.wav', 'rb')
audio_data36 = fr.read()
audio_data27 type is str,but the audio_data36 type is bytes. I want to make the type of audio_data36 also str, what should I do? Thanks!
bytes objects have a decode() function that returns a str object
Related
When I try to write cursor result coming from database execution (type is a list) to the csv, the error throws
a.writerow(lst) TypeError: write() argument 1 must be unicode, not str
This is for python2. I've tried in python3 like the script below. But the system requirement asks me to change Python2.
This is the correct script using python3.
results_percent = cursor.fetchall()
with open(file4,'w',encoding="utf-8",newline='') as fp:
a = csv.writer(fp, delimiter=',')
a.writerow(['MFIName','ClientCountAtSignUp','UploadCountLastMonth','UploadCount','80%','Status'])
a.writerows(results_percent)
The below is by using python2 which gives me error a.writerow(lst) TypeError: write() argument 1 must be unicode, not str
results_percent = cursor.fetchall()
with io.open(file4,'w',encoding='utf-8') as fp:
a = csv.writer(fp, delimiter=',')
lst = ['MFIName','ClientCountAtSignUp','UploadCountLastMonth','UploadCount','80%','Status']
a.writerow(lst)
a.writerows(results_percent)
The output is to write results_percent to csv file.
I wrote a httpserver to serve html files for python2.7 and python3.5.
def do_GET(self):
...
#if resoure is api
data = json.dumps({'message':['thanks for your answer']})
#if resource is file name
with open(resource, 'rb') as f:
data = f.read()
self.send_response(response)
self.send_header('Access-Control-Allow-Origin', '*')
self.end_headers()
self.wfile.write(data) # this line raise TypeError: a bytes-like object is required, not 'str'
the code works in python2.7, but in python 3, it raised the above the error.
I could use bytearray(data, 'utf-8') to convert str to bytes, but the html is changed in web.
My question:
How to do to support python2 and python3 without use 2to3 tools and without change the file's encoding.
is there a better way to read a file and sent it content to client with the same way in python2 and python3 ?
thanks in advance.
You just have to open your file in binary mode, not in text mode:
with open(resource,"rb") as f:
data = f.read()
then, data is a bytes object in python 3, and a str in python 2, and it works for both versions.
As a positive side-effect, when this code hits a Windows box, it still works (else binary files like images are corrupt because of the endline termination conversion when opened in text mode).
There is a Python function, that runs in CPython 2.5.3 but crashes in Jython 2.5.3 .
It is part of a user defined function in Apache Pig, which uses Jython 2.5.3 so i cannot change it.
The input is a array of singed bytes, but in fact that are unsigned bytes, so i need to cast it.
from StringIO import StringIO
import array
import ctypes
assert isinstance(input, array.array), 'unexpected input parameter'
assert input.typecode == 'b', 'unexpected input type'
buffer = StringIO()
for byte in input:
s_byte = ctypes.c_byte(byte)
s_byte_p = ctypes.pointer(s_byte)
u_byte = ctypes.cast(s_byte_p, ctypes.POINTER(ctypes.c_ubyte)).contents.value
buffer.write(chr(u_byte))
buffer.seek(0)
output = buffer.getvalue()
assert isinstance(output, str)
The error is:
s_byte = ctypes.cast(u_byte_p, ctypes.POINTER(ctypes.c_byte)).contents.value
AttributeError: 'module' object has no attribute 'cast'
I guess the ctypes.cast functions is not implemeted in Jython 2.5.3 . Is there a workaround for that issue?
Thanks,
Steffen
Here is my solution, that is quite ugly but works without additional dependecies.
It uses the bit representation of usinged und signed bytes (https://de.wikipedia.org/wiki/Zweierkomplement).
import array
assert isinstance(input, array.array), 'unexpected input parameter'
assert input.typecode == 'b', 'unexpected input type'
output = array.array('b', [])
for byte in input:
if byte > 127:
byte = byte & 127
byte = -128 + byte
output.append(byte)
with gzip.open(sys.argv[5] + ".json.gz", mode="w", encoding="utf-8") as outfile:
It throws:
TypeError: open() got an unexpected keyword argument 'encoding'
But the docs says it exists
https://docs.python.org/3/library/gzip.html
Update
How can i encode and zip the file in Python 2.7?
I tried now this:
(but it don't work)
with gzip.open(sys.argv[5] + ".json.gz", mode="w") as outfile:
outfile = io.TextIOWrapper(outfile, encoding="utf-8")
json.dump(fdata, outfile, indent=2, ensure_ascii=False)
TypeError: must be unicode, not str
What can i do?
Those are the Python 3 docs. The Python 2 version of gzip does not allow encoding= as a keyword argument to gzip.open().
Seems the question has been answered sufficiently, but for your peace of mind: Alternatively to ensure that Python2 uses utf-8 as standard perhaps try the following, as it then becomes unnecessary to specify an encoding:
import sys
reload(sys)
sys.setdefaultencoding('UTF8')
Scenario
I use a formData form to upload an image via ajax which is then added to a MongoDB GridFS database.
This was working:
my_image = request.files.my_image
raw = my_image.file.read()
fs.put(raw)
Desired Behaviour
I want to resize the image with Pillow before adding to GridFS.
What I Tried
I changed the above to:
my_image = request.files.my_image
raw = Image.open(my_image.file.read())
raw_resized = raw.resize((new_dimensions))
fs.put(raw_resized)
Actual Behaviour
I am now getting 500 errors. Tail shows:
TypeError: file() argument 1 must be encoded string without NULL bytes, not str
Question
How do I properly handle the Pillow image object so that I can add it to GridFS?
Troubelshooting
This is still unresolved, but I'm just adding my attempts to understand what is happening with file types etc at different stages of the process by using the interpretor:
>>> my_image = open("my_great_image.jpg")
>>> my_image
<open file 'my_great_image.jpg', mode 'r' at 0x0259CF40>
>>> type(my_image)
<type 'file'>
>>> my_image_read = my_image.read()
>>> my_image_read
# lots of image data
>>> type(my_image_read)
<type 'str'>
>>> my_pil_image = Image.open("my_great_image.jpg")
>>> my_pil_image
<PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=400x267 at 0x2760CD8>
>>> type(my_pil_image)
<type 'instance'>
So from this I think I can deduce that, originally, GridFS was accepting of the string version of the image generated from the read() method.
So I think I need to somehow make the Pillow image object a string in order to get it into GridFS.
Solution
Wo, check this out, it works, the logic is:
Form uploads image,
Pillow does three different resizes,
Use StringIO to convert Pillow objects to strings
Put resized images, as strings, in GridFS.
Python
from PIL import Image
import StringIO
# define three new image sizes
card_dimensions = (108,108)
main_dimensions = (42,38)
thumb_dimensions = (18,14)
# covert uploaded image to Pillow object
raw_pil = Image.open(my_image.file)
# conversion to card size
raw_card_output = StringIO.StringIO()
raw_card = raw_pil.resize((card_dimensions))
raw_card.save(raw_card_output,format=raw_pil.format)
raw_card_output_contents = raw_card_output.getvalue()
# conversion to main size
raw_main_output = StringIO.StringIO()
raw_main = raw_pil.resize((main_dimensions))
raw_main.save(raw_main_output,format=raw_pil.format)
raw_main_output_contents = raw_main_output.getvalue()
#conversion to thumb size
raw_thumb_output = StringIO.StringIO()
raw_thumb = raw_pil.resize((thumb_dimensions))
raw_thumb.save(raw_thumb_output,format=raw_pil.format)
raw_thumb_output_contents = raw_thumb_output.getvalue()
# put card image into GridFS
fs.put(raw_card_output_contents)
# put main image into GridFS
fs.put(raw_main_output_contents)
# put thumb image into GridFS
fs.put(raw_thumb_output_contents)
Further Explanation
Basically I deduced that GridFS was accepting a string, so therefore I needed to transform the Pillow object into a string.
The interpreter troubleshooting below should make some of the dynamics clearer and carries on from the troubleshooting in the original post:
>>> my_pil_image
<PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=400x267 at 0x2760CD8>
>>> my_pil_image_resized = my_pil_image.resize((50,50))
>>> my_pil_image_resized
<PIL.Image.Image image mode=RGB size=50x50 at 0x2898710>
>>> output = StringIO.StringIO()
>>> my_pil_image_resized.save(output,format="JPEG")
>>> contents = output.getvalue()
>>> type(contents)
<type 'str'>
>>> contents
# lots of image data
So basically the above process show the mechanics of how to convert the Pillow object into a string which can then be added to GridFS.
I just got a simple way to do this with flask.
I am using FileStorage for file holder before uploading to gridFS.
It also help me to wrap image from pillow after resize.
Document looks like this.
from mongoengine import Document ImageField
class SomeDocument(Document):
icon = ImageField(required=False, collection_name="collection_name")
And the controller is
from io import BytesIO
from werkzeug.datastructures import FileStorage
from PIL import Image
im = Image.open(request.files.get('icon'))
im.thumbnail((400, 400))
output = BytesIO()
im.save(output, format=im.format, quality=90)
original_extension = request.files.get('icon').filename.rsplit('.', 1)[1].lower()
SomeDocument.icon.put(FileStorage(output, content_type=f"image/{original_extension"))
SomeDocument.save()
im.close()