Task: read uploaded file to check structure. My test upload file has 5 lines with header and about 20-30 columns. Encoding is ISO-8859-1
Sounds simple but it drives me slowly into insanity.
The only working solution at the moment is detour about Models:
file = request.FILES.getlist('job_file', None)[0]
newdoc = models.Jobs(job_file=file)
newdoc.save()
with codecs.open(newdoc.job_file.name, "r", encoding='iso-8859-1') as fp:
file_content = list(csv.reader(fp, delimiter=';', quotechar='"'))
Dirty, crazy and far from acceptable
Non working solutions:
1:
file_content = list(csv.reader(file, delimiter=';', quotechar='"'))
print(file_content)
>>>_csv.Error: iterator should return strings, not bytes (did you open the file in text mode?)
2:
file_content = list(csv.reader(file.open('r'), delimiter=';', quotechar='"'))
print(file_content)
>>> TypeError: argument 1 must be an iterator
3:
file_content = list(csv.reader(file.read(), delimiter=';', quotechar='"'))
print(file_content)
>>>_csv.Error: iterator should return strings, not int (did you open the file in text mode?)
Some hints:
print(file.read())
>>>b';"";""\r\n' <-- WRONG see file content at the top
print(file.readlines())
>>>[]
Please save me!
There is no need to open file, you can convert your uploaded file to TextIOWrapper. Here is cleaner example
from io import StringIO
file = request.FILES.getlist('job_file', None)[0]
newdoc = models.Jobs.objects.create(job_file=file)
fp = StringIO(file.read(), encoding='iso-8859-1')
file_content = list(csv.reader(fp, delimiter=';', quotechar='"'))
uploaded = request.FILES.getlist('job_file', None)[0]
decoded_file = uploaded_file.read().decode('ISO-8859-1').splitlines()
file_content = list(csv.reader(decoded_file, delimiter=';', quotechar='"'))
Related
I have some files in my azure-storage account. i need to download them using get_blob_to_stream.it is returning azure.storage.blob.models.Blob object. so i couldn't download it by using below code.
def download(request):
file_name=request.POST['tmtype']
fp = open(file_name, 'wb')
generator = block_blob_service.list_blobs(container_name)
for blob in generator:
print(blob.name)
if blob.name==file_name:
blob=block_blob_service.get_blob_to_stream(container_name, blob.name, fp,max_connections= 2)
response = HttpResponse(blob, content_type="image/png")
response['Content-Disposition'] = "attachment; filename="+file_name
return response
You can actually use the get_blob_to_path property, below is an example in python:
from azure.storage.blob import BlockBlobService
bb = BlockBlobService(account_name='', account_key='')
container_name = ""
blob_name_to_download = "test.txt"
file_path ="/home/Adam/Downloaded_test.txt"
bb.get_blob_to_path(container_name, blob_name_to_download, file_path, open_mode='wb', snapshot=None, start_range=None, end_range=None, validate_content=False, progress_callback=None, max_connections=2, lease_id=None, if_modified_since=None, if_unmodified_since=None, if_match=None, if_none_match=None, timeout=None)
This example with download a blob file named: "test.txt", in a container, to File_path"/home/Adam/Downloaded_test.txt" , you can also keep the same name if you'd like to as well. You can find more samples including this one in https://github.com/adamsmith0016/Azure-storage
If you want to use get_blob_to_stream. You can download with below code:
with io.open(file_path, 'wb') as file:
blob = block_blob_service.get_blob_to_stream(
container_name=container_name,
blob_name=blob_name, stream=file,
max_connections=2)
Just note that the file content will be streamed to the file rather than the returned blob object. The blob.content should be None. That is by design. See https://github.com/Azure/azure-storage-python/issues/538.
I am trying create dynamic text file and download it when my method called.I used steps at here.But when I change the file name, file isn't downloaded, displayed on browser. What can I do ? My code is below. Thanks for advice.
def view_method(request):
file_name = 'students.txt'
lines = []
data = Student.objects.all()
for d in data:
lines.append('{0};{1};{2}'.format(d.name,d.surname,d.amount))
response_content = '\n'.join(lines)
response = HttpResponse(response_content, content_type="text/plain,charset=utf8")
response['Content-Disposition'] = 'attachment; filename={0}'.format(file_name)
return response
I solved this problem with correct the file name chars.My file name contains utf-8 characters and points.
I am trying to get some xlsx files from a form, i load them using openpyxl and do some data processing.. and finally i need to download all processed xlsx files zipped to the user.
here is an example of what i did so far
if form.is_valid():
s = StringIO.StringIO()
zf = zipfile.ZipFile(s, mode="w")
for xlsx in request.FILES.getlist('xlsxs'):
element_column = "G"
element_row = 16
massar_column = "C"
massar_row_start = 18
loop = column_index_from_string(element_column)
while (loop <= ws.max_column):
for i in range(massar_row_start, ws.max_row+1):
# ...
ws["%s%s" % (element_column,i)] = 0
# ...
loop+=2
element_column = get_column_letter(loop)
buf = save_virtual_workbook(wb)
zf.write(buf) # or zf.write(wb)
zf.close()
response = HttpResponse(s.getvalue(), content_type="application/x-zip-compressed")
response['Content-Disposition'] = "attachment; filename=notes.zip"
return response
I get the error
TypeError at My_view
stat() argument 1 must be encoded string without null bytes, not str
Thanks in advance for any help you can offer.
save_virtual_workbook returns a bytestream - source.
You are passing this value to ZipFile.write which is expecting a filename.
I think you should be using ZipFile.writestr, and you need to provide a filename that will be used inside the archive. I'm not sure how you are getting the error message you see, but this is the first mistake I can see.
I am having a bit of trouble in obtaining a file path so that I can open and execute my data from the specified (text) file. Below is the code I have written so far:
def pickfile():
options={}
options['defaultextension'] = '.txt'
options['filetypes'] = [('all files','.*'), ('text files', '.*txt')]
options['initialfile'] = 'sample.txt'
options['initialdir'] = 'C:\Users\profcs\Desktop'
filename=open(tkFileDialog.askopenfilename(**options))
if filename:
print(filename)
return
with open(filename, 'rb') as f:
reader = csv.reader(f)
try:
for row in reader:
print row
except csv.Error as e:
sys.exit('file %s, line %d: %s' % (filename, reader.line_num,e))
but1 = Button(widget1, text='Pick Your File', command=pickfile)
but1.pack(side=BOTTOM, padx=10, pady=1, anchor=SE)
but1.config(relief=RAISED, bd=2)
When I display a filename, I now get the path in this form:
================ RESTART: C:\Users\profcs\Desktop\BD TEST.py ================
<open file u'C:/Users/profcs/Desktop/sample.txt', mode 'r' at 0x01EFF128>
How can I filter this path and only get 'C:/Users/profcs/Desktop/sample.txt' so that I can open my file?
Thanks in advance.
filename.name gives you the path from filename object.
I hope this helps :
filename = open(tkFileDialog.askopenfilename(**options))
print (filename.name)
'C:/Users/profcs/Desktop/sample.txt'
In your case filename is an object which represents an open file.
When I run this:
import os.path
import pyproj
srcProj = pyproj.Proj(proj='longlat', ellps='GRS80', datum='NAD83')
dstProj = pyproj.Proj(proj='longlat', ellps='WGS84', datum='WGS84')
f = file(os.path.join("DISTAL-data", "countries.txt"), "r")
heading = f.readline() # Ignore field names.
with open('C:\Python27\DISTAL-data\geonames_20160222\countries.txt', 'r') as f:
for line in f.readlines():
parts = line.rstrip().split("|")
featureName = parts[1]
featureClass = parts[2]
lat = float(parts[9])
long = float(parts[10])
if featureClass == "Populated Place":
long,lat = pyproj.transform(srcProj, dstProj, long, lat)
f.close()
I get this error:
File "C:\Python27\importing world datacountriesfromNAD83 toWGS84.py",
line 13, in for line in f.readlines() : MemoryError.
I have downloaded countries file from http://geonames.nga.mil/gns/html/namefiles.html as entire country file dataset.
Please help me to get out of this.
readlines() for large files creates a large structure in memory, you can try using:
f = open('somefilename','r')
for line in f:
dosomthing()
Answer given by Yael is helpful, I would like to improve it. A Good way to read a file or large file
with open(filename) as f:
for line in f:
print f
I like to use 'with' statement which ensure file will be properly closed.