I've been struggling with this for a few hours. I want to send a text file generated by Django to another server. For that I use scp and subprocess.call(). Everything goes well and I got a return_code == 0, but scp sends 0 bytes. The file created on the server side is empty.
I printed the exact command executed, the path is right, and if I put in in a shell it works perfectly.
Here is the code:
form = SubmitForm(request.POST or None)
context['form'] = form
if request.method == 'POST':
if form.is_valid():
# write file in ~/hipercic/apps/dcor/jobs/
params_file = open('apps/dcor/jobs/job_' + datetime.today().strftime("%Y%m%d_%H%M%S") + '_params.txt', 'wb')
for key, val in form.cleaned_data.iteritems():
params_file.write(str(val) + ' \n')
params_file.close
cmd = 'scp /home/guillet/hipercic/' + params_file.name + ' guillet#helios.public.stolaf.edu:'
context['cmd'] = cmd
return_code = subprocess.call(cmd, shell=True)
context['return_code'] = return_code
return render(request, 'base_pending.html', context)
I thought about a possible race condition, the file not having time to be completely written before being send, but nothing changes with a time.sleep(3).
Also, something really weird and the heart of the issue, if I tried to reopen and read the file right after closing it, the file is empty:
with open('/home/guillet/hipercic/' + params_file.name, 'rb') as f:
print f.read() # prints nothing!!
you have done params_file.close instead of params_file.close()
Closing the file properly will flush the data to the file you want to write to
It is good practice to use the with keyword when dealing with file objects. This has the advantage that the file is properly closed after its suite finishes.
Related
I want to download multiple specific links(images´ urls) into a txt file(or any file where all links can be listed underneath each others).
I get them but the code wrtite each link on the top of the other one and at the end it stays only a link :(. Also I want not repeated urls
def dlink(self, image_url):
r = self.session.get(image_url, stream=True)
with open('Output.txt','w') as f:
f.write(image_url + '\n')
The issue is most simply that opening a file with mode 'w' truncates any existing file. You should change 'w' to 'a' instead. This will open an existing file for writing, but append instead of truncating.
More fundamentally, the problem may be that you are opening the file over and over in a loop. This is very inefficient. The only time the approach you use could be really useful is if your program is approaching the OS-imposed limit on number of open files. If this is not the case, I would recommended putting the loop inside the with block, keeping the mode as 'w' since you open the file just once now, and passing the open file to your dlink function.
Edit
Huge mistake of my part, as it is a method, and you will call it several times, if you open it in write mode ('w') or similar, it will Overwrites the existing file if the file exists.
So, if you use the 'a' way, you can see that:
Opens a file for appending. The file pointer is at the end of the file
if the file exists. That is, the file is in the append mode. If the
file does not exist, it creates a new file for writing.
The other problem radics in image_url is a list, so you need to write it line by line:
def dlink(self, image_url):
r = self.session.get(image_url, stream=True)
with open('Output.txt','a') as f:
for url in list(set(image_url)):
f.write(image_url + '\n')
another way to do it:
your_file = open('Output.txt', 'a')
r = self.session.get(image_url, stream=True)
for url in list(set(image_url)):
your_file.write("%s\n" % url)
your_file.close() #dont forget close it :)
the file open mode is wrong，'w' mode make this file was overwritten every time you open it,not appended to it. replace it to 'a' mode.
you can see this https://stackoverflow.com/a/23566951/8178794 for more detail
Open a file with option w overwrite the file if existring, use the mode a to append data to an existing file.
Try :
import requests
from os.path import splitext
# use mode='a' to append result without erasing filename
def dlink(url, filename, mode='w'):
r = requests.get(url)
if r.status_code != 200:
return
# here the link is valid
with open(filename, mode) as desc:
desc.write(url)
def dimg(img_url, img_name):
r = requests.get(img_url, stream=True)
if r.status_code != 200:
return
_, ext = splitext(img_url)
with open(img_name + ext, 'wb') as desc:
for chunk in r:
desc.write(chunk)
dlink('https://image.flaticon.com/teams/slug/freepik.jpg', 'links.txt')
dlink('https://image.flaticon.com/teams/slug/freepik.jpg', 'links.txt', 'a')
dimg('https://image.flaticon.com/teams/slug/freepik.jpg', 'freepik')
I have 5 files in a folder App:
App|
|--A.txt
|--B.txt
|--C.txt
|--D.txt
|--E.txt
|--Run.py
|--Other Folders or Files
Now I want to know if files (A.txt,B.txtC.txt,C.txt,D.txt,E.txt) is present or not and if its there than I want to call a function Cleaner which will supply names of these files to that function. I have written this code but nothing is happening.The function is not getting called.
import glob
import csv
import itertools
files = glob.glob("*.txt")
i = 0
def sublist(a, b):
seq = iter(b)
try:
for x in a:
while next(seq) != x: pass
else:
return True
except StopIteration:
pass
return False
required_files = ['Alternate_ADR6_LFB1.txt', 'Company_Code.txt', 'Left_LIFNR.txt', 'LFA1.txt', 'LFB1.TXT', 'LFBK.TXT']
if sublist(required_files,files):
for files in required_files:
try:
f = open(files , 'r')
f.close()
except IOError as e:
print 'Error opening or accessing files'
i = 1
else:
print 'Required files are not in correct folder'
if i == 1:
for files in required_files:
Cleansing(files)
def Cleansing(filename):
with open('filename', 'rb') as f_input:
...
...
break
with open('filename', 'rb') as f_input, open('filename_Cleaned.csv', 'wb') as f_output:
csv_output = csv.writer(f_output)
csv_output.writerow('something')
Upadate
I think now I am able to call the function and also able to check the valid files but its not that pythonic. And I am not able to open or create a file with the name of the file plus _cleaned :filename_cleaned.csv.
You want to check if a list of files (required_files) are in a folder.
You successfully get the complete list of text files in the folder with files = glob.glob("*.txt")
So the first question is: Checking for sublist in list
As the order is not important, we can use sets:
if set(required_files) <= set(files):
# do stuff
else:
#print warning
Next question: How to open the files and create an outputs with names like "filename_Cleaned.csv"
A very important thing you have to understand: "filename" is not the same thing as filename. The first is a string, it will always be the same thing, it will not be replaced by real filenames. When writing open('filename', 'rb') you're trying to open a file called "filename".
filename however can be a variable name and take different values.
for filename in required_files:
Cleansing(filename)
def Cleansing(filename):
with open(filename, 'rb') as f_input, open(filename+'_Cleaned.csv', 'wb') as f_output:
#read stuff in f_input
#write stuff in f_output
I have written following code I am able to print out the parsed values of Lat and lon but i am unable to write them to a file. I tried flush and also i tried closing the file but of no use. Can somebody point out whats wrong here.
import os
import serial
def get_present_gps():
ser=serial.Serial('/dev/ttyUSB0',4800)
ser.open()
# open a file to write gps data
f = open('/home/iiith/Desktop/gps1.txt', 'w')
data=ser.read(1024) # read 1024 bytes
f.write(data) #write data into file
f = open('/home/iiith/Desktop/gps1.txt', 'r')# fetch the required file
f1 = open('/home/iiith/Desktop/gps2.txt', 'a+')
for line in f.read().split('\n'):
if line.startswith('$GPGGA'):
try:
lat, _, lon= line.split(',')[2:5]
lat=float(lat)
lon=float(lon)
print lat/100
print lon/100
a=[lat,lon]
f1.write(lat+",")
f1.flush()
f1.write(lon+"\n")
f1.flush()
f1.close()
except:
pass
while True:
get_present_gps()
You're covering the error up by using the except: pass. Don't do that... ever. At least log the exception.
One error which it definitely covers is lat+",", which is going to fail because it's float+str and it's not implemented. But there may be more.
I am processing certain output binary files using sloginfo on QNX, I have used ftp/telnet/vmware to get to a point where I upload the binary from my machine to the vmware instance and then run the sloginfo command.
The issue is that binary files which need to be processed are of inconsistent size (ranging from 50mb to 200mb), and the time needed to process each of these files is different, thus making it impossible to determine the wait/sleep time required.
I need to know if sloginfo returns a value which can be used as a flag. I tried using tn.read_until(), without getting desired results.
#
import os, sys, telnetlib, time
from ftplib import FTP
def upload(ftp, filed):
ext = os.path.splitext(filed)[1]
if ext in (".txt", ".htm", ".html"):
ftp.storlines("STOR " + filed, open(filed))
else:
ftp.storbinary("STOR " + filed, open(filed, "rb"), 1024)
def gettext(ftp, filename, outfile=None):
# fetch a text file
if outfile is None:
outfile = sys.stdout
# use a lambda to add newlines to the lines read from the server
ftp.retrlines("RETR " + filename, lambda s, w=outfile.write: w(s+"\n"))
if __name__ == '__main__':
dbfile = "LOG1"
nonpassive = False
remotesite = '192.168.0.128'
ftp_port = '21'
tel_port = '23'
password = 'root'
ftp = FTP()
ftp.connect(remotesite, ftp_port)
ftp.login('root','root')
print 'Uploading the Log file... Please wait...'
upload (ftp, dbfile)
print 'File Uploaded Successfully...'
tn = telnetlib.Telnet(remotesite, tel_port)
tn.read_until("login: ")
tn.write('root' + "\n")
if password:
tn.write(password + "\n")
tn.write("sloginfo LOG1 >> LOG1.txt\n")
**#need to get more control on this sleep time**
time.sleep(300)
print 'Downloading text file...'
gettext(ftp, "LOG1.txt", open(r'LOG1.txt','wb'))
ftp.close()
tn.close()
tn.write("sloginfo LOG1 >> LOG1.txt\n") modified the above comment with tn.write ('sloginfo '+ strdbfile + '>> ' + strdbfiletxt+ '; echo Done!\n') and this has resolved the issue
I want to return some files in a HttpResponse and I'm using the following function. The file that is returned always has a filesize of 1kb and I do not know why. I can open the file, but it seems that it is not served correctly. Thus I wanted to know how one can return files with django/python over a HttpResponse.
#login_required
def serve_upload_files(request, file_url):
import os.path
import mimetypes
mimetypes.init()
try:
file_path = settings.UPLOAD_LOCATION + '/' + file_url
fsock = open(file_path,"r")
#file = fsock.read()
#fsock = open(file_path,"r").read()
file_name = os.path.basename(file_path)
file_size = os.path.getsize(file_path)
print "file size is: " + str(file_size)
mime_type_guess = mimetypes.guess_type(file_name)
if mime_type_guess is not None:
response = HttpResponse(fsock, mimetype=mime_type_guess[0])
response['Content-Disposition'] = 'attachment; filename=' + file_name
except IOError:
response = HttpResponseNotFound()
return response
Edit:
The bug is actually not a bug ;-)
This solution is working in production on an apache server, thus the source is ok.
While writing this question I tested it local with the django development server and was wondering why it does not work. A friend of mine told me that this issue could arise if the mime types are not set in the server. But he was not sure if this is the problem. But one thing for sure.. it has something to do with the server.
Could it be that the file contains some non-ascii characters that render ok in production but not in development?
Try reading the file as binary:
fsock = open(file_path,"rb")
Try passing the fsock iterator as a parameter to HttpResponse(), rather than to its write() method which I think expects a string.
response = HttpResponse(fsock, mimetype=...)
See http://docs.djangoproject.com/en/dev/ref/request-response/#passing-iterators
Also, I'm not sure you want to call close on your file before returning response. Having played around with this in the shell (I've not tried this in an actual Django view), it seems that the response doesn't access the file until the response itself is read. Trying to read a HttpResponse created using a file that is now closed results in a ValueError: I/O operation on closed file.
So, you might want to leave fsock open, and let the garbage collector deal with it after the response is read.
Try disabling "django.middleware.gzip.GZipMiddleware" from your MIDDLEWARE_CLASSES in settings.py
I had the same problem, and after I looked around the middleware folder, this middleware seemed guilty to me and removing it did the trick for me.