Paramiko error: size mismatch in put - python-2.7

I am trying to copy few files from my local windows directory to remote linux dir.
It is working for file having same kind of extension. But breaks when there are different extensions in a folder.
The Code:
import os
import glob
import paramiko
glob_pattern='*.*'
try:
ssh.connect(host,username=user,password=pwd)
ftp = ssh.open_sftp()
try:
ftp.mkdir(dir_remote)
command=dir_remote+'/setuplog'
ftp.mkdir(command)
commande=dir_remote+'/emsfolder'
ftp.mkdir(commande)
try:
for fname in glob.glob(uploadfolder + os.sep + glob_pattern):
local_file = os.path.join(uploadfolder, fname)
remote_file = dir_remote + '/' + os.path.basename(local_file)
ftp.put(local_file,remote_file)
ftp.chmod(remote_file ,0777)
except IOError, e:
print (e)
except IOError, e:
print (e)
except paramiko.AuthenticationException, ae:
print (ae)
finally:
ssh.close()
I was trying to transfer 2 files only(1.sh and 2.pl). While 1.sh got copied a 0 byte 2.pl file is created at the remote server and then I get The Error:
size mismatch in put! 0 != 2200
I am using:
python 2.7, Paramiko - 1.15.2
Kindly help.

I doubt this has anything to do with different extensions in a folder. The code in paramiko's sftp_client.py:putfo() reads at the end:
s = self.stat(remotepath)
if s.st_size != size:
raise IOError('size mismatch in put! %d != %d' % (s.st_size, size))
I had a similar issue and it turned out that the remote filesystem was full and thus paramiko couldn't write/put the file.
BTW, instead of uploadfolder + os.sep + glob_pattern (and similar) you can use os.path.join(uploadfolder, glob_pattern)

Related

How to download a snappy.parquet file from s3 using Boto in Python

I'm new to this, and trying to download a snappy.parquet file from Amazon s3 I can later convert to CSV file.
I tried working with the following example I've found online, and I get an empty folder. can anyone please help me?
import boto
import sys, os
from boto.s3.key import Key
from boto.exception import S3ResponseError
DOWNLOAD_LOCATION_PATH =""
BUCKET_NAME = ""
AWS_ACCESS_KEY_ID= ""
AWS_ACCESS_SECRET_KEY = ""
conn = boto.connect_s3(AWS_ACCESS_KEY_ID, AWS_ACCESS_SECRET_KEY)
bucket = conn.get_bucket(BUCKET_NAME)
#goto through the list of files
bucket_list = bucket.list()
for l in bucket_list:
key_string = str(l.key)
s3_path = DOWNLOAD_LOCATION_PATH + key_string
try:
print ("Current File is ", s3_path)
l.get_contents_to_filename(s3_path)
except (OSError, S3ResponseError) as e:
pass
# check if the file has been downloaded locally
if not os.path.exists(s3_path):
try:
os.makedirs(s3_path)
except OSError as exc:
# let guard againts race conditions
import errno
if exc.errno != errno.EEXIST:
raise
The script you are using appears to recursively download the contents of the specified S3 bucket (BUCKET_NAME) to the specified local directory (DOWNLOAD_LOCATION_PATH). FWIW, I notice this script looks like it comes from here.
The "Current File is ..." output line should show you the progress of these files being written. One problem you might be having is due to this line:
s3_path = DOWNLOAD_LOCATION_PATH + key_string
If you had specified DOWNLOAD_LOCATION_PATH at the top as a directory without a trailing '/' character, e.g. like this:
DOWNLOAD_LOCATION_PATH = '/tmp/my_dir'
then the files being downloaded would be written not underneath the /tmp/my_dir directory, but directly in /tmp/ with a my_dir prefix on each filename! You can fix this by changing this line to:
s3_path = os.path.join(DOWNLOAD_LOCATION_PATH, key_string)
Other than that, the script appears to work alright. You may want to add this line at the very top:
from __future__ import print_function
if you are still using Python 2.x, otherwise the print output will look a bit odd (print will think you are printing a 2-Tuple).
Your question also makes it sound like you really only want/need to download a single file from the bucket -- if so, this isn't really a great script to be using, since it's downloading everything.

Tensorboard Error - NameError: name 'tensorboard' is not defined

I am now learning tensorflow but am unable to get tensorboard to work. I tried the simple program below with no luck. The program works before I use the tensorboard code but when I use the tensorboard code I get the following error:
NameError: name 'tensorboard' is not defined
Please any assistance is apppreciated.
import tensorflow as tf
a = tf.constant(5, name="input_a")
b = tf.constant(3, name="input_a")
c = tf.multiply(a,b, name="mul_c")
d = tf.add(a,b, name="add_d")
e = tf.add(c,d, name="add_e")
sess = tf.Session()
sess.run(e)
output = sess.run(e)
writer = tf.summary.FileWriter('/tmp/newtest', graph=sess.graph)
print(sess.run(e))
tensorboard --logdir /tmp/newtest
I believe this is already 'answered', but, to give a sample of what I did, regarding this, and I hope it helps you or others.
This is just covering ending overhead of triggering & showing tensorboard.
import subprocess
import webbrowser
import time
logLocation = 'tflearn_logs'
print("\r\nWould you like to see the visual results (y/N)? ", end='', flush=True)
answer = input()
if answer.strip().lower() == "y":
port = str(8018)
print("Starting Tensorboard to visualize... ")
process = subprocess.Popen(['tensorboard', "--logdir='" + logLocation + "'", '--port=' + port])
# Wait for a few seconds, give the tensorboard a headstart
time.sleep(5)
print("Opening Tensorboard webpage... ")
url = 'http://127.0.0.1:' + port + '/'
# Path differs per OS (Windows, Linux, iOS)
chrome_path = 'C:/Program Files (x86)/Google/Chrome/Application/chrome.exe %s'
webbrowser.get(chrome_path).open(url)
print("Press enter to quit... ", end='', flush=True)
answer = input()
if process is not None:
process.kill()

Can't close file, being used by another process, windows error 32?

I'm trying to have a script that will automatically download files and install them in the correct directories. However When i try to delete the downloaded file after everything has been extracted correctly, I get an OS error that its being used by another process. I use with so the file should close correctly, but even if i put manual closes for source and zip_file it still gives this error. Does anyone know why?
It is most definitely being used by the python interpreter and not any other program, i tested this using "Unlocker". So my only conclusion is the file isn't closing after i open it.
# Downloads, updates, and installs the most recent version of flippy. Only tested on Windows operating systems.
import os
import shutil
import zipfile
import urllib
from os.path import expanduser
import maya.cmds as mc
import maya.mel as mel
download_dir = expanduser("~")
scripts_path = mc.internalVar(usd=True)
prefs_path = mc.internalVar(upd=True)
urllib.urlretrieve(r"https://drive.google.com/uc?export=download&id=0BwF-kFX824PuXzZVQmJja3JzNkE", r"%s/flippy.zip" %(download_dir))
with zipfile.ZipFile("%s/flippy.zip" %(download_dir)) as zip_file:
for member in zip_file.namelist():
with zip_file.open(member) as source:
filename = os.path.basename(member)
if filename == 'flippy.py' or filename == 'flippy_invert_attrs.txt':
target = file(os.path.join(scripts_path, filename), "wb")
copy = True
elif filename == 'flippy_icon.png':
target = file(os.path.join("%s/icons" %(prefs_path), filename), "wb")
copy = True
elif filename == 'shelf_NeoTools.mel':
target = file(os.path.join("%s/shelves" %(prefs_path), filename), "wb")
copy = True
elif filename == 'CHANGELOG.txt':
copy = False
for line in source:
print line
else:
copy = False
# copy file (taken from zipfile's extract)
if copy:
with source, target:
shutil.copyfileobj(source, target)
os.remove("%s/flippy.zip" %(download_dir))
This code gives the following error:
# Error: 32
# Traceback (most recent call last):
# File "<maya console>", line 40, in <module>
# WindowsError: [Error 32] The process cannot access the file because it is being used by another process: 'F:/My Documents/flippy.zip' #

Determining wait time while executing a binary on QNX prompt using telnet

I am processing certain output binary files using sloginfo on QNX, I have used ftp/telnet/vmware to get to a point where I upload the binary from my machine to the vmware instance and then run the sloginfo command.
The issue is that binary files which need to be processed are of inconsistent size (ranging from 50mb to 200mb), and the time needed to process each of these files is different, thus making it impossible to determine the wait/sleep time required.
I need to know if sloginfo returns a value which can be used as a flag. I tried using tn.read_until(), without getting desired results.
#
import os, sys, telnetlib, time
from ftplib import FTP
def upload(ftp, filed):
ext = os.path.splitext(filed)[1]
if ext in (".txt", ".htm", ".html"):
ftp.storlines("STOR " + filed, open(filed))
else:
ftp.storbinary("STOR " + filed, open(filed, "rb"), 1024)
def gettext(ftp, filename, outfile=None):
# fetch a text file
if outfile is None:
outfile = sys.stdout
# use a lambda to add newlines to the lines read from the server
ftp.retrlines("RETR " + filename, lambda s, w=outfile.write: w(s+"\n"))
if __name__ == '__main__':
dbfile = "LOG1"
nonpassive = False
remotesite = '192.168.0.128'
ftp_port = '21'
tel_port = '23'
password = 'root'
ftp = FTP()
ftp.connect(remotesite, ftp_port)
ftp.login('root','root')
print 'Uploading the Log file... Please wait...'
upload (ftp, dbfile)
print 'File Uploaded Successfully...'
tn = telnetlib.Telnet(remotesite, tel_port)
tn.read_until("login: ")
tn.write('root' + "\n")
if password:
tn.write(password + "\n")
tn.write("sloginfo LOG1 >> LOG1.txt\n")
**#need to get more control on this sleep time**
time.sleep(300)
print 'Downloading text file...'
gettext(ftp, "LOG1.txt", open(r'LOG1.txt','wb'))
ftp.close()
tn.close()
tn.write("sloginfo LOG1 >> LOG1.txt\n") modified the above comment with tn.write ('sloginfo '+ strdbfile + '>> ' + strdbfiletxt+ '; echo Done!\n') and this has resolved the issue

Program for Backup - Python

Im trying to execute the following code in Python 2.7 on Windows7. The purpose of the code is to take back up from the specified folder to a specified folder as per the naming pattern given.
However, Im not able to get it work. The output has always been 'Backup Failed'.
Please advise on how I get resolve this to get the code working.
Thanks.
Code :
backup_ver1.py
import os
import time
import sys
sys.path.append('C:\Python27\GnuWin32\bin')
source = 'C:\New'
target_dir = 'E:\Backup'
target = target_dir + os.sep + time.strftime('%Y%m%d%H%M%S') + '.zip'
zip_command = "zip -qr {0} {1}".format(target,''.join(source))
print('This is a program for backing up files')
print(zip_command)
if os.system(zip_command)==0:
print('Successful backup to', target)
else:
print('Backup FAILED')
See if escaping the \'s helps :-
source = 'C:\\New'
target_dir = 'E:\\Backup'