How does SSH Window Size affect in paramiko - python-2.7

I am building an application where it SSHs to Cisco devices to collect output of show commands. I am using paramiko module in python to get this done.
While comparing the command output with that of plink, got to know that the output from paramiko is truncated. Tried with unbuffering and increased buffer size and it didn't help. Later, just tried with window_size parameter and it seems to work.
Below is my code:
import paramiko
sshclient = None
try:
sshclient = paramiko.SSHClient()
sshclient.set_missing_host_key_policy(paramiko.AutoAddPolicy())
sshclient.connect('mydevice', username='admin', password='admin12345')
chan = sshclient.get_transport().open_session(window_size=500000)
chan.settimeout(10800)
chan.exec_command('show tech-support fcip')
value = chan.recv(1024)
while value:
print(value)
value = chan.recv(1024)
finally:
if sshclient:
sshclient.close()
As per paramiko document for Transport, default_window_size=2097152; 1597152 lesser than the default_value.
Also the partial log output with default_window_size=2097152 is:
Authentication (password) successful!
[chan 0] Max packet in: 32768 bytes
[chan 0] Max packet out: 32768 bytes
Secsh channel 0 opened.
[chan 0] Sesch channel 0 request ok
EOF in transport thread
Where as with window_size=500000 is:
Authentication (password) successful!
[chan 0] Max packet in: 32768 bytes
[chan 0] Max packet out: 32768 bytes
Secsh channel 0 opened.
[chan 0] Sesch channel 0 request ok
[chan 0] EOF received (0)
EOF in transport thread
Here it looks like, when the window_size is default value, the channel is being closed even before the termination signal from the server.
Experts, please advise me the consequences of reduced window_size in SSH and how will it affect my application?

By modifying the window_size, you effect the maximum length a line can be before a return character is entered. If your application requires you to perform some parsing, and you want to call something like
value.split('\n')
You may find that the strings are being split more/less frequently than you predicted.

Related

How do I set the column width of a pexpect ssh session?

I am writing a simple python script to connect to a SAN via SSH, run a set of commands. Ultimately each command will be logged to a separate log along with a timestamp, and then exit. This is because the device we are connecting to doesn't support certificate ssh connections, and doesn't have decent logging capabilities on its current firmware revision.
The issue that I seem to be running into is that the SSH session that is created seems to be limited to 78 characters wide. The results generated from each command are significantly wider - 155 characters. This is causing a bunch of funkiness.
First, the results in their current state are significantly more difficult to parse. Second, because the buffer is significantly smaller, the final volume command won't execute properly because the pexpect launched SSH session actually gets prompted to "press any key to continue".
How do I change the column width of the pexpect session?
Here is the current code (it works but is incomplete):
#!/usr/bin/python
import pexpect
import os
PASS='mypassword'
HOST='1.2.3.4'
LOGIN_COMMAND='ssh manage#'+HOST
CTL_COMMAND='show controller-statistics'
VDISK_COMMAND='show vdisk-statistics'
VOL_COMMAND='show volume-statistics'
VDISK_LOG='vdisk.log'
VOLUME_LOG='volume.log'
CONTROLLER_LOG='volume.log'
DATE=os.system('date +%Y%m%d%H%M%S')
child=pexpect.spawn(LOGIN_COMMAND)
child.setecho(True)
child.logfile = open('FetchSan.log','w+')
child.expect('Password: ')
child.sendline(PASS)
child.expect('# ')
child.sendline(CTL_COMMAND)
print child.before
child.expect('# ')
child.sendline(VDISK_COMMAND)
print child.before
child.expect('# ')
print "Sending "+VOL_COMMAND
child.sendline(VOL_COMMAND)
print child.before
child.expect('# ')
child.sendline('exit')
child.expect(pexpect.EOF)
print child.before
The output expected:
# show controller-statistics
Durable ID CPU Load Power On Time (Secs) Bytes per second IOPS Number of Reads Number of Writes Data Read Data Written
---------------------------------------------------------------------------------------------------------------------------------------------------------
controller_A 0 45963169 1573.3KB 67 386769785 514179976 6687.8GB 5750.6GB
controller_B 20 45963088 4627.4KB 421 3208370173 587661282 63.9TB 5211.2GB
---------------------------------------------------------------------------------------------------------------------------------------------------------
Success: Command completed successfully.
# show vdisk-statistics
Name Serial Number Bytes per second IOPS Number of Reads Number of Writes Data Read Data Written
------------------------------------------------------------------------------------------------------------------------------------------------
CRS 00c0ff13349e000006d5c44f00000000 0B 0 45861 26756 3233.0MB 106.2MB
DATA 00c0ff1311f300006dd7c44f00000000 2282.4KB 164 23229435 76509765 5506.7GB 1605.3GB
DATA1 00c0ff1311f3000087d8c44f00000000 2286.5KB 167 23490851 78314374 5519.0GB 1603.8GB
DATA2 00c0ff1311f30000c2f8ce5700000000 0B 0 26 4 1446.9KB 65.5KB
FRA 00c0ff13349e000001d8c44f00000000 654.8KB 5 3049980 15317236 1187.3GB 1942.1GB
FRA1 00c0ff13349e000007d9c44f00000000 778.7KB 6 3016569 15234734 1179.3GB 1940.4GB
------------------------------------------------------------------------------------------------------------------------------------------------
Success: Command completed successfully.
# show volume-statistics
Name Serial Number Bytes per second IOPS Number of Reads Number of Writes Data Read Data Written
-----------------------------------------------------------------------------------------------------------------------------------------------------
CRS_v001 00c0ff13349e0000fdd6c44f01000000 14.8KB 5 239611146 107147564 1321.1GB 110.5GB
DATA1_v001 00c0ff1311f30000d0d8c44f01000000 2402.8KB 218 1701488316 336678620 33.9TB 3184.6GB
DATA2_v001 00c0ff1311f3000040f9ce5701000000 0B 0 921 15 2273.7KB 2114.0KB
DATA_v001 00c0ff1311f30000bdd7c44f01000000 2303.4KB 209 1506883611 250984824 30.0TB 2026.6GB
FRA1_v001 00c0ff13349e00001ed9c44f01000000 709.1KB 28 25123082 161710495 1891.0GB 2230.0GB
FRA_v001 00c0ff13349e00001fd8c44f01000000 793.0KB 34 122052720 245322281 3475.7GB 3410.0GB
-----------------------------------------------------------------------------------------------------------------------------------------------------
Success: Command completed successfully.
The output as printed to the terminal (as mentioned, the 3rd command won't execute in its current state):
show controller-statistics
Durable ID CPU Load Power On Time (Secs) Bytes per second
IOPS Number of Reads Number of Writes Data Read
Data Written
----------------------------------------------------------------------
controller_A 3 45962495 3803.1KB
73 386765821 514137947 6687.8GB
5748.9GB
controller_B 20 45962413 5000.7KB
415 3208317860 587434274 63.9TB
5208.8GB
----------------------------------------------------------------------
Success: Command completed successfully.
Sending show volume-statistics
show vdisk-statistics
Name Serial Number Bytes per second IOPS
Number of Reads Number of Writes Data Read Data Written
----------------------------------------------------------------------------
CRS 00c0ff13349e000006d5c44f00000000 0B 0
45861 26756 3233.0MB 106.2MB
DATA 00c0ff1311f300006dd7c44f00000000 2187.2KB 152
23220764 76411017 5506.3GB 1604.1GB
DATA1 00c0ff1311f3000087d8c44f00000000 2295.2KB 154
23481442 78215540 5518.5GB 1602.6GB
DATA2 00c0ff1311f30000c2f8ce5700000000 0B 0
26 4 1446.9KB 65.5KB
FRA 00c0ff13349e000001d8c44f00000000 1829.3KB 14
3049951 15310681 1187.3GB 1941.2GB
FRA1 00c0ff13349e000007d9c44f00000000 1872.8KB 14
3016521 15228157 1179.3GB 1939.5GB
----------------------------------------------------------------------------
Success: Command completed successfully.
Traceback (most recent call last):
File "./fetchSAN.py", line 34, in <module>
child.expect('# ')
File "/Library/Python/2.7/site-packages/pexpect-4.2.1-py2.7.egg/pexpect/spawnbase.py", line 321, in expect
timeout, searchwindowsize, async)
File "/Library/Python/2.7/site-packages/pexpect-4.2.1-py2.7.egg/pexpect/spawnbase.py", line 345, in expect_list
return exp.expect_loop(timeout)
File "/Library/Python/2.7/site-packages/pexpect-4.2.1-py2.7.egg/pexpect/expect.py", line 107, in expect_loop
return self.timeout(e)
File "/Library/Python/2.7/site-packages/pexpect-4.2.1-py2.7.egg/pexpect/expect.py", line 70, in timeout
raise TIMEOUT(msg)
pexpect.exceptions.TIMEOUT: Timeout exceeded.
<pexpect.pty_spawn.spawn object at 0x105333910>
command: /usr/bin/ssh
args: ['/usr/bin/ssh', 'manage#10.254.27.49']
buffer (last 100 chars): '-------------------------------------------------------------\r\nPress any key to continue (Q to quit)'
before (last 100 chars): '-------------------------------------------------------------\r\nPress any key to continue (Q to quit)'
after: <class 'pexpect.exceptions.TIMEOUT'>
match: None
match_index: None
exitstatus: None
flag_eof: False
pid: 19519
child_fd: 5
closed: False
timeout: 30
delimiter: <class 'pexpect.exceptions.EOF'>
logfile: <open file 'FetchSan.log', mode 'w+' at 0x1053321e0>
logfile_read: None
logfile_send: None
maxread: 2000
ignorecase: False
searchwindowsize: None
delaybeforesend: 0.05
delayafterclose: 0.1
delayafterterminate: 0.1
searcher: searcher_re:
0: re.compile("# ")
And here is what is captured in the log:
Password: mypassword
HP StorageWorks MSA Storage P2000 G3 FC
System Name: Uninitialized Name
System Location:Uninitialized Location
Version:TS230P008
# show controller-statistics
show controller-statistics
Durable ID CPU Load Power On Time (Secs) Bytes per second
IOPS Number of Reads Number of Writes Data Read
Data Written
----------------------------------------------------------------------
controller_A 3 45962495 3803.1KB
73 386765821 514137947 6687.8GB
5748.9GB
controller_B 20 45962413 5000.7KB
415 3208317860 587434274 63.9TB
5208.8GB
----------------------------------------------------------------------
Success: Command completed successfully.
# show vdisk-statistics
show vdisk-statistics
Name Serial Number Bytes per second IOPS
Number of Reads Number of Writes Data Read Data Written
----------------------------------------------------------------------------
CRS 00c0ff13349e000006d5c44f00000000 0B 0
45861 26756 3233.0MB 106.2MB
DATA 00c0ff1311f300006dd7c44f00000000 2187.2KB 152
23220764 76411017 5506.3GB 1604.1GB
DATA1 00c0ff1311f3000087d8c44f00000000 2295.2KB 154
23481442 78215540 5518.5GB 1602.6GB
DATA2 00c0ff1311f30000c2f8ce5700000000 0B 0
26 4 1446.9KB 65.5KB
FRA 00c0ff13349e000001d8c44f00000000 1829.3KB 14
3049951 15310681 1187.3GB 1941.2GB
FRA1 00c0ff13349e000007d9c44f00000000 1872.8KB 14
3016521 15228157 1179.3GB 1939.5GB
----------------------------------------------------------------------------
Success: Command completed successfully.
# show volume-statistics
show volume-statistics
Name Serial Number Bytes per second
IOPS Number of Reads Number of Writes Data Read
Data Written
----------------------------------------------------------------------
CRS_v001 00c0ff13349e0000fdd6c44f01000000 11.7KB
5 239609039 107145979 1321.0GB
110.5GB
DATA1_v001 00c0ff1311f30000d0d8c44f01000000 2604.5KB
209 1701459941 336563041 33.9TB
3183.3GB
DATA2_v001 00c0ff1311f3000040f9ce5701000000 0B
0 921 15 2273.7KB
2114.0KB
DATA_v001 00c0ff1311f30000bdd7c44f01000000 2382.8KB
194 1506859273 250871273 30.0TB
2025.4GB
FRA1_v001 00c0ff13349e00001ed9c44f01000000 1923.5KB
31 25123006 161690520 1891.0GB
2229.1GB
FRA_v001 00c0ff13349e00001fd8c44f01000000 2008.5KB
37 122050872 245301514 3475.7GB
3409.1GB
----------------------------------------------------------------------
Press any key to continue (Q to quit)%
As a starting point: According to the manual, that SAN has a command to disable the pager. See the documentation for set cli-parameters pager off. It may be sufficient to execute that command. It may also have a command to set the terminal rows and columns that it uses for formatting output, although I wasn't able to find one.
Getting to your question: When an ssh client connects to a server and requests an interactive session, it can optionally request a PTY (pseudo-tty) for the server side of the session. When it does that, it informs the server of the lines, columns, and terminal type which the server should use for the TTY. Your SAN may honor PTY requests and use the lines and columns values to format its output. Or it may not.
The ssh client gets the rows and columns for the PTY request from the TTY for its standard input. This is the PTY which pexpect is using to communicate with ssh.
this question discusses how to set the terminal size for a pexpect session. Ssh doesn't honor the LINES or COLUMNS environment variables as far as I can tell, so I doubt that would work. However, calling child.setwinsize() after spawning ssh ought to work:
child = pexpect.spawn(cmd)
child.setwinsize(400,400)
If you have trouble with this, you could try setting the terminal size by invoking stty locally before ssh:
child=pexpect.spawn('stty rows x cols y; ssh user#host')
Finally, you need to make sure that ssh actually requests a PTY for the session. It does this by default in some cases, which should include the way you are running it. But it has a command-line option -tt to force it to allocate a PTY. You could add that option to the ssh command line to make sure:
child=pexpect.spawn('ssh -tt user#host')
or
child=pexpect.spawn('stty rows x cols y; ssh -tt user#host')

Plot sine wave of higher frequencies

I am sending serial data from uart to pc and trying to plot sine wave(using Python) sent from function generator through ADC12 of MSP430F5438A.
I am able to plot the wave for lower sampling frequencies(<120Hz) but when I increase the sampling frequency the digits get concatenated i.e. if two values 2563 , 2879 are sent through uart then python reads them as 25632879. So, I am not able to plot the graph as values are not correct.
I am sending the values without new line between them, if I send with new line then the values are not read correctly - python reads them with space in between so then again I get another error: could not convert string to float.
I tried data = ser.readline() as well but no luck
I am attaching the code below.Please see if anything can be done to solve this problem.
import sys
import serial
import numpy as np
import matplotlib.pyplot as plt
from collections import deque
port = "COM11"
baud = 9600
timeout=1
ser = serial.Serial()
ser.port = port
ser.baudrate = baud
ser.timeout = timeout
a1 = deque([0.0]*100)
#ax = plt.axes(xlim=(0, 100), ylim=(0, 1000))
line, = plt.plot(a1)
plt.ion()
plt.ylim([0,1000])
try:
ser.open()
except:
sys.stderr.write("Error opening serial port %s\n" % (ser.portstr) )
sys.exit(1)
#ser.setRtsCts(0)
while 1:
# Read from serial port, blocking
data = ser.read(1)
# If there is more than 1 byte, read the rest
n = ser.inWaiting()
data = data + ser.read(n)
#sys.stdout.write(data)
print(a1)
a1.appendleft((data))
datatoplot = a1.pop()
line.set_ydata(a1)
plt.draw()
Thanks
I can think of two ways for doing this reliably:
Use a delimiter . All you need is to parse the value correctly:
line = serial.readline()
reading = int(line)
If you don't want to use a delimiter then send a formatted reading from the msp:
uint8_t buffer[5];
snprintf(buffer, 4, "%04d", reading);
uart_print(buffer);
this way you will always get 4 characters per reading, so you can do this in the python code:
line = serial.read(4)
reading = int(line)
I'd still go for the first alternative, though.

plot goes to zero in between

import sys
import serial
import numpy as np
import matplotlib.pyplot as plt
from collections import deque
port = "COM11"
baud = 9600
timeout=1
ser = serial.Serial()
ser.port = port
ser.baudrate = baud
ser.timeout = timeout
a1 = deque([0.0]*100)
#ax = plt.axes(xlim=(0, 100), ylim=(0, 1000))
line, = plt.plot(a1)
plt.ion()
plt.ylim([0,1000])
try:
ser.open()
except:
sys.stderr.write("Error opening serial port %s\n" % (ser.portstr) )
sys.exit(1)
#ser.setRtsCts(0)
while 1:
# Read from serial port, blocking
data = ser.read(1)
# If there is more than 1 byte, read the rest
n = ser.inWaiting()
data = data + ser.read(n)
#sys.stdout.write(data)
print(a1)
a1.appendleft((data))
datatoplot = a1.pop()
line.set_ydata(a1)
plt.draw()
I am using msp430f5438a board.If I send the data with a new line between each data then I am not able to plot the data because in python the sometimes data gets printed as 78_9, 7_89,_789 where _ means space so python gives me a error cannot convert string to float. But If I say send the data from uart without any new line between them then I get a nice plot but in the plot after some irregular short intervals the plot goes to zero and then becomes fine again although I checked in hyperterminal I am not receiving any zero values
My question is:
Are the two cases I described are related to each other?What can be done to rectify this problem of plot going to zero in between?Because of this I am not getting a smooth wave.
Thanks
The problem might be in the way you handle the serial interface. As you are not parsing the serial input when it comes, it is possible that you receive two messages as follows:
1.23
4.56
7.
and
89
10.11
etc. This is because your code may split the input at any point. It may be so fast that you get one digit at a time, which is probably not what you wanted.
I suggest that if you pad your data with newlines and if the data is good in a terminal program, you use the readline method.
while 1:
# read a line from the input
line = ser.readline()
# try to make a float out of it
try:
a1.appendleft(float(line))
except ValueError:
# in case we got bad input, print it and go to the next line
print "Received an invalid line '{0}'".format(line)
continue
# do the plotting
This most probably fixes your problem.
Reading asynchronous serial line is surprisingly complicated, you usually need to parse the input in-the-fly with timeouts. Fortunately, this is done by pyserial when using readline.

Should i use regex look back to search this string?

My intended goal:
Need to pull IP address out of a string as i loop through pinging different subnets.
Example of string i will be searching:
>>> pingout = subprocess.getoutput('ping -c' + ' ' + str(1) + ' ' + '10.20.' + str(234.) + str(4))
>>> print(pingout)
PING 10.20.234.04 (10.20.234.4) 56(84) bytes of data.
64 bytes from 10.20.234.4: icmp_seq=1 ttl=64 time=4.01 ms
--- 10.20.234.04 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 4.016/4.016/4.016/0.000 ms
>
I want to use something like this.
match = re.search(r'?<=...........)ping', pingout)
Or this but need to exclude the 'ping' in the result.
match = re.search(r'.............ping',pingout)
Given:
txt='''\
PING 10.20.234.04 (10.20.234.4) 56(84) bytes of data.
64 bytes from 10.20.234.4: icmp_seq=1 ttl=64 time=4.01 ms
--- 10.20.234.04 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 4.016/4.016/4.016/0.000 ms'''
For the first IP address (with 'PING' in the line), you can do:
print(re.search(r'^PING\s*(\d+\.\d+\.\d+\.\d+)', txt, re.M).group(1))
For the second IP address (with 'ping' in the line), you can do:
print(re.search(r'^\D+(\d+\.\d+\.\d+\.\d+)\s*ping', txt, re.M).group(1))
Either case, prints 10.20.234.04
You can use tons of different regexes to extract those IP, I think the closest one to your solutions is:
>>> s
'--- 10.20.234.04 ping statistics ---'
>>> re.findall(".{12}(?= ping)", s)
['10.20.234.04']
But;
>>> import socket
>>> socket.gethostbyname("google.com")
'173.194.39.199'
Would be much better I think(Don't know if it works on subnets tho, but I think they must be the same thing.).
>>> import re
>>> m = re.match(r"PING .+ \((.+)\) .+ bytes of data.", "PING 10.20.234.04 (10.20.234.4) 56(84) bytes of data.")
>>> m
>>> m.groups()
('10.20.234.4',)
If I were going to use the regex repeatedly I would do:
ip_re = re.compile(r"PING .+ \((.+)\) .+ bytes of data.")
m = ip_re.match(pingout)
Note this catches the IP exactly once when ping prints its first line. No need to keep searching. You could extend the regex to catch the amount of data sent or the responses if you need to do so.
Try this regex,
match = re.search(r'.*PING ([0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}).*', PINGOUT)
match.group(1)
Example:
>>> match = re.search(r'.*PING ([0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}).*', "PING 10.20.234.04")
>>> match.group()
'PING 10.20.234.04'
>>> match.group(1)
'10.20.234.04'

Parse ASCII Output of a Device-File in C++

i wrote a kernelspace driver for a USB-device. If it is connected it mounts to /dev/myusbdev0 for example.
Via command line with echo -en "command" > /dev/myusbdev0 i can send commands to the device and read results with cat /dev/myusbdev0.
Ok, now i have to write a C++ program. At first i would open the device file for read/write with:
int fd = open("/dev/echo", O_RDRW);
After that a cmd will be send to get the device working:
char cmd[] = { "\x02sEN LMDscandata 1\x03" };
write(fd, cmd, sizeof(cmd));
Now i get to the part i dont now how to handle yet. i now need to read from the device, as its keeping on sending me data continously. this data i need to read and parse now ...
char buf[512];
read(fd, buf, sizeof(buf);
The data looks like following, each one starts with \x02 and ends with \x03, they are not always the same size:
sRA LMDscandata 1 1 89A27F 0 0 343 347 27477BA9 2747813B 0 0 7 0 0
1388 168 0 1 DIST1 3F800000 00000000 186A0 1388 15 8A1 8A5 8AB 8AC 8A6
8AC 8B6 8C8 8C2 8C9 8CB 8C4 8E4 8E1 8EB 8E0 8F5 908 8FC 907 906 0 0 0
0 0 0 All Values are separated with a 20hex {SPC}
it think i need some kind of while loop to continiously read the data from an \x02 until i read a \x03.
if i have a complete scan, i need to parse this ascii message in its seperate parts (some variables uint_16, uint_8, enum_16, ...).
any idea how i can read a complete scan into a buf[] and then parse its components out?
As you say the device is sending continiously, i would recommend adding a queue to hold the chunks coming in, and some dispatching that takes out parts of the queue, i.e. x02 to x03, decoupling the work that is done from receiving chunks.
Furthermore you can have then single objects handling one complete block from x02 to x03, perhaps threaded (makes sense with the information given).
device => chunk reader => input queue => inputer reader => data handling
hope this helps