Expect Script to match on multi-line response - regex

I am receiving the response to a Unix cat /etc/hosts and would like to match on this data with Expect - so I can write it local disk.
I have
#!/usr/bin/expect -f
set timeout -1
spawn nc -lvnp 9090
match_max 100000
expect -re {.*(Connection received).*}
puts "Getting hosts file"
send -- "cat /etc/hosts\r"
expect -re "(.*)\n"
The host file is multiple lines and I am unable to construct the correct regular expression to match on it.
Help would be gratefully received
Update
When interact with when the remote service via netcat manually the response interaction looks like this :
nc -lvnp 9090 Listening on 0.0.0.0 9090 Connection received on
192.168.236.135 44686 cat /etc/hosts
127.0.0.1 localhost
127.0.1.1 ubuntu
# The following lines are desirable for IPv6 capable hosts ::1 ip6-localhost ip6-loopback fe00::0 ip6-localnet ff00::0 ip6-mcastprefix ff02::1 ip6-allnodes ff02::2 ip6-allrouters
my Expect code is :
#!/usr/bin/expect -f
set timeout -1
spawn nc -lvnp 9090
match_max 100000
expect -re {.*(Connection received).*}
puts "******Getting hosts file*******"
log_user 0
send -- "cat /etc/hosts \r"
expect -re "(.*\[\r\n])+"
#expect -re "(.*\[\r|\n]).*"
#expect -re "(.*\[\t|\r|\n]).*"
expect -re "(.*\[\r\n]).*$"
puts "Output 0 string"
puts <<<<$expect_out(0,string)>>>>
puts ""
puts "Output 1 string"
puts <<<<$expect_out(1,string)>>>>
puts ""
puts "Output buffer"
puts <<<<$expect_out(buffer)>>>>
puts ""
puts [open hosts.txt w] $expect_out(buffer)
send -- "\x03"
Just seem to be unable to match on the whole response - I'm not 100% sure if I need to match on EOF or EOL or something else. The newlines and tab are also part of the problem I think.
I basically just to to capture the whole response from the 'send -- "cat /etc/hosts" and write that to disk

turns out he response from the remote server was too quick.
A simple 'sleep 1' was good enough to slow things down
#!/usr/bin/expect -f
set timeout -1
spawn nc -lvnp 9090
match_max 100000
expect -re {.*(Connection received).*}
puts "Getting hosts file"
send -- "cat /etc/hosts\r"
sleep 1
expect -re "(.*)\n"

Related

Why doesn't this program copy all my files, using scp?

I wrote a little program which, in first case, slices a list of files and directory names so I get a list of files and a list of directories.
#!/usr/bin/expect -f
set mdproot "****"
set mySource "****"
set myDest "****"
set myDirs {}
set myFiles {}
set i 1
set y 0
lappend myDirs [lindex $argv 0]
foreach variable $argv {
lappend myDirs [lindex $argv $i+1]
lappend myFiles [lindex $argv $y+1]
incr y
incr y
incr i
incr i
}
set DIRECTORIES [lsearch -all -inline -not -exact $myDirs {}]
set FILES [lsearch -all -inline -not -exact $myFiles {}]
#puts " "
#puts $DIRECTORIES
#puts " "
#puts $FILES
foreach file $FILES dir $DIRECTORIES {
puts " Fichier : $file et repertoire : $dir"
spawn scp -p "$mySource/$file" "$myDest/$dir"
expect -re "(.*)assword: " {sleep 1; send -- "$mdproot\r" }
expect eof { return}
}
There are my lists:
$argv :
2017-11-30
2017-11-30_15-10-44_P_8294418.33_Q1
2017-11-30
2017-11-30_15-10-44_R_8294418.33_Q1
2018-03-07
2018-03-07_09-30-57_R_HOURS_Q1
2018-04-13
2018-04-13_13-23-25_R_HOURS_Q1
2018-05-02
2018-05-02_11-19-37_R_HOURS_Q1
2018-03-07
2018-3-7_9-30-57_P_HOURS_Q1
2018-04-13
2018-4-13_13-23-25_P_HOURS_Q1
2018-05-02
2018-5-2_11-19-37_P_HOURS_Q1
$DIRECTORIES :
2017-11-30
2017-11-30
2018-03-07
2018-04-13
2018-05-02
2018-03-07
2018-04-13
2018-05-02
$FILES :
2017-11-30_15-10-44_P_8294418.33_Q1
2017-11-30_15-10-44_R_8294418.33_Q1
2018-03-07_09-30-57_R_HOURS_Q1
2018-04-13_13-23-25_R_HOURS_Q1
2018-05-02_11-19-37_R_HOURS_Q1
2018-3-7_9-30-57_P_HOURS_Q1
2018-4-13_13-23-25_P_HOURS_Q1
2018-5-2_11-19-37_P_HOURS_Q1
Actually I have 2 problems (3 in the case we count how trash this code is).
First, when I run my program, I have my % indicator for each file I am copying and except the last one, they all stop before they get to 100%.
Then, I can see that the scp command isn't done on all the files, the program stops pretty much every time at the 4th file.
root#raspberrypi:~# ./recupFileName.sh
spawn scp -p /root/muonic_data/2017-11-30_15-10-44_P_8294418.33_Q1 marpic#192.168.110.90:/home/marpic/muonic_data/Data_Q1/2017-11-30
marpic#192.168.110.90's password:
2017-11-30_15-10-44_P_8294418.33_Q1 15% 68MB 6.9MB/s 00:53
spawn scp -p /root/muonic_data/2017-11-30_15-10-44_R_8294418.33_Q1 marpic#192.168.110.90:/home/marpic/muonic_data/Data_Q1/2017-11-30
marpic#192.168.110.90's password:
2017-11-30_15-10-44_R_8294418.33_Q1 41% 69MB 8.5MB/s 00:11
spawn scp -p /root/muonic_data/2018-03-07_09-30-57_R_HOURS_Q1 marpic#192.168.110.90:/home/marpic/muonic_data/Data_Q1/2018-03-07
marpic#192.168.110.90's password:
2018-03-07_09-30-57_R_HOURS_Q1 82% 51MB 7.2MB/s 00:01
spawn scp -p /root/muonic_data/2018-04-13_13-23-25_R_HOURS_Q1 marpic#192.168.110.90:/home/marpic/muonic_data/Data_Q1/2018-04-13
marpic#192.168.110.90's password:
2018-04-13_13-23-25_R_HOURS_Q1 100% 6940KB 6.8MB/s 00:01
As you can see, there should be 8 files copied with 100% accuracy, but there are no error messages so I don't know where to start my research.
EDIT :
I added the "set timeout -1" in my script, but now the script is copying only my first file with 100% accuracy, then stops. Any answers ?
root#raspberrypi:~# ./recupFileName.sh
Fichier : 2017-11-30_15-10-44_P_8294418.33_Q1 et repertoire : 2017-11-30
spawn scp -p /root/muonic_data/2017-11-30_15-10-44_P_8294418.33_Q1 marpic#192.168.110.90:/home/marpic/muonic_data/Data_Q1/2017-11-30
marpic#192.168.110.90's password:
2017-11-30_15-10-44_P_8294418.33_Q1 100% 437MB 7.5MB/s 00:58
root#raspberrypi:~#
The problem should be in expect eof. By default the timeout is 10 seconds so expect eof would return after 10 seconds though the scp is still running.
You can use a larger timeout.
Option #1:
# set the default `timeout'
set timeout 3600 ; # or -1 for no timeout
Option #2:
expect -timeout 3600 eof
Note that your
expect eof { return }
would exit the whole script so the foreach loop runs for only once, you need just
expect eof

AWK catching a regular expression

I have been using this little script for months now with success. Today I realize there is one output it cant seem to catch, screen comes up blank with a new prompt:
user#computer ~]$ myscan ipsFile 23
user#computer ~]$
Here is the code
#!/bin/bash
sudo nmap -v -Pn -p T:$2 -reason -i $1 | awk ' {
if (/syn-ack/) {
print "Yes"
c++
}
else if (/no-response|reset|host-unreach/) {
print "No"
c++
}
}
END { print c} '
If I run the nmap against one of the IPs then it returns
Starting Nmap 5.51 ( http://nmap.org ) at 2017-09-26 11:44 CDT
Initiating Parallel DNS resolution of 1 host. at 11:44
Completed Parallel DNS resolution of 1 host. at 11:44, 0.00s elapsed
Initiating Connect Scan at 11:44
Scanning 1.1.1.1 [1 port]
Completed Connect Scan at 11:44, 0.20s elapsed (1 total ports)
Nmap scan report for 1.1.1.1
Host is up, received user-set (0.20s latency).
PORT STATE SERVICE REASON
23/tcp filtered telnet host-unreach
Read data files from: /usr/share/nmap
Nmap done: 1 IP address (1 host up) scanned in 0.26 seconds
How can I catch the 'host-unreach' portion?
Let's try and debug this. Execute this:
nmap -v -Pn -p T:23 -reason -i ipsFile | awk '{print $0}/syn-ack/{print "Yes";c++}/no-response|reset|host-unreach/{print "No";c++}END {print c}' > out.txt
The only difference here is that the awk script prints $0 (i.e. the output of your nmap calls) to file out.txt. Try to grep your unreach value.
I tried this myself and found that instead of a host-unreach I got a net-unreach. Might be the same thing in your case.
Have you tried piping stderr to stdout like
#!/bin/bash
sudo nmap -v -Pn -p T:$2 -reason -i $1 2>&1 | awk ' {
if (/syn-ack/) {
print "Yes"
c++
}
else if (/no-response|reset|host-unreach/) {
print "No"
c++
}
}
END { print c} '

Regular expressions in expect and shell script

Friends , im trying to automate a routing using expect , basically its a debug plugin in a special equipment that i need to log some data , to access this debug plugin my company needs to give me a responsekey based on a challengekey , its a lot of hosts and i need to deliver this by friday , what i've done so far.
#!/usr/bin/expect -f
match_max 10000
set f [open "cimc.txt"]
set hosts [split [read $f] "\n"]
close $f
foreach host $hosts {
spawn ssh ucs-local\\marcos#10.2.8.2
expect "Password: "
send "Temp1234\r"
expect "# "
send "connect cimc $host\r"
expect "# "
send "load debug plugin\r"
expect "ResponseKey#>"
sleep 2
set buffer $expect_out(buffer)
set fid [open output.txt w]
puts $fid $buffer
close $fid
sleep 10
spawn ./find-chag.sh
sleep 2
set b [open "key.txt"]
set challenge [read $b]
close $b
spawn ./find-rep.sh $challenge
sleep 3
set c [open "rep.txt"]
set response [read $c]
close $c
puts Response-IS
send "\r"
expect "ResponseKey#> "
send "$response"
}
$ cat find-chag.sh
cat output.txt | awk 'match($0,"ChallengeKey"){print substr($0,RSTART+15,38)}' > key.txt
$ cat find-rep.sh
curl bla-blabla.com/CIMC-key/generate?key=$1 | grep ResponseAuth | awk 'match($0,"</td><td>"){print substr($0,RSTART+9,35)}' > rep.txt
i dont know how to work with regexp on expect so i put the buffer output to a file and used bash script , the problem is that after i run the scripts with spawn looks like my ssh session is lost , does anyone have any tips? should i use something else instead of spawn to invoke my scripts?
expect -re "my tcl compatible regular expression goes here"
Should allow you to use regular expressions.

Bash Script: sed/awk/regex to match an IP address and replace

I have a string in a bash script that contains a line of a log entry such as this:
Oct 24 12:37:45 10.224.0.2/10.224.0.2 14671: Oct 24 2012 12:37:44.583 BST: %SEC_LOGIN-4-LOGIN_FAILED: Login failed [user: root] [Source: 10.224.0.58] [localport: 22] [Reason: Login Authentication Failed] at 12:37:44 BST Wed Oct 24 2012
To clarify; the first IP listed there "10.224.0.2" was the machine the submitted this log entry, of a failed login attempt. Someone tried to log in, and failed, from the machine at the 2nd IP address in the log entry, "10.224.0.58".
I wish to replace the first occurrence of the IP address "10.224.0.2" with the host name of that machine, as you can see presently is is "IPADDRESS/IPADDRESS" which is useless having the same info twice. So here, I would like to grep (or similar) out the first IP and then pass it to something like the host command to get the reverse host and replace it in the log output.
I would like to repeat this for the 2nd IP "10.224.0.58". I would like to find this IP and also replace it with the host name.
It's not just those two specific IP address though, any IP address. So I want to search for 4 integers between 1 and 3, separated by 3 full stops '.'
Is regex the way forward here, or is that over complicating the issue?
Many thanks.
Replace a fixed IP address with a host name:
$ cat log | sed -r 's/10\.224\.0\.2/example.com/g'
Replace all IP addresses with a host name:
$ cat log | sed -r 's/[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}/example.com/g'
If you want to call an external program, it's easy to do that using Perl (just replace host with your lookup tool):
$ cat log | perl -pe 's/(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})/`host \1`/ge'
Hopefully this is enough to get you started.
There's variou ways to find th IP addresses, here's one. Just replace "printf '<<<%s>>>' " with "host" or whatever your command name is in this GNU awk script:
$ cat tst.awk
{
subIp = gensub(/\/.*$/,"","",$4)
srcIp = gensub(/.*\[Source: ([^]]+)\].*/,"\\1","")
"printf '<<<%s>>>' " subIp | getline subName
"printf '<<<%s>>>' " srcIp | getline srcName
gsub(subIp,subName)
gsub(srcIp,srcName)
print
}
$
$ gawk -f tst.awk file
Oct 24 12:37:45 <<<10.224.0.2>>>/<<<10.224.0.2>>> 14671: Oct 24 2012 12:37:44.583 BST: %SEC_LOGIN-4-LOGIN_FAILED: Login failed [user: root] [Source: <<<10.224.0.58>>>] [localport: 22] [Reason: Login Authentication Failed] at 12:37:44 BST Wed Oct 24 2012
googled this one line command together. but was unable to pass the founded ip address to the ssh command:
sed -n 's/\([0-9]\{1,3\}\.\)\{3\}[0-9]\{1,3\}/\nip&\n/gp' test | grep ip | sed 's/ip//' | sort | uniq
the "test" is the file the sed command is searching for for the pattern

Getting a value from expect shell and use it outside

I am using the following code within a procedure in order to get an IP address of eth1 and use it as a gateway. Another thing I want is to use this variable (the IP address) outside, in the user scope.
global my_gw_ip_address
expect "# "
send "ifconfig eth1 | grep 'inet addr:' | cut -d: -f2 | awk '{print \$1}' > prod_ip_base.txt\r"
expect "# "
send " my_internal_gw=`cat prod_ip_base.txt`\r"
expect "# "
send " echo \$my_internal_gw\r"
expect "root#cnode-pp81:~ "
set checking_buffer_out $expect_out(buffer)
regexp {(?:\d+\.){3}\d+} $checking_buffer_out my_gw_ip_address
puts "internal gw: $my_gw_ip_address\n"
the output of the function is:
1. the line send " echo \$my_internal_gw\r" returns the correct IP address 192.168.138.50
2. the line puts "internal gw: $my_gw_ip_address\n" returns internal gw: 0.
can anyone please tell me what I do wrong? Why the variable $my_gw_ip_address is 0?
I solve the problem.
I should have added sleep 1 between the following commands.
change
set checking_buffer_out $expect_out(buffer)
regexp {(?:\d+\.){3}\d+} $checking_buffer_out my_gw_ip_address
to be:
set checking_buffer_out $expect_out(buffer)
sleep 1
regexp {(?:\d+\.){3}\d+} $checking_buffer_out my_gw_ip_address
Amigal