regexp Matching capture and exp_send - regex

Currently I have a script which is extracting the line # number from the below menu response which has the SNMP user I’m interested in “SalVhugoV3”.
Here is the menu response in blue
Select an SNMP user:
1) cmssnmp
2) atlas
3) SalVhugoV3
Enter choice (1-3):
This working section of the script looks for the string containing “SalVhugoV3” and takes away the option #3 as that’s the option of the user it’s looking for. Then sends “3” as the snmpuserChoice for the “Enter choice” prompt.
Choose V3 User From the List/Menu, grep for $V3USER and send it's number
expect -re ".*\\(.*\\):" {
append rawOut $expect_out(buffer)
regexp { .*(([0-9])\) [puts $V3USER].*) } $expect_out(buffer) cmdOut snmpUser snmpUserChoice
exp_send -- "$snmpUserChoice\r"
}
I’m trying to do the same thing but with this extended response , as you can see the option for the user I’m looking for in this case is option “5” as it contains “SalVhugoV3” at the end of the string. I believe this is what needs to change “ regexp { .(([0-9])) [puts $V3USER].) } “ but I don’t exactly how to modify in order for it to grab the option “5” number so it can be sent via the “exp_send”.
Select an SNMP Connection to delete:
1) Connection to NMS: Host 135.9.60.143, Port 162, User cmssnmp
2) Connection to SAL: Host 135.9.158.252, Port 162, User cmssnmp
3) Connection to SAL: Host 135.9.158.236, Port 162, User cmssnmp
4) Connection to NMS: Host 135.105.6.67, Port 162, User atlas
5) Connection to SAL: Host 135.60.25.147, Port 162, User SalVhugoV3
Enter choice (1-5):
Choose V3 User From the List/Menu, grep for $V3USER and send it's number
expect -re ".*\\(.*\\):" {
append rawOut $expect_out(buffer)
regexp { .*(([0-9])\) [puts $V3USER].*) } $expect_out(buffer) cmdOut snmpUser snmpUserChoice
exp_send -- "$snmpUserChoice\r"
}

regexp "((\[0-9\])\\)\[^\\)\]*$V3USER)" $expect_out(buffer) cmdOut snmpUser snmpUserChoice
For V3USER=atlas it would give:
snmpUser='4) Connection to NMS: Host 135.105.6.67, Port 162, User atlas'
snmpUserChoice='4'
For better readability and to avoid escaping:
set gen_expr {([0-9])\)[^\)]*}
set spec_expr "($choice_expr$V3USER)"
regexp $spec_expr $expect_out(buffer) cmdOut snmpUser snmpUserChoice
BTW: term { ... [puts $V3USER] } puzzles me. There is no escaping in braces so this is just interpreted as piece of regexp thus expects one of the characters 'p', 'u', 't', 's', ' ', $', etc ...

Related

Grepping two patterns from event logs

I am seeking to extract timestamps and ip addresses out of log entries containing a varying amount of information. The basic structure of a log entry is:
<timestamp>, <token_1>, <token_2>, ... ,<token_n>, <ip_address> <token_n+2>, <token_n+3>, ... ,<token_n+m>,-
The number of tokens n between the timestamp and ip address varies considerably.
I have been studying regular expressions and am able to grep timestamps as follows:
grep -o "[0-9]\{4\}-[0-9]\{2\}-[0-9]\{2\}T[0-9]\{2\}:[0-9]\{2\}:[0-9]\{2\}"
And ip addresses:
grep -Eo '[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}'
But I have not been able to grep both patterns out of log entries which contain both. Every log entry contains a timestamp, but not every entry contains an ip address.
Input:
2021-04-02T09:06:44.248878+00:00,Creation Time,EVT,WinEVTX,[4624 / 0x1210] Source Name: Microsoft-Windows-Security-Auditing Message string: An account was successfully logged on.\n\nSubject:\n\tSecurity ID:\t\tS-1-5-18\n\tAccount Name:\t\tREDACTED$\n\tAccount Domain:\t\tREDACTED\n\tLogon ID:\t\tREDACTED\n\nLogon Type:\t\t\t10\n\nNew Logon:\n\tSecurity ID:\t\tREDACTED\n\tAccount Name:\t\tREDACTED\n\tAccount Domain:\t\tREDACTED\n\tLogon ID:\t\REDACTED\n\tLogon GUID:\t\tREDACTED\n\nProcess Information:\n\tProcess ID:\t\tREDACTED\n\tProcess Name:\t\tC:\Windows\System32\winlogon.exe\n\nNetwork Information:\n\tWorkstation:\tREDACTED\n\tSource Network Address:\t255.255.255.255\n\tSource Port:\t\t0\n\nDetailed Authentication Information:\n\tLogon Process:\t\tUser32 \n\tAuthentication Package:\tNegotiate\n\tTransited Services:\t-\n\tPackage Name (NTLM only):\t-\n\tKey Length:\t\t0\n\nThis event is generated when a logon session is created. It is generated on the computer that was accessed.\n\nThe subject fields indicate the account on the local system which requested the logon. This is most commonly a service such as the Server service or a local process such as Winlogon.exe or Services.exe.\n\nThe logon type field indicates the kind of logon that occurred. The most common types are 2 (interactive) and 3 (network).\n\nThe New Logon fields indicate the account for whom the new logon was created i.e. the account that was logged on.\n\nThe network fields indicate where a remote logon request originated. Workstation name is not always available and may be left blank in some cases.\n\nThe authentication information fields provide detailed information about this specific logon request.\n\t- Logon GUID is a unique identifier that can be used to correlate this event with a KDC event.\n\t- Transited services indicate which intermediate services have participated in this logon request.\n\t- Package name indicates which sub-protocol was used among the NTLM protocols.\n\t- Key length indicates the length of the generated session key. This will be 0 if no session key was requested. Strings: ['S-1-5-18' 'DEVICE_NAME$' 'NETWORK' 'REDACTED' 'REDACTED' 'USERNAME' 'WORKSTATION' 'REDACTED' '10' 'User32 ' 'Negotiate' 'REDACTED' '{REDACTED}' '-' '-' '0' 'REDACTED' 'C:\\Windows\\System32\\winlogon.exe' '255.255.255.255' '0' '%%1833'] Computer Name: REDACTED Record Number: 1068355 Event Level: 0,winevtx,OS:REDACTED,-
Desired Output:
2021-04-02T09:06:44, 255.255.255.255
$ sed -En 's/.*([0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}).*[^0-9]([0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}).*/\1, \2/p' file
2021-04-02T09:06:44, 255.255.255.255
Your regexps can be reduced by removing some of the explicit repetition though:
$ sed -En 's/.*([0-9]{4}(-[0-9]{2}){2}T([0-9]{2}:){2}[0-9]{2}).*[^0-9](([0-9]{1,3}\.){3}[0-9]{1,3}).*/\1, \4/p' file
2021-04-02T09:06:44, 255.255.255.255
It could be simpler still if all of the lines in your log file start with a timestamp:
$ sed -En 's/([^,.]+).*[^0-9](([0-9]{1,3}\.){3}[0-9]{1,3}).*/\1, \2/p' file
2021-04-02T09:06:44, 255.255.255.255
If you are looking for lines that contain both patterns, it may be easiest to do it two separate searches.
If you're searching your log file for lines that contain both "dog" and "cat", it's usually easiest to do this:
grep dog filename.txt | grep cat
The grep dog will find all lines in the file that match "dog", and then the grep cat will search all those lines for "cat".
You seem not to know the meaning of the "-o" switch.
Regular "grep" (without "-o") means: give the entire line where the pattern can be found. Adding "-o" means: only show the pattern.
Combining two "grep" in a logical AND-clause can be done using a pipe "|", so you can do this:
grep <pattern1> <filename> | grep <pattern2>

Awk - replace coumn 2 in table 1 from coumn 2 in table 2 based on matching data in column 1 (common between tables)

After my company purchased new servers I'm doing a top-down upgrade of the server room. since all the hardware is changing I'm not able to use bare-metal cloning tool to migrate. Using the newusers command from Debian I am able to create in bulk all the users from the old server. For the /etc/shadow file, you can copy the second column from your shadow.sync (from the old server) file into the second column of the associated account in the new system. This will transfer the passwords for your accounts to the new system. However I'm not sure how to do this programmatically using awk (or something else I can integrate into my shell script I already have setup).
shadow.sync contains the following (users & passwords changed for security reasons) This is the file to be copied INTO the current shadow file which looks almost identical except the data in the second column has the INCORECT values.
An in-depth explanation of the fields for the /etc/shadow file can be found here
user1:$6$HiwQEKYDgT$xYU9F3Wv0jFWHmZxN60nFMkTqWn87RRIOvx7Epp57rOmdHN9plJgjhC.jRVVNc1.HUaqSpX/ZcCEFSn6RmQQA0:17531::0:99999:7:::
user2:$6$oOuwJtrIKk$THLsfDppLI8QVw9xEOAaIoZ90Mcz3xGukVdyWGJJqygsavtXvtJ8X9ECc0CfuGzHp0pHNSAqdZY9TAzF5YKLc.:17531::0:99999:7:::
user3:$6$IEHAyRsokQ$e5K3RicE.PUAej8IxG9GnF/SUl1NQ57pqzUVuAzsP8.89SNhuaKE1W7kG5P4hbzV23Bb2zWHx353t.e9ERSVy.:17531::0:99999:7:::
user4:$6$lFOIUQvxdb$W5ITiH/Y021xw1vo8uw6ZtIOmfKjnNnC/SttQjN85MHtLbFeQ2Th5kfAIijXC81CRG4T0kJQ3rzRNRSyQHjyb1:17531::0:99999:7:::
user5:$6$RZbtYxWiwE$lnP8.tTbs0JbLZg5FsmPR8QvrJARbcRuJi2nYm1okwjfkWPkj212mBPjVF1BTo2hVCxLGSw64Cp6DgXheacSx.:17531::0:99999:7:::
Essentially i need to match column 1 (username) between the sync file and the shadow file and copy column 2 from the sync file over-top of the same column on the actual shadow file. Doing this by hand would be terrible as I have 90 servers that I'm migrating with over 900 users total.
Random shadow.sync file for demonstration was generated using:
#!/usr/bin/env python
import random, string, crypt, datetime
userList = ['user1','user2','user3','user4','user5']
dateNow = (datetime.datetime.utcnow() - datetime.datetime(1970,1,1)).days
for user in userList:
randomsalt = ''.join(random.sample(string.ascii_letters,10))
randompass = ''.join(random.sample(string.ascii_letters,10))
print("%s:%s:%s::0:99999:7:::" % (user, crypt.crypt(randompass, "$6$"+randomsalt), dateNow))
Please note this python script was ONLY for demonstration and not for actual production data. As users are added to the server the /etc/shadow file is generated with the password presented on the command line. The Original data (from shadow.sync) needs to be "Merged" with the data in /etc/shadow after the newusers command is run (which essentially sets every password to the letter x)
#!/usr/bin/env python
with open('/etc/shadow','rb') as file:
for line in file:
TargetLine = line.rstrip().split(":")
with open('shadow.sync','rb') as shadow:
for row in shadow:
SyncLine = row.rstrip().split(":")
if TargetLine[0] == SyncLine[0]:
TargetLine[1] = SyncLine[1]
break
print "NEW MODIFIED LINE: %s" % ":".join(TargetLine)
This will open the /etc/shadow file and loop through the lines. For each line on the /etc/shadow file we loop through the shadow.sync file once a match for the usernames TargetLine[0] == SyncLine[0] the password field is modified and the loop is broken.
If a match is NOT found (username in /etc/shadow but NOT in the shadow.sync file) the if block on the inner loop falls through and the line is left untouched the results are handled on the final print statement. As this answers the question I will leave the data output and file manipulation up to the user.
use Data::Dumper;
# we only need to process the sync file once -
# and store what we find in a hash (dictionary)
open $fh1, '<', 'shadow.sync.txt';
while (<$fh1>)
{
m/^([^:]+):(.*)$/;
$hash->{$1} = $2;
}
close $fh1;
# this shows us what we found & stored
print Dumper $hash;
# now we'll process the shadow file which needs updating -
# here we output a side-by-side comarison of what the passwords
# currently are & what they will be updated to (from the hash)
open $fh2, '<', 'shadow.txt';
open $fh3, '>', 'shadow.UPDATED.txt';
while (<$fh2>)
{
m/^([^:]+):(.*)$/;
printf ( "%s => %s\n", $1, $2 );
printf ( "%s => %s\n\n", $1, $hash->{$1} );
printf $fh3 ( "%s:%s\n", $1, $hash->{$1} );
}
close $fh3;
close $fh2;
Sample output:
$VAR1 = {
'user5' => '$6$RZbtYxWiwE$lnP8w64Cp6DgXheacSx.:17531::0:99999:7:::',
'user1' => '$6$HiwVVNc1.HUaqSpX/ZcCEFSn6RmQQA0:17531::0:99999:7:::',
'user4' => '$6$lFOIUQv1CRG4T0kJQ3rzRNRSyQHjyb1:17531::0:99999:7:::',
'user3' => '$6$P8.89SNhu23Bb2zWHx353t.e9ERSVy.:17531::0:99999:7:::',
'user2' => '$6$Cc0CfuGzHp0pHNSAqdZY9TAzF5YKLc.:17531::0:99999:7:::'
};
user1 => $6$RANDOM1RANDOM1RANDOM1RANDOM1:17531::0:99999:7:::
user1 => $6$HiwVVNc1.HUaqSpX/ZcCEFSn6RmQQA0:17531::0:99999:7:::
user2 => $6$RANDOM2RANDOM2RANDOM2RANDOM2:17531::0:99999:7:::
user2 => $6$Cc0CfuGzHp0pHNSAqdZY9TAzF5YKLc.:17531::0:99999:7:::
user3 => $6$RANDOM3RANDOM3RANDOM3RANDOM3:17531::0:99999:7:::
user3 => $6$P8.89SNhu23Bb2zWHx353t.e9ERSVy.:17531::0:99999:7:::
user4 => $6$RANDOM4RANDOM4RANDOM4RANDOM4:17531::0:99999:7:::
user4 => $6$lFOIUQv1CRG4T0kJQ3rzRNRSyQHjyb1:17531::0:99999:7:::
user5 => $6$RANDOM5RANDOM5RANDOM5RANDOM5:17531::0:99999:7:::
user5 => $6$RZbtYxWiwE$lnP8w64Cp6DgXheacSx.:17531::0:99999:7:::

Scripting the cisco banner with Net::Appliance::Session

Has anyone ran into this issue? When the script gets to the banner text the script just hangs.
I am using Net::Appliance::Session
Here is the error I get in debug. The rest of the script inserts code perfectly. I did test what I read about adding a # to the banner for each line. Same result.
banner login +
[ 4.092880] tr nope, doesn't (yet) match (?-xism:[\/a-zA-Z0-9._\[\]-]+ ?(?:\(config[^)]*\))? ?[#>] ?$)
[ 4.093124] du SEEN:
banner login +
[ 4.093304] tr nope, doesn't (yet) match (?-xism:[\/a-zA-Z0-9._\[\]-]+ ?(?:\(config[^)]*\))? ?[#>] ?$)
[ 4.305872] du SEEN:
Enter TEXT message. End with the character '+'
[ 4.306121] tr nope, doesn't (yet) match (?-xism:[\/a-zA-Z0-9._\[\]-]+ ?(?:\(config[^)]*\))? ?[#>] ?$)
We had an issue when accessing the device : 10.49.216.74
The reported error was : read timed-out at /usr/lib/perl5/site_perl/5.10.0/Net/CLI/Interact/Transport/Wrapper/Net_Telnet.pm line 35
Here is a snip of code.
my $session_obj = Net::Appliance::Session->new(
host => $ios_device_ip,
transport => 'Telnet',
personality => 'ios',
timeout => 60,
);
#interace
$session_obj->set_global_log_at('debug');
eval {
# try to login to the ios device, ignoring host check
$session_obj->connect(
username => $ios_username,
password => $ios_password,
#SHKC => 0
);
# get our running config
$version_info = $session_obj->begin_privileged;
$session_obj->cmd('conf t');
$session_obj->cmd('line con 0');
$session_obj->cmd('exec-character-bits 8');
$session_obj->cmd('international');
$session_obj->cmd('line vty 0 4');
$session_obj->cmd('exec-character-bits 8');
$session_obj->cmd('international');
$session_obj->cmd('line vty 5 15');
$session_obj->cmd('exec-character-bits 8');
$session_obj->cmd('international');
$session_obj->cmd('exit');
$session_obj->cmd('no banner login');
$session_obj->cmd('banner login +');
$session_obj->cmd('*************************************************************************');
$session_obj->cmd('* test *');
$session_obj->cmd('* *');
$session_obj->cmd('*************************************************************************');
$session_obj->cmd('+');
$session_obj->cmd('no banner MOTD');
$session_obj->cmd('banner motd +');
$session_obj->cmd('*************************************************************************');
$session_obj->cmd('* test *');
$session_obj->cmd('* *');
$session_obj->cmd('*************************************************************************');
$session_obj->cmd('+');
$session_obj->cmd('exit');
$session_obj->cmd('write memory');
$session_obj->end_privileged;
# close down our session
$session_obj->close;
};
If you look at the regexp that matches the prompt before sending a new command you'll see that it requires a specific string that closely matches user, privileged or config mode of a router.
When you send the banner login + command you get the Enter TEXT message. End with the character '+' followed by blank line from a router (instead of Router(config)# that your script expects. After a while it just times out since there is no match for the regexp.
The easiest solution is to try to send the whole banner in one command. Try concatenating your banner with a \r in one string and sending it as a one command that looks like (note the double quotes):
$session_obj->cmd("banner login + line1 \r line2 \r line3\r +");
Took way too long to figure this out... spaces are not your friend.
$session_obj->cmd("banner login + \rline1\rline2\rline3\r+");
Example with my orginal problem:
$session_obj->cmd('*************************************************************************\r* test *\r* *\r*************************************************************************');

Perl Regex issues

why isn't this perl REGEX working? i'm grabbing the date and username (date works fine), but it will grab all the usernames then when it hits bob.thomas and grabs the entire line
Code:
m/^(.+)\s-\sUser\s(.+)\s/;
print "$2------\n";
Sample Data:
Feb 17, 2013 12:18:02 AM - User plasma has logged on to client from host
Feb 17, 2013 12:13:00 AM - User technician has logged on to client from host
Feb 17, 2013 12:09:53 AM - User john.doe has logged on to client from host
Feb 17, 2013 12:07:28 AM - User terry has logged on to client from host
Feb 17, 2013 12:04:10 AM - User bob.thomas has been logged off from host because its web server session timed out. This means the web server has not received a request from the client in 3 minute(s). Possible causes: the client process was killed, the client process is hung, or a network problem is preventing access to the web server.
for the user that asked for the full code
open (FILE, "log") or die print "couldn't open file";
$record=0;
$first=1;
while (<FILE>)
{
if(m/(.+)\sto now/ && $first==1) # find the area to start recording
{
$record=1;
$first=0;
}
if($record==1)
{
m/^(.+)\s-\sUser\s(.+)\s/;
<STDIN>;
print "$2------\n";
if(!exists $user{$2})
{
$users{$2}=$1;
}
}
}
.+ is greedy, it matches the longest possible string. If you want it to match the shortest, use .+?:
/^(.+)\s-\sUser\s(.+?)\s/;
Or use a regexp that doesn't match whitespace:
/^(.+)\s-\sUser\s(\S+)/;
Use the reluctant/ungreedy quantifier to match up until the first occurrence rather than the last. You should do this in both cases just in case the "User" line also has " - User "
m/^(.+?)\s-\sUser\s(.+?)\s/;

show cdp neighbor in expect script

I'm pretty new to scripting in general. I'm writing an expect script that ssh'es into a Cisco switch, and runs the "show cdp neighbors" command to get a list of all the devices connected to the switch. I then save the output into a variable and exit the ssh session.
I have the username and password being set in the included file.
#!/usr/bin/expect -f
#exp_internal 1
source accountfile
set timeout 10
spawn $env(SHELL)
expect "#"
send "ssh $USERNAME#<hostname>\r"
expect {
"continue connecting" {
send_user "Adding host to ssh known hosts list...\n"
send "yes\n"
exp_continue
}
"Do you want to change the host key on disk" {
send_user "Changing host key on disk...\n"
send "yes\n"
exp_continue
}
"assword:" {
send "$PASSWORD\r"
}
}
expect "#"
send "term len 0\r"
expect "#"
send "show cdp neighbors\r"
expect "#"
set result $expect_out(buffer)
send "exit\r"
expect "#"
So then I want to take $result and look for lines that contain ' R ', and save those lines to a file (R with spaces on either side indicates a router, which is what I'm interested in)
The problem is that if the name of a connected device is long, it puts the name of the device on one line, and then the rest of the data about the device on the next line. So if I match the ' R ' string, I won't get the name of the device, since the name is on the previous line.
Device ID Local Intrfce Holdtme Capability Platform Port ID
...
<device_name_really_long>
Gig 2/0/52 171 R S I WS-C6509 Gig 3/14
<device_name2> Gig 2/0/1 131 H P M IP Phone Port 1
...
Any ideas? there's probably a regex that would do it, but I don't know squat about regex.
SOLVED: thanks to Glenn Jackman
I ended up having to add an expect condition to check if I had a full buffer, so my final code looks like this:
#!/usr/bin/expect
#exp_internal 1
match_max 10000
set expect_out(buffer) {}
set timeout 30
source accountfile
spawn $env(SHELL)
expect "#"
send "ssh $USERNAME#ol2110-3750stack.sw.network.local\r"
expect {
"continue connecting" {
send_user "Adding host to ssh known hosts list...\n"
send "yes\n"
exp_continue
}
"Do you want to change the host key on disk" {
send_user "Changing host key on disk...\n"
send "yes\n"
exp_continue
}
"assword:" {
send "$PASSWORD\r"
}
}
expect "#"
send "term len 0\r"
expect "#"
send "show cdp neighbors\r"
set result ""
expect {
{full_buffer} {
puts "====== FULL BUFFER ======"
append result $expect_out(buffer)
exp_continue
}
"#" {
append result $expect_out(buffer)
}
}
send "exit\r"
expect "#"
set devices [list]
set current_device ""
set lines [split $result "\n"]
foreach line $lines {
set line [string trim $line]
if {[llength $line] == 1} {
set current_device $line
continue
}
set line "$current_device$line\n"
if {[string match {* R *} $line]} {
lappend devices $line
}
set current_device ""
}
puts $devices
set devices [list]
set current_device ""
foreach line [split $result \n] {
if {[llength [split [string trim $line]]] == 1} {
set current_device $line
continue
}
set line "$current_device$line"
if {[string match {* R *} $line]} {
lappend devices $line
}
set current_device ""
}
# devices list should now contain "joined" routers.