I am writing a program that somewhat mimics the last command in UNIX, and I am trying to use backreferencing in my solution. My program does exactly what it is supposed to do but I get a run time error/warning. My question is why is this error/warning coming up and how can I fix an issue like this?
If you need more information I can provide.
Program Execution
./last dodoherty
OUTPUT
Here is a listing of the logins for dodoherty:
1. dodohert pts/1 pc-618-012.omhq. Wed Feb 8 09:19 still logged in
2. dodohert pts/6 ip98-168-203-118 Tue Feb 7 19:19 - 20:50 (01:31)
3. dodohert pts/3 137.48.207.178 Tue Feb 7 14:00 - 15:06 (01:05)
4. dodohert pts/1 137.48.219.250 Tue Feb 7 12:32 - 12:36 (00:04)
5. dodohert pts/21 137.48.207.237 Tue Feb 7 12:07 - 12:23 (00:16)
6. dodohert pts/11 ip98-168-203-118 Mon Feb 6 20:50 - 23:29 (02:39)
7. dodohert pts/9 ip98-168-203-118 Mon Feb 6 20:31 - 22:57 (02:26)
8. dodohert pts/5 pc-618-012.omhq. Fri Feb 3 10:24 - 10:30 (00:05)
Use of uninitialized value $1 in addition (+) at ./odoherty_last.pl line 43.
Use of uninitialized value $2 in addition (+) at ./odoherty_last.pl line 44.
Here is a summary of the time spent on the system for dodoherty:
dodoherty
8
8:6
The Code (Snippet of where the error is coming from, Also this is the only time $1 and $2 are used.)
foreach my $line2 (#user)
{
$line2 =~ /\S*\((\d{2,2})\:(\d{2,2})\)\s*/;
$hours = $hours + $1;
$mins = $mins + $2;
if( $mins >= 60 )
{
$hours = $hours + 1;
$mins = $mins - 60;
}
}
I think the problem might be in the following line.
1. dodohert pts/1 pc-618-012.omhq. Wed Feb 8 09:19 still logged in
That is because nothing matches the pattern so $1 and $2 are undefined.
As has been noted in other answers, your regex does not match, and therefore $1 and $2 are undefined. It is necessary to always check to make sure the appropriate regex matches before using these variables.
Below I have upgraded your script with some proper perl code. += and %= are handy operator in this case. You can read about them in perlop
Your regex uses \S* and \s*, both of which are completely unnecessary here, since your regex is not anchored to anything else. In other words, \S*foo\s* will match any string that contains foo, since it can match the empty string around foo. Also, {2,2} means "match at least 2 times, max 2", which in effect is the same as {2} "match 2 times".
You will see that I changed your math around, and that is because it assumes that $mins will never be higher than 120. I suppose technically, that is a safe assumption, but doing it like below, it can handle all values of minutes and successfully turn them into hours.
The script below is for demonstration. If you remove DATA and leave <>, you can use this script as-is like so:
last user | perl script.pl
Code:
use strict;
use warnings;
use v5.10; # required for say()
my ($hours, $mins);
while (<DATA>) { # replace with while (<>) for live usage
if (/\((\d{2})\:(\d{2})\)/) {
$hours += $1;
$mins += $2;
if( $mins >= 60 ) {
$hours += int ($mins / 60); # take integer part of division
$mins %= 60; # remove excess minutes
}
}
}
say "Hours: $hours";
say "Mins : $mins";
__DATA__
1. dodohert pts/1 pc-618-012.omhq. Wed Feb 8 09:19 still logged in
2. dodohert pts/6 ip98-168-203-118 Tue Feb 7 19:19 - 20:50 (01:31)
3. dodohert pts/3 137.48.207.178 Tue Feb 7 14:00 - 15:06 (01:05)
4. dodohert pts/1 137.48.219.250 Tue Feb 7 12:32 - 12:36 (00:04)
5. dodohert pts/21 137.48.207.237 Tue Feb 7 12:07 - 12:23 (00:16)
6. dodohert pts/11 ip98-168-203-118 Mon Feb 6 20:50 - 23:29 (02:39)
7. dodohert pts/9 ip98-168-203-118 Mon Feb 6 20:31 - 22:57 (02:26)
8. dodohert pts/5 pc-618-012.omhq. Fri Feb 3 10:24 - 10:30 (00:05)
#!/usr/bin/perl
use strict;
my $hours = 0;
my $mins = 0;
my $loggedIn = 0;
while (<STDIN>)
{
chomp;
if (/\S*\((\d{2,2})\:(\d{2,2})\)\s*/)
{
$hours = $hours + $1;
$mins = $mins + $2;
if($mins >= 60 )
{
$hours = $hours + 1;
$mins = $mins - 60;
}
}
elsif (/still logged in$/)
{
$loggedIn = 1;
}
}
print "Summary: $hours:$mins ", ($loggedIn) ? " (Currently logged in)" : "", "\n";
When ever your RE fails to match, $1 and $2 have no value.
For this reason, it's considered best practice on ever to use $1, $2 etc. inside a conditional which tests the success of the RE.
So don't do:
$string =~ m/(somepattern)/sx;
my $var = $1;
But instead to do something like:
my $var = 'some_default_value';
if($string =~ m/(somepattern)/sx){
$var = $1;
}
Related
so here's my problem: I have big log files and want a script to grep certain periods of time and safe them to a file (sorted), basically
bash script.sh Jul 4 Sep 30
will return for example
Sep 30 user0 logged in
Sep 15 user1 logged in
Aug 6 user0 logged in
Aug 3 user1 logged in
Jul 28 user2 logged in
Jul 27 user2 logged in
Jul 4 user0 logged in
My first attempt was that every month and date gets his own variable like
bash script.sh Jul 4 Sep 3 0
so I can use $1 for start month (July), $2 for start date (4) and so on in grep like
for logs in logs*
do
grep -qEe "^\"$1\" [\"$2\"-9]\s" $messages >> result.txt
done
to get all logs from July 4 to 9 but I don't know how to get logs from the entire time period that aren't in the same month nor in a period like 1-9 or 10-19 and so on
Any help greatly appreciated!
EDIT:
As some people asked, here's how my log files look like (just much bigger and not sorted):
Sep 30 user0 logged in
Jul 27 user2 logged in
Aug 6 user0 logged in
Aug 31 user1 logged in
Jul 8 user2 logged in
Sep 5 user1 logged in
Jul 27 user2 logged in
Jul 14 user0 logged in
[...]
Here's my take:
#/bin/bash
year="$(date +"%Y")"
start="$(date -d"$1 $2, $year" +'%s')"
end="$(($(date -d"$3 $4, $year" +'%s')+86400))"
for log in logs*; do
while IFS= read -r line; do
d="$(date -d"$(cut -d' ' -f1,2 <<< "$line"), $year" +'%s')"
if (( $start <= $d && $d < $end )); then
echo "$s"
fi
done < "$log"
done
You run it like that: ./script.sh Jul 04 Sep 03. Since no year is included in the logs, it assumes that all dates (including the ones in the command line) are for the current year. It's probably not the most optimal solution but it works. It relies on date which it repeatedly calls to parse dates into a unix timestamp. unix timestamps are nice because they are just numbers and thus can be used in numeric comparisons.
$ range="Jul 4 Sep 30"
$ awk -v range="$range" '
BEGIN {
numMths = split("Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec",m)
for (i in m) {
mths[m[i]] = i
}
split(range,r)
beg = sprintf("%02d%02d", mths[r[1]], r[2])
end = sprintf("%02d%02d", mths[r[3]], r[4])
}
{ cur = sprintf("%02d%02d", mths[$1], $2) }
(cur >= beg) && (cur <= end) { vals[$1,$2] = $0 }
END {
for (mthNr=numMths; mthNr>0; mthNr--) {
for (dayNr=31; dayNr>0; dayNr--) {
date = m[mthNr] SUBSEP dayNr
if (date in vals) {
print vals[date]
}
}
}
}
' file
Sep 30 user0 logged in
Sep 5 user1 logged in
Aug 31 user1 logged in
Aug 6 user0 logged in
Jul 27 user2 logged in
Jul 14 user0 logged in
Jul 8 user2 logged in
This question already has answers here:
Extracting specific lines with Perl
(4 answers)
Closed 8 years ago.
I'm trying to parse out data from a log file spanning over multiple lines (shown below).
Archiver Started: Fri May 16 00:35:00 2014
Daily Archive for (Thu) May. 15, 2014 STATUS: Successful Fri May 16 00:37:43 2014
Daily Archive for (Thu) May. 15, 2014 STATUS: Successful Fri May 16 00:39:54 2014
Archiver Completed: Fri May 16 00:42:37 2014
I want to split on Archiver Started: on the first line and split on Archiver Completed: on the last line for anything in between these lines. So I would be left with the following:
Daily Archive for (Thu) May. 15, 2014 STATUS: Successful Fri May 16 00:37:43 2014
Daily Archive for (Thu) May. 15, 2014 STATUS: Successful Fri May 16 00:39:54 2014
As sometimes the there can be a single or multiple entry for one day, week or month.
Is this possible with a Regex?
Use a Range Operator ...
The return value of a flipflop is a sequence number (starting with 1), so you simply need to filter out 1 and the ending number which has the the string "E0" appended to it.
use strict;
use warnings;
while (<DATA>) {
if (my $range = /Archiver Started/ .. /Archiver Completed/ ) {
print if $range != 1 && $range !~ /E/;
}
}
__DATA__
stuff
more stuff
Archiver Started: Fri May 16 00:35:00 2014
Daily Archive for (Thu) May. 15, 2014 STATUS: Successful Fri May 16 00:37:43 2014
Daily Archive for (Thu) May. 15, 2014 STATUS: Successful Fri May 16 00:39:54 2014
Archiver Completed: Fri May 16 00:42:37 2014
other stuff
ending stuff
Outputs:
Daily Archive for (Thu) May. 15, 2014 STATUS: Successful Fri May 16 00:37:43 2014
Daily Archive for (Thu) May. 15, 2014 STATUS: Successful Fri May 16 00:39:54 2014
you can use next trick:
my #result = ();
my $catch;
LINE:
for my $line ( #lines ) {
if ( $line =~ m/^Archiver Started/i ) {
$catch = 1;
next LINE;
} elsif ( $line =~ m/^Archiver Completed/i ) {
$catch = 0;
next LINE;
}
next LINE unless $catch;
push #result, $line;
}
Consider this log file
SN PID Date Status
1 P01 Fri Feb 14 19:32:36 IST 2014 Alive
2 P02 Fri Feb 14 19:32:36 IST 2014 Alive
3 P03 Fri Feb 14 19:32:36 IST 2014 Alive
4 P04 Fri Feb 14 19:32:36 IST 2014 Alive
5 P05 Fri Feb 14 19:32:36 IST 2014 Alive
6 P06 Fri Feb 14 19:32:36 IST 2014 Alive
7 P07 Fri Feb 14 19:32:36 IST 2014 Alive
8 P08 Fri Feb 14 19:32:36 IST 2014 Alive
9 P09 Fri Feb 14 19:32:36 IST 2014 Alive
10 P010 Fri Feb 14 19:32:36 IST 2014 Alive
When i do => grep "P01" File
output is : (as expected)
1 P01 Fri Feb 14 19:32:36 IST 2014 Alive
10 P010 Fri Feb 14 19:32:36 IST 2014 Alive
But when i do => grep " P01 " File (notice the space before and after P01)
I do not get any output!
Question : grep matches pattern in a line, so " P01 " ( with space around ) should match the first PID of P01 as it has spaces around it....but seems that this logic is wrong....what obvious thing i am missing here!!!?
If the log uses tabs not spaces, your grep pattern won't match. I would add word boundaries to the word you want to find:
grep '\<P01\>' file
If you really want to use whitespace in your pattern, use one of:
grep '[[:blank:]]P01[[:blank:]]' file # horizontal whitespace, tabs and spaces
grep -P '\sP01\s' file # using Perl regex
In my MVC3 application, on one of the entities I am saving the Date of Birth as a string. Why ? because my application allows the storage of the date of birth of people long dead, eg. Socrates, Plato, Epicurus ... etc and as far as I know the DateTime class doesn't allow that.
Now obviously we don't know the exact date of birth of Epicurus for example, we just know the year of birth [ 341 BCE ], so what I am thinking of doing is building a custom validator, that will validate the input string for the Date of Birth and make sure that they all match the following format:
12 Feb 1809
Feb 1809
341
341 BCE
Oct 341 BCE
11 Mar 5 BCE
I need to write a regular expression that will match any of the above, and of course not match anything else.
Update
Thank you very much, I wish I was as good as you lot in building RegExes ! Since my application is with ASP.net MVC3, I would like to stick with the .NET RegEx class (for convenience's sake).
luastoned answer seems to work; I can't seem to break its logic with all the test data I've thrown at it.
One thing though, can I also allow BC? Because some people use BC and others use BCE < would that be possible? And, am I right that the regular expression can not replace BC with BCE? I have to do that manually through my C# code - the RegEx would just either match or not, is that correct?
Update 2
M42's Regular Expression seems to be working better. I've just copied it and used it in my Custom Validator (code in PasteBin link below).
How about :
/^(?:\d+\s)?(Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec)?(?:\s?\d+)(?:\sBCE)?$/
Here is a perl script with test cases:
#!/usr/local/bin/perl
use strict;
use warnings;
use Test::More;
my $re1 = qr/^(?:\d+\s)?(Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec)?(?:\s?\d+)(?:\sBCE)?$/;
while(<DATA>) {
chomp;
if (/$re1/) {
ok(1, "input = $_");
} else {
ok(0, "input = $_");
}
}
done_testing;
__DATA__
12 Feb 1809
Feb 1809
341
341 BCE
Oct 341 BCE
11 Mar 5 BCE
12D09
1s909
A3 43 4 BCE
a 1
3F9
abc
BCE
123b456
output:
# Looks like you failed 9 tests of 15.
ok 1 - input = 12 Feb 1809
ok 2 - input = Feb 1809
ok 3 - input = 341
ok 4 - input = 341 BCE
ok 5 - input = Oct 341 BCE
ok 6 - input = 11 Mar 5 BCE
not ok 7 - input = 12D09
not ok 8 - input = 1s909
not ok 9 - input = A3 43 4 BCE
not ok 10 - input = a 1
not ok 11 - input = 3F9
not ok 12 - input =
not ok 13 - input = abc
not ok 14 - input = BCE
not ok 15 - input = 123b456
this looks likes the weirdest Regex I've ever made:
(\d+\s?)?(Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec)?\s(\d+\s)?(BCE)?
I have no idea how many false positives would go through though..
You can check the sample on Regexr
Not quite a friend of RegExr (and not knowing the limitations of regexes in MVC3), allow me to present a PHP version with named captures (demo):
(?:(?:(?P<date>\d{1,2})\s)?(?P<month>Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec))?(?:(?:^|\s)(?P<year>\d+))?(?:\s(?P<bce>BCE))?
This is based on #luastoned's answer.
I have a working perl script that grabs the data I need and displays them to STDOUT, but now I need to change it to generate a data file (csv, tab dellimited, any delimiter file).
The regular expression is filtering the data that I need, but I don't want the entire string, just snippets of the output. I'm assuming I would need to store this in another variable to create my output file.
I need a good example of this or suggestions to alter this code. Thank you in advance. :-)
Here's my code:
#!/usr/bin/perl -w
# Usage: ./bakstatinfo.pl Jul 28 2010 /var/log/mybackup.log <server1> <server2>
use strict;
use warnings;
#This piece added to view the arguments passed in
$" = "][";
print "===================================================================================\n";
print "[#ARGV]\n";
#Declare Variables
my($mon,$day,$year,$file,$server) = #ARGV;
my $regex_flag = 0;
splice(#ARGV, 0, 4, ());
foreach my $server ( #ARGV ) { #foreach will take Xn of server entries and add to the loop
print "===================================================================================\n";
print "REPORTING SUMMARY for SERVER : $server\n";
open(my $fh,"ssh $server cat $file |") or die "can't open log $server:$file: $!\n";
while (my $line = <$fh>) {
if ($line =~ m/.* $mon $day \d{2}:\d{2}:\d{2} $year:.*(ERROR:|backup-date=|backup-size=|backup-time=|backup-status)/) {
print $line;
$regex_flag=1; #Set to true
}
}
if ($regex_flag==0) {
print "NOTHING TO REPORT FOR $server: $mon $day $year \n";
}
$regex_flag=0;
close($fh);
}
Sample raw log file I am using: (recently added to provide better representation of log)
Tue Jul 27 23:00:06 2010: test202.bak_lvm:backup:ERROR: mybak-abc appears to be already running for this backupset
Tue Jul 27 23:00:06 2010: test202.bak_lvm:backup:ERROR: If you are sure mybak-abc is not running, please remove the file /etc/mybak-abc/test202.bak_lvm/.mybak-abc.pid and restart mybak-abc
Tue Jul 27 23:00:06 2010: test202.bak_lvm:backup:INFO: PHASE START: Cleanup
Tue Jul 27 23:00:06 2010: test202.bak_lvm:backup:INFO: PHASE END: Cleanup
Tue Jul 27 23:00:06 2010: test202.bak_lvm:backup:INFO: END OF BACKUP
Wed Jul 28 00:00:04 2010: db9.abc.bak:backup:INFO: START OF BACKUP
Wed Jul 28 00:00:04 2010: db9.abc.bak:backup:INFO: PHASE START: Initialization
Wed Jul 28 00:00:05 2010: db9.abc.bak:backup:WARNING: Binary logging is off.
Wed Jul 28 00:00:05 2010: db9.abc.bak:backup:INFO: License check successful
Wed Jul 28 00:00:05 2010: db9.abc.bak:backup:INFO: License check successful for lvm-snapshot.pl
Wed Jul 28 00:00:05 2010: db9.abc.bak:backup:INFO: backup-set=db9.abc.bak
Wed Jul 28 00:00:05 2010: db9.abc.bak:backup:INFO: backup-date=20100728000004
Wed Jul 28 00:00:05 2010: db9.abc.bak:backup:INFO: SQL-server-os=Linux/Unix
Wed Jul 28 00:00:05 2010: db9.abc.bak:backup:INFO: backup-type=regular
Wed Jul 28 00:00:05 2010: db9.abc.bak:backup:INFO: host=db9.abc.bak.test.com
Wed Jul 28 00:00:05 2010: db9.abc.bak:backup:INFO: backup-date-epoch=1280300404
Wed Jul 28 00:00:05 2010: db9.abc.bak:backup:INFO: retention-policy=3D
Wed Jul 28 00:00:05 2010: db9.abc.bak:backup:INFO: mybak-abc-version=ABC for SQL Enterprise Edition - version 3.1
Wed Jul 28 00:00:05 2010: db9.abc.bak:backup:INFO: SQL-version=5.1.32-test-SMP-log
Wed Jul 28 00:00:05 2010: db9.abc.bak:backup:INFO: backup-directory=/home/backups/db9.abc.bak/20100728000004
Wed Jul 28 00:00:05 2010: db9.abc.bak:backup:INFO: backup-level=0
Wed Jul 28 00:00:05 2010: db9.abc.bak:backup:INFO: backup-mode=raw
Wed Jul 28 00:00:05 2010: db9.abc.bak:backup:INFO: PHASE END: Initialization
Wed Jul 28 00:00:05 2010: db9.abc.bak:backup:INFO: PHASE START: Running pre backup plugin
Wed Jul 28 00:00:05 2010: db9.abc.bak:backup:INFO: PHASE START: Flushing logs
Wed Jul 28 00:00:05 2010: db9.abc.bak:backup:INFO: PHASE END: Flushing logs
Wed Jul 28 00:00:05 2010: db9.abc.bak:backup:INFO: PHASE START: Creating snapshot based backup
Wed Jul 28 00:00:11 2010: db9.abc.bak:backup:INFO: Wed Jul 28 00:49:53 2010: test203.bak_lvm:backup:INFO: raw-databases-snapshot=test SQL sgl
Wed Jul 28 00:49:53 2010: test203.bak_lvm:backup:INFO: PHASE END: Creating snapshot based backup
Wed Jul 28 00:49:53 2010: test203.bak_lvm:backup:INFO: PHASE START: Calculating backup size & checksums
Wed Jul 28 00:49:54 2010: test203.bak_lvm:backup:INFO: last-backup=/home/backups/test203.bak_lvm/20100726200004
Wed Jul 28 00:49:54 2010: test203.bak_lvm:backup:INFO: backup-size=417.32 GB
Wed Jul 28 00:49:54 2010: test203.bak_lvm:backup:INFO: PHASE END: Calculating backup size & checksums
Wed Jul 28 00:49:54 2010: test203.bak_lvm:backup:INFO: read-locks-time=00:00:05
Wed Jul 28 00:49:54 2010: test203.bak_lvm:backup:INFO: flush-logs-time=00:00:00
Wed Jul 28 00:49:54 2010: test203.bak_lvm:backup:INFO: backup-time=04:49:51
Wed Jul 28 00:49:54 2010: test203.bak_lvm:backup:INFO: backup-status=Backup succeeded
My working output now:
===================================================================================
[Jul][28][2010][/var/log/mybackup.log][server1]
===================================================================================
REPORTING SUMMARY for SERVER : server1
Wed Jul 28 00:49:54 2010: test203.bak_lvm:backup:INFO: backup-size=417.32 GB
Wed Jul 28 00:49:54 2010: test203.bak_lvm:backup:INFO: backup-time=04:49:51
Wed Jul 28 00:49:54 2010: test203.bak_lvm:backup:INFO: backup-status=Backup succeeded
The output I need to see would be something like this:(data file with separated by ';' for example)
MyDate=Wed Jul 28;MyBackupSet= test203.bak_lvm;MyBackupSize=187.24 GB;MyBackupTime=04:49:51;MyBackupStat=Backup succeeded
Use 'capturing parentheses' to identify the bits you want to deal with.
if ($line =~ m/(.* $mon $day) \d{2}:\d{2}:\d{2} $year:.*
(ERROR:|backup-date=|backup-size=|
backup-time=|backup-status)/x) {
You will need to do some surgery on the second set of parentheses - those surrounding the start of the various keywords. You may have to chop those out in bits and pieces inside the condition.
When you have all the data extracted into variables, use Text::CSV to handle CSV output (and input).
There are a myriad modules to handle HTML or XML (over 2000, and I think over 3000, with HTML in their name - I happened to look yesterday). Many of those won't be applicable, but CPAN is your friend.
Answering questions posed by comments
Would I split them off into separate variables as well? The first part gives me the date/time that I need. The next filter then gives me 1) Error: 2)backup-date= 3)backup-size= ...etc.
More or less. Unfortunately, you don't show some representative input lines, which means it is hard to tell what might be best. However, it seems likely that a scheme such as:
while (my $line = <$fh>)
{
chomp $line;
if ($line =~ m/(.* $mon $day) \d\d:\d\d:\d\d $year:/)
{
my $date = $1;
my %items = ();
$line =~ s/.* $mon $day \d\d:\d\d:\d\d $year://;
while ($line =~ m/(ERROR|backup-date|backup-size|
backup-time|backup-status)
[:=]([^:]+)/x)
{
my $key = $1;
my $val = $2;
$items{$key} = $val;
$line =~ s/$key[:=]$val[:=]?//;
}
# The %items hash contains the split out information.
# Now write the data for this line of the log file.
}
}
There might well be better ways to handle the trimming (but it is Perl so TMTOWTDI), but the basic idea here is to catch the lines that are interesting, then progressively chop the bits of interest out of the line, so the line grows shorter on each iteration (therefore, eventually terminating the inner while loop).
Note the use of the /x modifier to allow for a more readable regex split over lines (I edited the original answer version to use that too). I've also allowed 'ERROR' to be followed by an '=' or the other keywords to be followed by ':'; it seems unlikely that you'd get false matches that way, and it simplifies the regex substitute operations. The initial pattern match no longer requires one of the subsections to be present, either. You must judge for yourself whether those small changes (which might pick up non-conforming information) matter or not. For most of my purposes, the chance of the mismatch is small enough not to be an issue - but for legal reasons, it might not be acceptable to you.
Answering questions posed by 'answer'
I manufactured some data:
Wed Jul 30 00:49:51 2010: test203.bak_lvm:backup:INFO: backup-size=417.32 GB
Wed Jul 30 00:49:52 2010: test203.bak_lvm:backup:INFO: backup-time=04:49:51
Wed Jul 30 00:49:53 2010: test203.bak_lvm:backup:INFO: backup-status=Backup succeeded
Wed Jul 30 00:49:51 2010: backup-size=417.32 GB:backup-time=04:49:51:backup-status=Backup succeeded
I took the script in the answer and hacked and instrumented it - making it standalone.
I also removed the dependency on specific files - it reads standard input and writes to standard output. It makes my testing easier - and the code more flexible.
use strict;
use warnings;
use constant debug => 0;
my $mon = 'Jul';
my $day = 30;
my $year = 2010;
while (my $line = <>)
{
chomp $line;
print "Line: $line\n" if debug;
if ($line =~ m/(.* $mon $day) \d\d:\d\d:\d\d $year:/) #Mon Jul 26 22:00:02 2010:
{
print "### Scan\n";
my $date = $1;
print "$date\n";
my %items = ();
$line =~ s/.* $mon $day \d\d:\d\d:\d\d $year://;
print "Line: $line\n" if debug;
while ($line =~ m/(ERROR|backup-date|backup-size|backup-time|backup-status)[:=]([^:]+)/)
{
my $key = $1;
my $val = $2;
$items{$key} = $val;
$line =~ s/$key[:=]$val[:=]?//;
print "$key=$val\n";
print "Line: $line\n" if debug;
}
print "### Verify\n";
for my $key (sort keys %items)
{
print "$key = $items{$key}\n";
}
}
}
The output I get is:
### Scan
Wed Jul 30
backup-size=417.32 GB
### Verify
backup-size = 417.32 GB
### Scan
Wed Jul 30
backup-time=04
### Verify
backup-time = 04
### Scan
Wed Jul 30
backup-status=Backup succeeded
### Verify
backup-status = Backup succeeded
### Scan
Wed Jul 30
backup-size=417.32 GB
backup-time=04
backup-status=Backup succeeded
### Verify
backup-size = 417.32 GB
backup-status = Backup succeeded
backup-time = 04
The verify loop prints out the data from the '%items' hash quite happily. With the debug value set to 1 instead of 0, the output I get is:
Line: Wed Jul 30 00:49:51 2010: test203.bak_lvm:backup:INFO: backup-size=417.32 GB
### Scan
Wed Jul 30
Line: test203.bak_lvm:backup:INFO: backup-size=417.32 GB
backup-size=417.32 GB
Line: test203.bak_lvm:backup:INFO:
### Verify
backup-size = 417.32 GB
Line: Wed Jul 30 00:49:52 2010: test203.bak_lvm:backup:INFO: backup-time=04:49:51
### Scan
Wed Jul 30
Line: test203.bak_lvm:backup:INFO: backup-time=04:49:51
backup-time=04
Line: test203.bak_lvm:backup:INFO: 49:51
### Verify
backup-time = 04
Line: Wed Jul 30 00:49:53 2010: test203.bak_lvm:backup:INFO: backup-status=Backup succeeded
### Scan
Wed Jul 30
Line: test203.bak_lvm:backup:INFO: backup-status=Backup succeeded
backup-status=Backup succeeded
Line: test203.bak_lvm:backup:INFO:
### Verify
backup-status = Backup succeeded
Line: Wed Jul 30 00:49:51 2010: backup-size=417.32 GB:backup-time=04:49:51:backup-status=Backup succeeded
### Scan
Wed Jul 30
Line: backup-size=417.32 GB:backup-time=04:49:51:backup-status=Backup succeeded
backup-size=417.32 GB
Line: backup-time=04:49:51:backup-status=Backup succeeded
backup-time=04
Line: 49:51:backup-status=Backup succeeded
backup-status=Backup succeeded
Line: 49:51:
### Verify
backup-size = 417.32 GB
backup-status = Backup succeeded
backup-time = 04
The substitute operations delete the previously matched part of the line. There are ways of continuing a match where you left off - see \G at the 'perlre' page.
Note that the regex is crafted to stop at the first colon after the 'colon or equals' after the keyword. That means it truncates the backup time. One moral is "do not use a separator that can appear in the data". Another is "provide sample data so people can help you more easily". Another is "provide complete but minimal working scripts where possible".
Processing the sample data
Now that we have the sample input data, we can see that you need slightly different processing. This script:
use strict;
use warnings;
use constant debug => 0;
my $mon = 'Jul';
my $day = 28;
my $year = 2010;
my %items = ();
while (my $line = <>)
{
chomp $line;
print "Line: $line\n" if debug;
if ($line =~ m/(.* $mon $day) \d\d:\d\d:\d\d $year: ([^:]+):backup:/) #Mon Jul 26 22:00:02 2010:
{
print "### Scan\n" if debug;
my $date = $1;
my $set = $2;
print "$date ($set): " if debug;
$items{$set}->{'a-logdate'} = $date;
$items{$set}->{'a-dataset'} = $set;
if ($line =~ m/(ERROR|backup-date|backup-size|backup-time|backup-status)[:=](.+)/)
{
my $key = $1;
my $val = $2;
$items{$set}->{$key} = $val;
print "$key=$val\n" if debug;
}
}
}
print "### Verify\n";
for my $set (sort keys %items)
{
print "Set: $set\n";
my %info = %{$items{$set}};
for my $key (sort keys %info)
{
printf "%s=%s;", $key, $info{$key};
}
print "\n";
}
produces this result on the sample data file.
### Verify
Set: db9.abc.bak
a-dataset=db9.abc.bak;a-logdate=Wed Jul 28;backup-date=20100728000004;
Set: test203.bak_lvm
a-dataset=test203.bak_lvm;a-logdate=Wed Jul 28;backup-size=417.32 GB;backup-status=Backup succeeded;backup-time=04:49:51;
Note that now we have sample data, we can see that there is only one key/value pair per line, but there are multiple systems backed up per day. So, the inner while loop becomes a simple if. The printing out occurs at the end. And I'm using a 'two-tier' hash. The %items contains an entry for each data set; the entry, though, is a reference to a hash. Not necessarily something for novices to play with, but it fell into place very naturally with the previous code. Note, too, that this version doesn't hack the line - there's no need since there's only one lot of data per line.
Can it be improved - yes, undoubtedly. Does it work? Yes, more or less... Can it be hacked into shape? Yes, it can be hacked to work as you need.
#Jonathan- I wrote out the text file within the while loop. It seems to work. I tried doing it after the second while loop as you suggested in your comment. I'm not sure why it didn't work.
open (my $MYDATAFILE, ">/home/test/myout.txt") || die "cannot append $!";
open(my $fh,"ssh $server cat $file |") or die "can't open log $server:$file: $!\n";
while (my $line = <$fh>)
{
chomp $line;
if ($line =~ m/(.* $mon $day) \d\d:\d\d:\d\d $year:/) #Mon Jul 26 22:00:02 2010:
{
my $date = $1;
#print $date;
my %items = ();
$line =~ s/.* $mon $day \d\d:\d\d:\d\d $year://;
while ($line =~ m/(ERROR|backup-date|backup-size|backup-time|backup-status)[:=]([^:]+)/)
{
my $key = $1;
my $val = $2;
$items{$key} = $val;
$line =~ s/$key[:=]$val[:=]?//;
#print "[$key]";
#print "[$val]";
print $MYDATAFILE "$key=$val";
}
# The %items hash contains the split out information.
# Now write the data for this line of the log file.
}
}