This question already has answers here:
Extracting specific lines with Perl
(4 answers)
Closed 8 years ago.
I'm trying to parse out data from a log file spanning over multiple lines (shown below).
Archiver Started: Fri May 16 00:35:00 2014
Daily Archive for (Thu) May. 15, 2014 STATUS: Successful Fri May 16 00:37:43 2014
Daily Archive for (Thu) May. 15, 2014 STATUS: Successful Fri May 16 00:39:54 2014
Archiver Completed: Fri May 16 00:42:37 2014
I want to split on Archiver Started: on the first line and split on Archiver Completed: on the last line for anything in between these lines. So I would be left with the following:
Daily Archive for (Thu) May. 15, 2014 STATUS: Successful Fri May 16 00:37:43 2014
Daily Archive for (Thu) May. 15, 2014 STATUS: Successful Fri May 16 00:39:54 2014
As sometimes the there can be a single or multiple entry for one day, week or month.
Is this possible with a Regex?
Use a Range Operator ...
The return value of a flipflop is a sequence number (starting with 1), so you simply need to filter out 1 and the ending number which has the the string "E0" appended to it.
use strict;
use warnings;
while (<DATA>) {
if (my $range = /Archiver Started/ .. /Archiver Completed/ ) {
print if $range != 1 && $range !~ /E/;
}
}
__DATA__
stuff
more stuff
Archiver Started: Fri May 16 00:35:00 2014
Daily Archive for (Thu) May. 15, 2014 STATUS: Successful Fri May 16 00:37:43 2014
Daily Archive for (Thu) May. 15, 2014 STATUS: Successful Fri May 16 00:39:54 2014
Archiver Completed: Fri May 16 00:42:37 2014
other stuff
ending stuff
Outputs:
Daily Archive for (Thu) May. 15, 2014 STATUS: Successful Fri May 16 00:37:43 2014
Daily Archive for (Thu) May. 15, 2014 STATUS: Successful Fri May 16 00:39:54 2014
you can use next trick:
my #result = ();
my $catch;
LINE:
for my $line ( #lines ) {
if ( $line =~ m/^Archiver Started/i ) {
$catch = 1;
next LINE;
} elsif ( $line =~ m/^Archiver Completed/i ) {
$catch = 0;
next LINE;
}
next LINE unless $catch;
push #result, $line;
}
Related
I am trying to extract a part of a string as date-timestamp.
Example string:
Upgrade starting on Mon Aug 9 06:46:00 UTC 2021 with ...
Extracted values should be:
Mon Aug 9 06:46:00 UTC 2021
I tried applying the following regex to extract the timestamp:
(\d{2}:\d{2}:\d{2})
How can I extract the day month and year as well.
Use regex to extract part of string from raw string, the following is the whole code
package main
import (
"fmt"
"regexp"
)
func main() {
// extract part of string using regex
str := "Upgrade starting on Mon Aug 9 06:46:00 UTC 2021 with ..."
// extract string "Mon Aug 9 06:46:00 UTC 2021" using regex
re := regexp.MustCompile(`(Mon|Tue|Wed|Thu|Fri|Sat|Sun) (Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec) \d{1,2} \d{2}:\d{2}:\d{2} (\S{3}) \d{4}`)
t := re.FindString(str)
fmt.Println(t)
}
Consider this log file
SN PID Date Status
1 P01 Fri Feb 14 19:32:36 IST 2014 Alive
2 P02 Fri Feb 14 19:32:36 IST 2014 Alive
3 P03 Fri Feb 14 19:32:36 IST 2014 Alive
4 P04 Fri Feb 14 19:32:36 IST 2014 Alive
5 P05 Fri Feb 14 19:32:36 IST 2014 Alive
6 P06 Fri Feb 14 19:32:36 IST 2014 Alive
7 P07 Fri Feb 14 19:32:36 IST 2014 Alive
8 P08 Fri Feb 14 19:32:36 IST 2014 Alive
9 P09 Fri Feb 14 19:32:36 IST 2014 Alive
10 P010 Fri Feb 14 19:32:36 IST 2014 Alive
When i do => grep "P01" File
output is : (as expected)
1 P01 Fri Feb 14 19:32:36 IST 2014 Alive
10 P010 Fri Feb 14 19:32:36 IST 2014 Alive
But when i do => grep " P01 " File (notice the space before and after P01)
I do not get any output!
Question : grep matches pattern in a line, so " P01 " ( with space around ) should match the first PID of P01 as it has spaces around it....but seems that this logic is wrong....what obvious thing i am missing here!!!?
If the log uses tabs not spaces, your grep pattern won't match. I would add word boundaries to the word you want to find:
grep '\<P01\>' file
If you really want to use whitespace in your pattern, use one of:
grep '[[:blank:]]P01[[:blank:]]' file # horizontal whitespace, tabs and spaces
grep -P '\sP01\s' file # using Perl regex
I have a multi-threaded application which generates logs as mentioned:
D Fri Feb 01 00:21:23 2013 <no machine> pin_deferred_act:10233 pin_mta_conf.c:636 1:App-BRM-Prod-Pri.acttv.in:pin_deferred_act:10233:1:0:1359658283:1
pin_mta_convert_cmdline_options_to_flist parameters flist
D Fri Feb 01 00:21:23 2013 App-BRM-Prod-Pri.acttv.in pin_deferred_act:10233 pcpst.c(78):406 1:App-BRM-Prod-Pri.acttv.in:pin_deferred_act:10233:2:0:1359658283:0
connect to host=172.16.87.14, port=11962 OK
D Fri Feb 01 00:21:24 2013 App-BRM-Prod-Pri.acttv.in pin_deferred_act:10233 pin_mta.c:2479 1:App-BRM-Prod-Pri.acttv.in:pin_deferred_act:10233:1:0:1359658283:0
Config object search input flist
0 PIN_FLD_POID POID [0] 0.0.0.1 /search/pin -1 0
0 PIN_FLD_FLAGS INT [0] 0
0 PIN_FLD_TEMPLATE STR [0] "select X from /config/mta where F1 = V1 "
0 PIN_FLD_ARGS ARRAY [1] allocated 20, used 1
1 PIN_FLD_CONFIG_MTA ARRAY [0] allocated 20, used 1
2 PIN_FLD_NAME STR [0] "pin_deferred_act"
0 PIN_FLD_RESULTS ARRAY [0] allocated 20, used 1
1 PIN_FLD_POID POID [0] NULL poid pointer
D Fri Feb 01 00:21:24 2013 App-BRM-Prod-Pri.acttv.in pin_deferred_act:10233 pin_mta.c:2484 1:App-BRM-Prod-Pri.acttv.in:pin_deferred_act:10233:3:7:1359658284:2
Config object search output flist
0 PIN_FLD_POID POID [0] 0.0.0.1 /search/pin -1 0
D Fri Feb 01 00:21:24 2013 App-BRM-Prod-Pri.acttv.in pin_deferred_act:10233 pin_mta.c:3138 1:App-BRM-Prod-Pri.acttv.in:pin_deferred_act:10233:1:7:1359658284:2
So the threads update the logs, like pin_deferred_act:10233:1:7 --> where 1 specifies the log from the first thread, in the logfile.
I want to create log file for each thread, where the start point should be:
1:App-BRM-Prod-Pri.acttv.in:pin_deferred_act:10233:1:
and end- point should be:
D Fri Feb 01 00:21:24 2013 App-BRM-Prod-Pri.acttv.in
(where date/timestamp will keep on modifying).
All the instances should go in one file.
For e.g.:
D Fri Feb 01 00:21:23 2013 <no machine> pin_deferred_act:10233 pin_mta_conf.c:636 1:App-BRM-Prod-Pri.acttv.in:pin_deferred_act:10233:1:0:1359658283:1
pin_mta_convert_cmdline_options_to_flist parameters flist
D Fri Feb 01 00:21:23 2013 App-BRM-Prod-Pri.acttv.in pin_deferred_act:10233 pcpst.c(78):406 1:App-BRM-Prod-Pri.acttv.in:pin_deferred_act:10233:2:0:1359658283:0
connect to host=172.16.87.14, port=11962 OK
D Fri Feb 01 00:21:24 2013 App-BRM-Prod-Pri.acttv.in pin_deferred_act:10233 pin_mta.c:2479 1:App-BRM-Prod-Pri.acttv.in:pin_deferred_act:10233:1:0:1359658283:0
Config object search input flist
0 PIN_FLD_POID POID [0] 0.0.0.1 /search/pin -1 0
0 PIN_FLD_FLAGS INT [0] 0
0 PIN_FLD_TEMPLATE STR [0] "select X from /config/mta where F1 = V1 "
0 PIN_FLD_ARGS ARRAY [1] allocated 20, used 1
1 PIN_FLD_CONFIG_MTA ARRAY [0] allocated 20, used 1
2 PIN_FLD_NAME STR [0] "pin_deferred_act"
0 PIN_FLD_RESULTS ARRAY [0] allocated 20, used 1
1 PIN_FLD_POID POID [0] NULL poid pointer
D Fri Feb 01 00:21:24 2013 App-BRM-Prod-Pri.acttv.in pin_deferred_act:10233 pin_mta.c:3138 1:App-BRM-Prod-Pri.acttv.in:pin_deferred_act:10233:1:7:1359658284:2
should go to one file - Thread1.log, and similarly for other threads, the file should be created as Threadn.log with the respectively.
Files are a messy, non-scalable way of handling logs to begin with. A better approach is to handle logs as streams of log entry messages connected source(s) -> sink(s). Consider syslog, logplex or similar if Oracle provides alternative means of data collection. Custom re-implementation might be feasible depending on logging IOPS bottlenecks or other factors.
Use of high resolution monotonic clocks and/or globally-ordered GUID timestamps are highly recommended. With wall time, be sure to use non-backwards compensated UTC everywhere synced to low strata time sources.
Above recommendations may vary according to needs of the application of course, so experiment and implement wisely.
I think Barry's advice is useful, but in the event that you can't alter the application's log output, here is a quick Perl solution:
#!usr/bin/perl
use strict;
use warnings;
my %logs;
my $last_log;
while (<$main_log_file>) #open that application's log in this variable.
{
if (/pin_deferred_act:\d+:(\d+):\d/)
{
unless (defined $logs{$1})
{
open $fh,'>',"Thread$1.log") or die "Can't open Thread $1 log: $!";
$logs{$1} = $fh;
}
$last_log = $logs{$1};
}
if (defined $last_log)
{
print {$last_log} $_;
}
else
{
#Didn't find starting line. Error handling?
}
}
This solution maintains a hash of open file handles to the log files for all threads. I prefer this as it is more efficient if the input will have a lot of switching back and forth between the same threads. It would break, however, if the application has more threads than you are allowed to have files open on your system.
I am writing a program that somewhat mimics the last command in UNIX, and I am trying to use backreferencing in my solution. My program does exactly what it is supposed to do but I get a run time error/warning. My question is why is this error/warning coming up and how can I fix an issue like this?
If you need more information I can provide.
Program Execution
./last dodoherty
OUTPUT
Here is a listing of the logins for dodoherty:
1. dodohert pts/1 pc-618-012.omhq. Wed Feb 8 09:19 still logged in
2. dodohert pts/6 ip98-168-203-118 Tue Feb 7 19:19 - 20:50 (01:31)
3. dodohert pts/3 137.48.207.178 Tue Feb 7 14:00 - 15:06 (01:05)
4. dodohert pts/1 137.48.219.250 Tue Feb 7 12:32 - 12:36 (00:04)
5. dodohert pts/21 137.48.207.237 Tue Feb 7 12:07 - 12:23 (00:16)
6. dodohert pts/11 ip98-168-203-118 Mon Feb 6 20:50 - 23:29 (02:39)
7. dodohert pts/9 ip98-168-203-118 Mon Feb 6 20:31 - 22:57 (02:26)
8. dodohert pts/5 pc-618-012.omhq. Fri Feb 3 10:24 - 10:30 (00:05)
Use of uninitialized value $1 in addition (+) at ./odoherty_last.pl line 43.
Use of uninitialized value $2 in addition (+) at ./odoherty_last.pl line 44.
Here is a summary of the time spent on the system for dodoherty:
dodoherty
8
8:6
The Code (Snippet of where the error is coming from, Also this is the only time $1 and $2 are used.)
foreach my $line2 (#user)
{
$line2 =~ /\S*\((\d{2,2})\:(\d{2,2})\)\s*/;
$hours = $hours + $1;
$mins = $mins + $2;
if( $mins >= 60 )
{
$hours = $hours + 1;
$mins = $mins - 60;
}
}
I think the problem might be in the following line.
1. dodohert pts/1 pc-618-012.omhq. Wed Feb 8 09:19 still logged in
That is because nothing matches the pattern so $1 and $2 are undefined.
As has been noted in other answers, your regex does not match, and therefore $1 and $2 are undefined. It is necessary to always check to make sure the appropriate regex matches before using these variables.
Below I have upgraded your script with some proper perl code. += and %= are handy operator in this case. You can read about them in perlop
Your regex uses \S* and \s*, both of which are completely unnecessary here, since your regex is not anchored to anything else. In other words, \S*foo\s* will match any string that contains foo, since it can match the empty string around foo. Also, {2,2} means "match at least 2 times, max 2", which in effect is the same as {2} "match 2 times".
You will see that I changed your math around, and that is because it assumes that $mins will never be higher than 120. I suppose technically, that is a safe assumption, but doing it like below, it can handle all values of minutes and successfully turn them into hours.
The script below is for demonstration. If you remove DATA and leave <>, you can use this script as-is like so:
last user | perl script.pl
Code:
use strict;
use warnings;
use v5.10; # required for say()
my ($hours, $mins);
while (<DATA>) { # replace with while (<>) for live usage
if (/\((\d{2})\:(\d{2})\)/) {
$hours += $1;
$mins += $2;
if( $mins >= 60 ) {
$hours += int ($mins / 60); # take integer part of division
$mins %= 60; # remove excess minutes
}
}
}
say "Hours: $hours";
say "Mins : $mins";
__DATA__
1. dodohert pts/1 pc-618-012.omhq. Wed Feb 8 09:19 still logged in
2. dodohert pts/6 ip98-168-203-118 Tue Feb 7 19:19 - 20:50 (01:31)
3. dodohert pts/3 137.48.207.178 Tue Feb 7 14:00 - 15:06 (01:05)
4. dodohert pts/1 137.48.219.250 Tue Feb 7 12:32 - 12:36 (00:04)
5. dodohert pts/21 137.48.207.237 Tue Feb 7 12:07 - 12:23 (00:16)
6. dodohert pts/11 ip98-168-203-118 Mon Feb 6 20:50 - 23:29 (02:39)
7. dodohert pts/9 ip98-168-203-118 Mon Feb 6 20:31 - 22:57 (02:26)
8. dodohert pts/5 pc-618-012.omhq. Fri Feb 3 10:24 - 10:30 (00:05)
#!/usr/bin/perl
use strict;
my $hours = 0;
my $mins = 0;
my $loggedIn = 0;
while (<STDIN>)
{
chomp;
if (/\S*\((\d{2,2})\:(\d{2,2})\)\s*/)
{
$hours = $hours + $1;
$mins = $mins + $2;
if($mins >= 60 )
{
$hours = $hours + 1;
$mins = $mins - 60;
}
}
elsif (/still logged in$/)
{
$loggedIn = 1;
}
}
print "Summary: $hours:$mins ", ($loggedIn) ? " (Currently logged in)" : "", "\n";
When ever your RE fails to match, $1 and $2 have no value.
For this reason, it's considered best practice on ever to use $1, $2 etc. inside a conditional which tests the success of the RE.
So don't do:
$string =~ m/(somepattern)/sx;
my $var = $1;
But instead to do something like:
my $var = 'some_default_value';
if($string =~ m/(somepattern)/sx){
$var = $1;
}
I have a working perl script that grabs the data I need and displays them to STDOUT, but now I need to change it to generate a data file (csv, tab dellimited, any delimiter file).
The regular expression is filtering the data that I need, but I don't want the entire string, just snippets of the output. I'm assuming I would need to store this in another variable to create my output file.
I need a good example of this or suggestions to alter this code. Thank you in advance. :-)
Here's my code:
#!/usr/bin/perl -w
# Usage: ./bakstatinfo.pl Jul 28 2010 /var/log/mybackup.log <server1> <server2>
use strict;
use warnings;
#This piece added to view the arguments passed in
$" = "][";
print "===================================================================================\n";
print "[#ARGV]\n";
#Declare Variables
my($mon,$day,$year,$file,$server) = #ARGV;
my $regex_flag = 0;
splice(#ARGV, 0, 4, ());
foreach my $server ( #ARGV ) { #foreach will take Xn of server entries and add to the loop
print "===================================================================================\n";
print "REPORTING SUMMARY for SERVER : $server\n";
open(my $fh,"ssh $server cat $file |") or die "can't open log $server:$file: $!\n";
while (my $line = <$fh>) {
if ($line =~ m/.* $mon $day \d{2}:\d{2}:\d{2} $year:.*(ERROR:|backup-date=|backup-size=|backup-time=|backup-status)/) {
print $line;
$regex_flag=1; #Set to true
}
}
if ($regex_flag==0) {
print "NOTHING TO REPORT FOR $server: $mon $day $year \n";
}
$regex_flag=0;
close($fh);
}
Sample raw log file I am using: (recently added to provide better representation of log)
Tue Jul 27 23:00:06 2010: test202.bak_lvm:backup:ERROR: mybak-abc appears to be already running for this backupset
Tue Jul 27 23:00:06 2010: test202.bak_lvm:backup:ERROR: If you are sure mybak-abc is not running, please remove the file /etc/mybak-abc/test202.bak_lvm/.mybak-abc.pid and restart mybak-abc
Tue Jul 27 23:00:06 2010: test202.bak_lvm:backup:INFO: PHASE START: Cleanup
Tue Jul 27 23:00:06 2010: test202.bak_lvm:backup:INFO: PHASE END: Cleanup
Tue Jul 27 23:00:06 2010: test202.bak_lvm:backup:INFO: END OF BACKUP
Wed Jul 28 00:00:04 2010: db9.abc.bak:backup:INFO: START OF BACKUP
Wed Jul 28 00:00:04 2010: db9.abc.bak:backup:INFO: PHASE START: Initialization
Wed Jul 28 00:00:05 2010: db9.abc.bak:backup:WARNING: Binary logging is off.
Wed Jul 28 00:00:05 2010: db9.abc.bak:backup:INFO: License check successful
Wed Jul 28 00:00:05 2010: db9.abc.bak:backup:INFO: License check successful for lvm-snapshot.pl
Wed Jul 28 00:00:05 2010: db9.abc.bak:backup:INFO: backup-set=db9.abc.bak
Wed Jul 28 00:00:05 2010: db9.abc.bak:backup:INFO: backup-date=20100728000004
Wed Jul 28 00:00:05 2010: db9.abc.bak:backup:INFO: SQL-server-os=Linux/Unix
Wed Jul 28 00:00:05 2010: db9.abc.bak:backup:INFO: backup-type=regular
Wed Jul 28 00:00:05 2010: db9.abc.bak:backup:INFO: host=db9.abc.bak.test.com
Wed Jul 28 00:00:05 2010: db9.abc.bak:backup:INFO: backup-date-epoch=1280300404
Wed Jul 28 00:00:05 2010: db9.abc.bak:backup:INFO: retention-policy=3D
Wed Jul 28 00:00:05 2010: db9.abc.bak:backup:INFO: mybak-abc-version=ABC for SQL Enterprise Edition - version 3.1
Wed Jul 28 00:00:05 2010: db9.abc.bak:backup:INFO: SQL-version=5.1.32-test-SMP-log
Wed Jul 28 00:00:05 2010: db9.abc.bak:backup:INFO: backup-directory=/home/backups/db9.abc.bak/20100728000004
Wed Jul 28 00:00:05 2010: db9.abc.bak:backup:INFO: backup-level=0
Wed Jul 28 00:00:05 2010: db9.abc.bak:backup:INFO: backup-mode=raw
Wed Jul 28 00:00:05 2010: db9.abc.bak:backup:INFO: PHASE END: Initialization
Wed Jul 28 00:00:05 2010: db9.abc.bak:backup:INFO: PHASE START: Running pre backup plugin
Wed Jul 28 00:00:05 2010: db9.abc.bak:backup:INFO: PHASE START: Flushing logs
Wed Jul 28 00:00:05 2010: db9.abc.bak:backup:INFO: PHASE END: Flushing logs
Wed Jul 28 00:00:05 2010: db9.abc.bak:backup:INFO: PHASE START: Creating snapshot based backup
Wed Jul 28 00:00:11 2010: db9.abc.bak:backup:INFO: Wed Jul 28 00:49:53 2010: test203.bak_lvm:backup:INFO: raw-databases-snapshot=test SQL sgl
Wed Jul 28 00:49:53 2010: test203.bak_lvm:backup:INFO: PHASE END: Creating snapshot based backup
Wed Jul 28 00:49:53 2010: test203.bak_lvm:backup:INFO: PHASE START: Calculating backup size & checksums
Wed Jul 28 00:49:54 2010: test203.bak_lvm:backup:INFO: last-backup=/home/backups/test203.bak_lvm/20100726200004
Wed Jul 28 00:49:54 2010: test203.bak_lvm:backup:INFO: backup-size=417.32 GB
Wed Jul 28 00:49:54 2010: test203.bak_lvm:backup:INFO: PHASE END: Calculating backup size & checksums
Wed Jul 28 00:49:54 2010: test203.bak_lvm:backup:INFO: read-locks-time=00:00:05
Wed Jul 28 00:49:54 2010: test203.bak_lvm:backup:INFO: flush-logs-time=00:00:00
Wed Jul 28 00:49:54 2010: test203.bak_lvm:backup:INFO: backup-time=04:49:51
Wed Jul 28 00:49:54 2010: test203.bak_lvm:backup:INFO: backup-status=Backup succeeded
My working output now:
===================================================================================
[Jul][28][2010][/var/log/mybackup.log][server1]
===================================================================================
REPORTING SUMMARY for SERVER : server1
Wed Jul 28 00:49:54 2010: test203.bak_lvm:backup:INFO: backup-size=417.32 GB
Wed Jul 28 00:49:54 2010: test203.bak_lvm:backup:INFO: backup-time=04:49:51
Wed Jul 28 00:49:54 2010: test203.bak_lvm:backup:INFO: backup-status=Backup succeeded
The output I need to see would be something like this:(data file with separated by ';' for example)
MyDate=Wed Jul 28;MyBackupSet= test203.bak_lvm;MyBackupSize=187.24 GB;MyBackupTime=04:49:51;MyBackupStat=Backup succeeded
Use 'capturing parentheses' to identify the bits you want to deal with.
if ($line =~ m/(.* $mon $day) \d{2}:\d{2}:\d{2} $year:.*
(ERROR:|backup-date=|backup-size=|
backup-time=|backup-status)/x) {
You will need to do some surgery on the second set of parentheses - those surrounding the start of the various keywords. You may have to chop those out in bits and pieces inside the condition.
When you have all the data extracted into variables, use Text::CSV to handle CSV output (and input).
There are a myriad modules to handle HTML or XML (over 2000, and I think over 3000, with HTML in their name - I happened to look yesterday). Many of those won't be applicable, but CPAN is your friend.
Answering questions posed by comments
Would I split them off into separate variables as well? The first part gives me the date/time that I need. The next filter then gives me 1) Error: 2)backup-date= 3)backup-size= ...etc.
More or less. Unfortunately, you don't show some representative input lines, which means it is hard to tell what might be best. However, it seems likely that a scheme such as:
while (my $line = <$fh>)
{
chomp $line;
if ($line =~ m/(.* $mon $day) \d\d:\d\d:\d\d $year:/)
{
my $date = $1;
my %items = ();
$line =~ s/.* $mon $day \d\d:\d\d:\d\d $year://;
while ($line =~ m/(ERROR|backup-date|backup-size|
backup-time|backup-status)
[:=]([^:]+)/x)
{
my $key = $1;
my $val = $2;
$items{$key} = $val;
$line =~ s/$key[:=]$val[:=]?//;
}
# The %items hash contains the split out information.
# Now write the data for this line of the log file.
}
}
There might well be better ways to handle the trimming (but it is Perl so TMTOWTDI), but the basic idea here is to catch the lines that are interesting, then progressively chop the bits of interest out of the line, so the line grows shorter on each iteration (therefore, eventually terminating the inner while loop).
Note the use of the /x modifier to allow for a more readable regex split over lines (I edited the original answer version to use that too). I've also allowed 'ERROR' to be followed by an '=' or the other keywords to be followed by ':'; it seems unlikely that you'd get false matches that way, and it simplifies the regex substitute operations. The initial pattern match no longer requires one of the subsections to be present, either. You must judge for yourself whether those small changes (which might pick up non-conforming information) matter or not. For most of my purposes, the chance of the mismatch is small enough not to be an issue - but for legal reasons, it might not be acceptable to you.
Answering questions posed by 'answer'
I manufactured some data:
Wed Jul 30 00:49:51 2010: test203.bak_lvm:backup:INFO: backup-size=417.32 GB
Wed Jul 30 00:49:52 2010: test203.bak_lvm:backup:INFO: backup-time=04:49:51
Wed Jul 30 00:49:53 2010: test203.bak_lvm:backup:INFO: backup-status=Backup succeeded
Wed Jul 30 00:49:51 2010: backup-size=417.32 GB:backup-time=04:49:51:backup-status=Backup succeeded
I took the script in the answer and hacked and instrumented it - making it standalone.
I also removed the dependency on specific files - it reads standard input and writes to standard output. It makes my testing easier - and the code more flexible.
use strict;
use warnings;
use constant debug => 0;
my $mon = 'Jul';
my $day = 30;
my $year = 2010;
while (my $line = <>)
{
chomp $line;
print "Line: $line\n" if debug;
if ($line =~ m/(.* $mon $day) \d\d:\d\d:\d\d $year:/) #Mon Jul 26 22:00:02 2010:
{
print "### Scan\n";
my $date = $1;
print "$date\n";
my %items = ();
$line =~ s/.* $mon $day \d\d:\d\d:\d\d $year://;
print "Line: $line\n" if debug;
while ($line =~ m/(ERROR|backup-date|backup-size|backup-time|backup-status)[:=]([^:]+)/)
{
my $key = $1;
my $val = $2;
$items{$key} = $val;
$line =~ s/$key[:=]$val[:=]?//;
print "$key=$val\n";
print "Line: $line\n" if debug;
}
print "### Verify\n";
for my $key (sort keys %items)
{
print "$key = $items{$key}\n";
}
}
}
The output I get is:
### Scan
Wed Jul 30
backup-size=417.32 GB
### Verify
backup-size = 417.32 GB
### Scan
Wed Jul 30
backup-time=04
### Verify
backup-time = 04
### Scan
Wed Jul 30
backup-status=Backup succeeded
### Verify
backup-status = Backup succeeded
### Scan
Wed Jul 30
backup-size=417.32 GB
backup-time=04
backup-status=Backup succeeded
### Verify
backup-size = 417.32 GB
backup-status = Backup succeeded
backup-time = 04
The verify loop prints out the data from the '%items' hash quite happily. With the debug value set to 1 instead of 0, the output I get is:
Line: Wed Jul 30 00:49:51 2010: test203.bak_lvm:backup:INFO: backup-size=417.32 GB
### Scan
Wed Jul 30
Line: test203.bak_lvm:backup:INFO: backup-size=417.32 GB
backup-size=417.32 GB
Line: test203.bak_lvm:backup:INFO:
### Verify
backup-size = 417.32 GB
Line: Wed Jul 30 00:49:52 2010: test203.bak_lvm:backup:INFO: backup-time=04:49:51
### Scan
Wed Jul 30
Line: test203.bak_lvm:backup:INFO: backup-time=04:49:51
backup-time=04
Line: test203.bak_lvm:backup:INFO: 49:51
### Verify
backup-time = 04
Line: Wed Jul 30 00:49:53 2010: test203.bak_lvm:backup:INFO: backup-status=Backup succeeded
### Scan
Wed Jul 30
Line: test203.bak_lvm:backup:INFO: backup-status=Backup succeeded
backup-status=Backup succeeded
Line: test203.bak_lvm:backup:INFO:
### Verify
backup-status = Backup succeeded
Line: Wed Jul 30 00:49:51 2010: backup-size=417.32 GB:backup-time=04:49:51:backup-status=Backup succeeded
### Scan
Wed Jul 30
Line: backup-size=417.32 GB:backup-time=04:49:51:backup-status=Backup succeeded
backup-size=417.32 GB
Line: backup-time=04:49:51:backup-status=Backup succeeded
backup-time=04
Line: 49:51:backup-status=Backup succeeded
backup-status=Backup succeeded
Line: 49:51:
### Verify
backup-size = 417.32 GB
backup-status = Backup succeeded
backup-time = 04
The substitute operations delete the previously matched part of the line. There are ways of continuing a match where you left off - see \G at the 'perlre' page.
Note that the regex is crafted to stop at the first colon after the 'colon or equals' after the keyword. That means it truncates the backup time. One moral is "do not use a separator that can appear in the data". Another is "provide sample data so people can help you more easily". Another is "provide complete but minimal working scripts where possible".
Processing the sample data
Now that we have the sample input data, we can see that you need slightly different processing. This script:
use strict;
use warnings;
use constant debug => 0;
my $mon = 'Jul';
my $day = 28;
my $year = 2010;
my %items = ();
while (my $line = <>)
{
chomp $line;
print "Line: $line\n" if debug;
if ($line =~ m/(.* $mon $day) \d\d:\d\d:\d\d $year: ([^:]+):backup:/) #Mon Jul 26 22:00:02 2010:
{
print "### Scan\n" if debug;
my $date = $1;
my $set = $2;
print "$date ($set): " if debug;
$items{$set}->{'a-logdate'} = $date;
$items{$set}->{'a-dataset'} = $set;
if ($line =~ m/(ERROR|backup-date|backup-size|backup-time|backup-status)[:=](.+)/)
{
my $key = $1;
my $val = $2;
$items{$set}->{$key} = $val;
print "$key=$val\n" if debug;
}
}
}
print "### Verify\n";
for my $set (sort keys %items)
{
print "Set: $set\n";
my %info = %{$items{$set}};
for my $key (sort keys %info)
{
printf "%s=%s;", $key, $info{$key};
}
print "\n";
}
produces this result on the sample data file.
### Verify
Set: db9.abc.bak
a-dataset=db9.abc.bak;a-logdate=Wed Jul 28;backup-date=20100728000004;
Set: test203.bak_lvm
a-dataset=test203.bak_lvm;a-logdate=Wed Jul 28;backup-size=417.32 GB;backup-status=Backup succeeded;backup-time=04:49:51;
Note that now we have sample data, we can see that there is only one key/value pair per line, but there are multiple systems backed up per day. So, the inner while loop becomes a simple if. The printing out occurs at the end. And I'm using a 'two-tier' hash. The %items contains an entry for each data set; the entry, though, is a reference to a hash. Not necessarily something for novices to play with, but it fell into place very naturally with the previous code. Note, too, that this version doesn't hack the line - there's no need since there's only one lot of data per line.
Can it be improved - yes, undoubtedly. Does it work? Yes, more or less... Can it be hacked into shape? Yes, it can be hacked to work as you need.
#Jonathan- I wrote out the text file within the while loop. It seems to work. I tried doing it after the second while loop as you suggested in your comment. I'm not sure why it didn't work.
open (my $MYDATAFILE, ">/home/test/myout.txt") || die "cannot append $!";
open(my $fh,"ssh $server cat $file |") or die "can't open log $server:$file: $!\n";
while (my $line = <$fh>)
{
chomp $line;
if ($line =~ m/(.* $mon $day) \d\d:\d\d:\d\d $year:/) #Mon Jul 26 22:00:02 2010:
{
my $date = $1;
#print $date;
my %items = ();
$line =~ s/.* $mon $day \d\d:\d\d:\d\d $year://;
while ($line =~ m/(ERROR|backup-date|backup-size|backup-time|backup-status)[:=]([^:]+)/)
{
my $key = $1;
my $val = $2;
$items{$key} = $val;
$line =~ s/$key[:=]$val[:=]?//;
#print "[$key]";
#print "[$val]";
print $MYDATAFILE "$key=$val";
}
# The %items hash contains the split out information.
# Now write the data for this line of the log file.
}
}