split log of a multi-threaded application - regex

Question

I have a multi-threaded application which generates logs as mentioned:
D Fri Feb 01 00:21:23 2013 <no machine> pin_deferred_act:10233 pin_mta_conf.c:636 1:App-BRM-Prod-Pri.acttv.in:pin_deferred_act:10233:1:0:1359658283:1
pin_mta_convert_cmdline_options_to_flist parameters flist
D Fri Feb 01 00:21:23 2013 App-BRM-Prod-Pri.acttv.in pin_deferred_act:10233 pcpst.c(78):406 1:App-BRM-Prod-Pri.acttv.in:pin_deferred_act:10233:2:0:1359658283:0
connect to host=172.16.87.14, port=11962 OK
D Fri Feb 01 00:21:24 2013 App-BRM-Prod-Pri.acttv.in pin_deferred_act:10233 pin_mta.c:2479 1:App-BRM-Prod-Pri.acttv.in:pin_deferred_act:10233:1:0:1359658283:0
Config object search input flist
0 PIN_FLD_POID POID [0] 0.0.0.1 /search/pin -1 0
0 PIN_FLD_FLAGS INT [0] 0
0 PIN_FLD_TEMPLATE STR [0] "select X from /config/mta where F1 = V1 "
0 PIN_FLD_ARGS ARRAY [1] allocated 20, used 1
1 PIN_FLD_CONFIG_MTA ARRAY [0] allocated 20, used 1
2 PIN_FLD_NAME STR [0] "pin_deferred_act"
0 PIN_FLD_RESULTS ARRAY [0] allocated 20, used 1
1 PIN_FLD_POID POID [0] NULL poid pointer
D Fri Feb 01 00:21:24 2013 App-BRM-Prod-Pri.acttv.in pin_deferred_act:10233 pin_mta.c:2484 1:App-BRM-Prod-Pri.acttv.in:pin_deferred_act:10233:3:7:1359658284:2
Config object search output flist
0 PIN_FLD_POID POID [0] 0.0.0.1 /search/pin -1 0
D Fri Feb 01 00:21:24 2013 App-BRM-Prod-Pri.acttv.in pin_deferred_act:10233 pin_mta.c:3138 1:App-BRM-Prod-Pri.acttv.in:pin_deferred_act:10233:1:7:1359658284:2
So the threads update the logs, like pin_deferred_act:10233:1:7 --> where 1 specifies the log from the first thread, in the logfile.
I want to create log file for each thread, where the start point should be:
1:App-BRM-Prod-Pri.acttv.in:pin_deferred_act:10233:1:
and end- point should be:
D Fri Feb 01 00:21:24 2013 App-BRM-Prod-Pri.acttv.in
(where date/timestamp will keep on modifying).
All the instances should go in one file.
For e.g.:
D Fri Feb 01 00:21:23 2013 <no machine> pin_deferred_act:10233 pin_mta_conf.c:636 1:App-BRM-Prod-Pri.acttv.in:pin_deferred_act:10233:1:0:1359658283:1
pin_mta_convert_cmdline_options_to_flist parameters flist
D Fri Feb 01 00:21:23 2013 App-BRM-Prod-Pri.acttv.in pin_deferred_act:10233 pcpst.c(78):406 1:App-BRM-Prod-Pri.acttv.in:pin_deferred_act:10233:2:0:1359658283:0
connect to host=172.16.87.14, port=11962 OK
D Fri Feb 01 00:21:24 2013 App-BRM-Prod-Pri.acttv.in pin_deferred_act:10233 pin_mta.c:2479 1:App-BRM-Prod-Pri.acttv.in:pin_deferred_act:10233:1:0:1359658283:0
Config object search input flist
0 PIN_FLD_POID POID [0] 0.0.0.1 /search/pin -1 0
0 PIN_FLD_FLAGS INT [0] 0
0 PIN_FLD_TEMPLATE STR [0] "select X from /config/mta where F1 = V1 "
0 PIN_FLD_ARGS ARRAY [1] allocated 20, used 1
1 PIN_FLD_CONFIG_MTA ARRAY [0] allocated 20, used 1
2 PIN_FLD_NAME STR [0] "pin_deferred_act"
0 PIN_FLD_RESULTS ARRAY [0] allocated 20, used 1
1 PIN_FLD_POID POID [0] NULL poid pointer
D Fri Feb 01 00:21:24 2013 App-BRM-Prod-Pri.acttv.in pin_deferred_act:10233 pin_mta.c:3138 1:App-BRM-Prod-Pri.acttv.in:pin_deferred_act:10233:1:7:1359658284:2
should go to one file - Thread1.log, and similarly for other threads, the file should be created as Threadn.log with the respectively.

Answer 1

Files are a messy, non-scalable way of handling logs to begin with. A better approach is to handle logs as streams of log entry messages connected source(s) -> sink(s). Consider syslog, logplex or similar if Oracle provides alternative means of data collection. Custom re-implementation might be feasible depending on logging IOPS bottlenecks or other factors.
Use of high resolution monotonic clocks and/or globally-ordered GUID timestamps are highly recommended. With wall time, be sure to use non-backwards compensated UTC everywhere synced to low strata time sources.
Above recommendations may vary according to needs of the application of course, so experiment and implement wisely.

Answer 2

I think Barry's advice is useful, but in the event that you can't alter the application's log output, here is a quick Perl solution:
#!usr/bin/perl
use strict;
use warnings;
my %logs;
my $last_log;
while (<$main_log_file>) #open that application's log in this variable.
{
if (/pin_deferred_act:\d+:(\d+):\d/)
{
unless (defined $logs{$1})
{
open $fh,'>',"Thread$1.log") or die "Can't open Thread $1 log: $!";
$logs{$1} = $fh;
}
$last_log = $logs{$1};
}
if (defined $last_log)
{
print {$last_log} $_;
}
else
{
#Didn't find starting line. Error handling?
}
}
This solution maintains a hash of open file handles to the log files for all threads. I prefer this as it is more efficient if the input will have a lot of switching back and forth between the same threads. It would break, however, if the application has more threads than you are allowed to have files open on your system.