using awk to grep from each bunch of multiple lines - regex

Question

I have a log file which has bunch of lines, where each bunch is separated by blank lines. I want to grep certain lines (containing common pattern) from each bunch of line. Each bunch of line is about a mail. A sample log file is as follows:
#START#
03:48:19:798: : <23/08/2012 03:48:19:019>
03:48:19:798: : <---23/08/2012 03:48 --->
03:48:19:799: : MAIL FROM IP=1.2.3.4
03:48:19:799: : START CHECKING OF IPLIMIT
03:48:19:799: : STOP CHECKING OF IPLIMIT
03:48:20:848:In : MAIL FROM: <a#abc.com>
03:48:20:848: : [A:A:A]
03:48:20:849: : max attach size-->5242880
03:48:20:856: : User Is Authenticated with "a#abc.com and domain abc.com"
03:48:20:856: : Passed
03:48:20:987:In : RCPT TO: <x#xyz.com>
03:48:20:987: : email x#xyz.com
03:48:20:992: : [A:A:A]
03:48:20:999: : passed
03:48:20:999:Inside the Store Mails
03:48:20:999: : BCC feature is not applicable x#xyz.com
03:48:21:000: : BCC feature is not applicable from a#abc.com
03:48:21:000:Inside the Store
03:48:21:132:In : RCPT TO: <y#xyz.com>
03:48:21:132: : email y#xyz.com
03:48:21:133: : [A:A:A]
03:48:21:140: : passed
03:48:21:140:Inside the Store Mails
03:48:21:140: : BCC feature is not applicable y#xyz.com
03:48:21:140: : not authenticated
03:48:21:140:Inside the Store
03:48:21:271: : Data Received
03:50:32:049: : 552 Size Limit Exceeded(5242880)
03:50:32:049: : File Moved in LargeSize Folder....
03:50:32:049: : File Moved in LargeSize Folder....
03:50:32:049: : Connection closed
03:50:32:049: : File Deleted /home/Mail//mailbox/LargeSize/x#xyz.com:24085.444724474357(1345673901000)
03:50:32:051: : File Deleted /home/Mail//mailbox/LargeSize/y#xyz.com:39872.512978520455(1345673901140)
MAIL DATA : : 6815779 Bytes
Total: Conn : 16713 Quit By Host : 5565 Stored : 11134 Loop:0
#END#
W A R N I N G ---------------W A R N I N G
...Waiting for activity on port Total Thread Started & 16732 Stoped 16730
#START#
03:56:20:790: : <23/08/2012 03:56:20:020>
03:56:20:790: : <---23/08/2012 03:56 --->
03:56:20:791: : MAIL FROM IP=2.3.4.5
03:56:20:792: : IP IS FRIEND IN WHITELIST
03:56:20:834:In : MAIL FROM:<y#xyz.com>
03:56:20:834: : [A:A:A]
03:56:20:834: : null
03:56:20:834: : Passed
03:56:20:834:In : RCPT TO: <a#abc.com>
03:56:20:834: : email a#abc.com
03:56:20:835: : Mailing List
03:56:20:835: : [A:A:A]
03:56:20:836: : passed
03:56:20:836: : Proceesing maillist
03:56:20:839: : Data Received
03:56:20:865: : /home/Mail//mailbox/MailingList/a#abc.com:79602.39544573233(1345674380836) Msg Queued For Delivery
03:56:20:865: : Msg forward successfully
03:56:20:865: : /home/Mail//mailbox/MailingList/M14310.39892966699(1345674380837) Msg Queued For Delivery
MAIL DATA : : 27985 Bytes
Total: Conn : 16732 Quit By Host : 5582 Stored : 11135 Loop:0
#END#
...Waiting for activity on port Total Thread Started & 16735 Stoped 16731
#START#
03:56:23:957: : <23/08/2012 03:56:23:023>
03:56:23:957: : <---23/08/2012 03:56 --->
03:56:23:958: : MAIL FROM IP=2.3.4.5
03:56:23:959: : IP IS FRIEND IN WHITELIST
03:56:23:999:In : MAIL FROM: <x#xyz.com>
03:56:23:999: : [A:A:A]
03:56:23:999: : null
03:56:23:999: : Passed
03:56:23:999:In : RCPT TO: <y#xyz.com>
03:56:23:999: : email y#xyz.com
03:56:24:000: : [A:A:A]
03:56:24:007: : passed
03:56:24:008:Inside the Store Mails
03:56:24:009: : BCC feature is not applicable y#xyz.com
03:56:24:009: : not authenticated
03:56:24:009:Inside the Store
03:56:24:009: : Data Received
03:56:24:053: : /home/Mail//mailbox/External/y#xyz.com:50098.70335800691(1345674384009) Msg Queued For Delivery
03:56:24:054: : Msg forward successfully
MAIL DATA : : 28276 Bytes
Total: Conn : 16735 Quit By Host : 5582 Stored : 11136 Loop:0
#END#
Here, a#abc.com is an external mail id, and x#xyz.com, y#xyz.com are internal mail ids.
For each mail, the bunch of lines starting from #START# to #END# are generated.
From each bunch of lines I want to run some pattern matching. I only want those bunch of lines where mail is from an internal email id to external email id (second bunch of line).
I don't want bunch of lines where mail is from external email address/id to internal email id (1st bunch of line), or from an internal email id to internal email id. (3rd bunch of line).
And after I have the bunch of line where mail is from internal to external, I want to extract the line containing the word FROM and TO.
I tried using the RS, ORS, FS and OFS variables of awk to convert each bunch of line starting from and ending at #START# to make a single-line record, but couldn't. I could not replace the newlines by a separator such as | or ~. Also, I don't now how to run multiple pattern matching on each resource record.
I tried using /PATTERN/ option, but then could not run the grep command using system() function to get the lines to check the domain names. it gave me errors: sh: 1: not found. Could not break through it. I used the code:
if ($0 ~ /FROM/) { print $0 | system("egrep -i 'FROM|TO'") }
Also, if I try to export each record using following type of code, its not working:
for i in $(cat log_file | awk_file_givin_1_resource_record_at_a_time) ; do pattern_matching_commands ; done
It's no working cause the pattern matching is working on a line at a time, while I want it to work on the entire bunch at a time.

Answer 1

I think the following BASH script would work well, but you should benchmark it for the size of your logs:
#!/bin/bash
INTERNAL_DOMAINS="${1:-xyz.com|xyz.net}"
declare -i LINES BYTES VALIDS
LINES=0
BYTES=0
VALIDS=0
STATUS=stopped
while read LINE
do
if [ "$STATUS" = stopped ]
then
if [ "${LINE:0:7}" = "#START#" ]
then
STATUS=started
PARA=""
fi
else
if [ "${LINE:0:5}" = "#END#" ]
then
if [ $STATUS = valid ]
then
VALIDS+=1
echo "$PARA" | egrep -w "FROM|TO"
echo -e "$VALIDS matched\t----------------------------------------"
fi
STATUS=stopped
elif (echo "$LINE" | fgrep -q "RCPT TO") && (echo "$LINE" | egrep -qiv "#($INTERNAL_DOMAINS)")
then
STATUS=valid
PARA+="$LINE
"
else
PARA+="$LINE
"
fi
fi
LINES+=1
BYTES+=${#LINE}
BYTES+=1
echo -en "\rRead: lines: $LINES | kB: $(($BYTES/1024)) | matches: $VALIDS " >&2
done
You should set the above script as executable and run it like this to get progress output:
time ./filter.sh "one.int.com|two.int.com" < sample.log > report.out

Answer 2

If there is always a blank line between records, and never a blank line within a record, use awk's “paragraph mode”: set RS to the empty string.
awk -v RS= '
/^[0-9:]*In : MAIL FROM: <[^<>]*#example\.com>$/ &&
/^[0-9:]*In : RCPT TO: <[^<>]*#example\.com>$/ { … }
'
If you really need to use the #START# and #END# markers, accumulate data in variables as you go along. Do the processing then reset the variables when you reach #END#. Disable processing until the next #START# if necessary.
BEGIN { in_record = 1; }
/^#START#$/ { in_record = 1; }
!in_record { next; }
/^[0-9:]*In : MAIL FROM: <([^<>]*)>$/ { from = $0; sub(/.*</, "", from); sub(/>.*/, "", from); }
…
/^#END#$/ {
/* processing goes here */
from = "";
in_record = 0;
}