awk print lines before while match INFO untill match ERROR - regex

I want to print lines before my /ERROR/ match. The lines to be printed should be all containing INFO untill the previous ERROR is found.
So If I had a file
ERROR this is an error
INFO error found on line 2
INFO error is due to something
ERROR this is another error
I want the /ERROR/ from ERROR this is another error to print
INFO error found on line 2
INFO error is due to something
ERROR this is another error
Anyone know?
Part of my current script:
/CRITICAL/ {
print "\x1b[93;1m"
}
/ERROR/ {
print "\x1b[37m"
}
/ERROR|EMERGENCY|CRITICAL/ {
if (NR == n+1) print "";
n = NR;
print x;print
print "\x1b[0m"
};{x=$0}'

Try this one liner:
awk 'x;/ERROR/{x=1}' file
Out:
INFO error found on line 2
INFO error is due to something
ERROR this is another error
Long version:
x;/ERROR/{
x1=1;
print
}
If "ERROR" is found x=1, if x is true we've already gone through that line, then we print until we pass that line again.
Or maybe this, I don't have very clear what output you need.
awk '/ERROR/{x=1;next}/ERROR/{x=1}x'
Out
INFO error found on line 2
INFO error is due to something

Related

Perl Program to parse through error log file, extract error message and output to new file

I need to write a perl program where I parse through an error log and output the error messages to a new file. I am having issues with setting up the regex to do this. In the error log, an error code starts with the word "ERROR" and the end of each error message ends with a ". " (period and then a space). I want to find all the errors, count them, and also output the entire error message of each error message to a new file.
I tried this but am having issues:
open(FH,"<$filetoparse");
$outputfile='./errorlog.txt';
open(OUTPUT,">$outputfile");
$errorstart='ERROR';
$errorend=". ";
while(<FH>)
{
if (FH=~ /^\s*$errorstart/../$errorend/)
{
print OUTPUT "success";
}
else
{
print OUTPUT "failed";
}
}
}
the $errorstart and $errorend are something I saw online and am not sure if that is the correct way to code it.
Also I know the printing "Success" or "Failure" is not what I said I am looking for, I added that in to help with confirmed that the code works, I haven't tried coding for counting the error messages yet.
before this snippet of code I have a print statement asking the user for the location address of the .txt file they want to parse. I confirmed that particular section of code words properly. Thanks for any help! Let me know if more info is needed!
Here is an example of data that I will be using:
Sample Data
-----BEGIN LOAD-----
SUCCESS: file loaded properly .
SUCCESS: file loaded properly .
SUCCESS: file loaded properly .
SUCCESS: file loaded properly .
SUCCESS: file loaded properly .
SUCCESS: file loaded properly .
ERROR: the file was unable to load for an unknown reason .
SUCCESS: file loaded properly .
SUCCESS: file loaded properly .
ERROR: the file was unable to load this is just an example of a log file that will span
multiple lines .
SUCCESS: file loaded properly .
------END LOAD-------
While the log may not necessarily NEED to span multiple lines, there will be some data throughout the log that will similar to how it is above. Every message logged starts with either SUCCESS or ERROR and the message is done when a " . " (whitespace-period-whitespace) is encountered. The log I want to parse through is 50,000 entries long so needless to say I would like to code so it will also identify multi-line error msgs as well as output the entire multi-line message to the output file.
update
I have written the code but for some reason it won't work. I think it has to do with the delimiter but I can't figure out what it is. The file I am using has messages that are separated by "whitespace period newline". Can you see what I'm doing wrong??
{
local $/ = " .\n";
if ($outputtype == 1)
{
$outputfile="errorlog.txt";
open(OUTPUT,">>$outputfile");
$errorcount=0;
$errortarget="ERROR";
print OUTPUT "-----------Error Log-----------\n";
{
while(<FH>)
{
if ($_ =~ m/^$errortarget/)
{
print OUTPUT "$_\n";
print OUTPUT "next code is: \n";
$errorcount++;
}
}
print OUTPUT "\nError Count : $errorcount\n";
}
}
}
There are several problems with your code to start off.
ALWAYS use strict; and use warnings;.
3 argument open is much less error prone. open ( my $fh, "<", $filename ) or die $!;
Always check open actually worked.
FH =~ doesn't do what you think it does.
range operator tests if you're between two chunks of text in code. This is particularly relevant for multi-line operations. If your error log isn't, then it's not what you need.
Assuming you've error data like this:
ERROR: something is broken.
WARNING: something might be broken.
INFO: not broken.
ERROR: still broken.
This code will do the trick:
use strict;
use warnings;
my $filetoparse = "myfile.txt";
my $outputfile = "errorlog.txt";
open( my $input, "<", $filetoparse ) or die $!;
open( my $output, ">", $outputfile ) or die $!;
my $count_of_errors = 0;
#set record delimiter
local $/ = " . \n";
while ( my $lines = <$input> ) {
$lines =~ s/^-----\w+ LOAD-----\n//g; #discard any 'being/end load' lines.
if ( $lines =~ m/^ERROR/ ) {
$count_of_errors++;
print {$output} $lines;
}
}
close ( $input );
close ( $output );
print "$count_of_errors errors found\n";
If you've multi-line error message, then you'll need a slightly different approach though.

How to append the lines using SED or AWK until a particular pattern does not matches?

I want to append the line which does not start with a given pattern. But i am unable to do that using sed. Plz help me to solve the problem using either sed or awk. For an Example:
INPUT:
18:55:42[pool-1-thread-2] INFO jfileupload.download.http.a - Download completed
18:55:42[HTTPDOWNLOAD] ERROR jfileupload.download.ui.DownloadTransferUI -
java.io.IOException: Failed to show URI:file:/home/rahul/Desktop/
at sun.awt.X11.XDesktopPeer.launch(XDesktopPeer.java:114)
at sun.awt.X11.XDesktopPeer.open(XDesktopPeer.java:77)
at java.awt.Desktop.open(Desktop.java:272)
at jfileupload.download.ui.DownloadTransferUI.a(Unknown Source)
at jfileupload.download.http.HTTPDownloadTransfer.a(Unknown Source)
at jfileupload.download.a.a.run(Unknown Source)
at java.lang.Thread.run(Thread.java:745)
18:55:43[MultiThreadedHttpConnectionManager cleanup] DEBUG org.apache.commons.httpclient.MultiThreadedHttpConnectionManager - ReferenceQueueThread interrupted
java.lang.InterruptedException
at java.lang.Object.wait(Native Method)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151)
at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$ReferenceQueueThread.run(MultiThreadedHttpConnectionManager.java:1122)
Required Output:
18:55:42[pool-1-thread-2] INFO jfileupload.download.http.a - Download completed
18:55:42[HTTPDOWNLOAD] ERROR jfileupload.download.ui.DownloadTransferUI -java.io.IOException: Failed to show URI:file:/home/rahul/Desktop/, at sun.awt.X11.XDesktopPeer.launch(XDesktopPeer.java:114), at sun.awt.X11.XDesktopPeer.open(XDesktopPeer.java:77), at java.awt.Desktop.open,(Desktop.java:272), at jfileupload.download.ui.DownloadTransferUI.a(Unknown Source), at jfileupload.download.http.HTTPDownloadTransfer.a(Unknown Source), at jfileupload.download.a.a.run(Unknown Source), at java.lang.Thread.run(Thread.java:745)
18:55:43[MultiThreadedHttpConnectionManager cleanup] DEBUG org.apache.commons.httpclient.MultiThreadedHttpConnectionManager - ReferenceQueueThread interrupted,java.lang.InterruptedException, at java.lang.Object.wait(Native Method), at java.lang.ref.ReferenceQueue.remove,(ReferenceQueue.java:135), at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151), at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$ReferenceQueueThread.run(MultiThreadedHttpConnectionManager.java:1122)
In the above input i want to append the next line into current line with , until the new line start with such pattern 18:55:42.
Thanks.
not tried it but this should work
awk '/[0-9]+:[0-9]+:[0-9]+/{x=$0}{a[x]=a[x]?a[x]", "$0:$0}END{for (i in a)print a[i]}' file
.
/[0-9]+:[0-9]+:[0-9]+/ If this pattern is matched
{x=$0} Set x to the value of the line
{a[x]=a[x]?a[x]", "$0:$0} Create associative array with x($0) using ternary
operator to check there is a value in a[x] to
begin with. Adds the current lines value to the
a[x]
END{for (i in a)print a[i]} When all records are processed loop through array
and output values

C++-CLI Pipeline only returns the first row of the error message

I'm executing exchange powershell commands through a pipeline from my code.
The problem is that when the command is executed on the powershell directly the error returned is four rows long, but when I run the command through the pipeline it only returns the first row of the error message.
This is the code I have so far:
bool ps_calls::check_invoke(Pipeline^ pipeline)
{
if (pipeline->Error->Count <= 0)
return true;
Collection<System::Object^> errorCollection = pipeline->Error->ReadToEnd();
for (int i=0; i< errorCollection.Count; i++)
{
String^ msg = String::Format("Error {0}",errorCollection[i]->ToString());
m_errors->Add(msg);
m_logger->info_msg(String::Format("ERROR: {0}", msg));
}
return false;
}
This is an example of what my pipeline returns now:
ERROR: Error The operation couldn't be performed because object 'Testing' couldn't be found on 'server'.
This is an example of what the error message is if the same command is executed on the powershell console manually:
The operation couldn't be performed because object 'test' couldn't be found on 'server'.
+ CategoryInfo : NotSpecified: (:) [Get-MailboxDatabase], ManagementObjectNotFoundException
+ FullyQualifiedErrorId : 2F753561,Microsoft.Exchange.Management.SystemConfigurationTasks.GetMailboxDatabase
+ PSComputerName : server
I would like to be able to get the rest of this error message as well so that I can see the ErrorID when an error occurs.
PowerShell doesn't call ToString when it does formatting, that's why you don't see the same output.
Some host applications typically pipe output to Out-Default - if you did that, you'd see the expected format for errors.
Other host applications will pipe output to Out-String - this will format objects in the same way that the would be formatted if piped to Out-Default, but you'll get a string object that you can log however you need to.

Can googletest stringification produce a hexadecimal format?

I'm using Google Test to assert that certain error codes occur, and these are always hex constants. So this output is less than ideal:
mytest.cpp line 130 and its output:
EXPECT_EQ(0xBFFF0011, error) << "Expected second close to return an error";
[ RUN ] MyTest.CloseSessionFail
mytest.cpp(130): error: Value of: error
Actual: -1074130544
Expected: 0xBFFF0011
Which is: 3221159953
Expected second close to return an error
For EXPECT_EQ(expected, actual), is there some way to cause it to format hexadecimal output?
Ideally I'd like to see this:
Actual: 0xBFFA1190
Expected: 0xBFFF0011
Probably this message will help you:
https://github.com/google/googletest/issues/222

pig REPLACE gives error

Let's assume that my file is named 'data' and looks like this:
2343234 {23.8375,-2.339921102} {(343.34333,-2.0000022)} 5-23-2013-11-am
I need to convert the 2nd field to a pair of coordinate numbers. So I wrote the follwoing code and called it basic.pig:
A = LOAD 'data' AS (f1:int, f2:chararray, f3:chararray. f4:chararray);
B = foreach A generate STRSPLIT(f2,',').$0 as f5, STRSPLIT(f2,',').$1 as f6;
C = foreach B generate REPLACE(f5,'{',' ') as f7, REPLACE(f6,'}',' ') as f8;
and then used (float) to convert the string to a float. But, the command 'REPLACE' fails to work and I get the following error:
-bash-3.2$ pig -x local basic.pig
2013-06-24 16:38:45,030 [main] INFO org.apache.pig.Main - Apache Pig version 0.11.1 (r1459641) compiled
Mar 22 2013, 02:13:53 2013-06-24 16:38:45,031 [main] INFO org.apache.pig.Main - Logging error messages to: /home/--/p/--test/pig_1372117125028.log
2013-06-24 16:38:45,321 [main] INFO org.apache.pig.impl.util.Utils - Default bootup file /home/isl/pmahboubi/.pigbootup not found
2013-06-24 16:38:45,425 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: file:///
2013-06-24 16:38:46,069 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1000: Error during parsing. Lexical error at line 7, column 0. Encountered: <EOF> after : ""
Details at logfile: /home/--/p/--test/pig_1372117125028.log
And this is the details of the pig_137..log
Pig Stack Trace
---------------
ERROR 1000: Error during parsing. Lexical error at line 7, column 0. Encountered: <EOF> after : ""
org.apache.pig.tools.pigscript.parser.TokenMgrError: Lexical error at line 7, column 0. Encountered: <EOF> after : ""
at org.apache.pig.tools.pigscript.parser.PigScriptParserTokenManager.getNextToken(PigScriptParserTokenManager.java:3266)
at org.apache.pig.tools.pigscript.parser.PigScriptParser.jj_ntk(PigScriptParser.java:1134)
at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:104)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:194)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:170)
at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:84)
at org.apache.pig.Main.run(Main.java:604)
at org.apache.pig.Main.main(Main.java:157)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:197)
================================================================================
I've got data like this:
2724 1919 2012-11-18T23:57:56.000Z {(33.80981975),(-118.105289)}
2703 6401 2012-11-18T23:57:56.000Z {(55.83525609),(-4.07733138)}
1200 4015 2012-11-18T23:57:56.000Z {(41.49609152),(13.8411998)}
7104 9227 2012-11-18T23:57:56.000Z {(-24.95351118),(-53.46538723)}
and I can do this:
A = LOAD 'my_tsv_data' USING PigStorage('\t') AS (id1:int, id2:int, date:chararray, loc:chararray);
B = FOREACH A GENERATE REPLACE(loc,'\\{|\\}|\\(|\\)','');
C = LIMIT B 10;
DUMP C;
This error
ERROR 1000: Error during parsing. Lexical error at line 7, column 0. Encountered: <EOF> after : ""
came to me because I had used different types of quotation marks. I started with ' and ended with ยด or `, and it took quite a while to find what went wrong. So it had nothing to do with line 7 (my script was not so long, and I shortened data to four lines which naturally did not help), nothing to do with column 0, nothing to do with EOF of data, and hardly anything to do with " marks which I didn't use. So quite misleading error message.
I found the cause by using grunt - pig command shell.