after some investigation I have managed to find my wanted regular expression to work on /bin/sh - busybox:
INPUT:
Mar 8 09:58:29 mysuperhost kern.alert kernel: Rejected OUT -- IN=br0 OUT=vlan2 SRC=192.168.1.8 DST=3.26.211.8 LEN=95 TOS=0x00 PREC=0x00 TTL=127 ID=648 PROTO=UDP SPT=22008 DPT=51413 LEN=75
REGEXP:
grep -o -E '((^.{0,16})|(IN=.\S*)|(IN=.\S*)|(OUT=.\S*)|(SRC=.\S*)|(DST=.\S*)|(PROTO=.\S*)|(SPT=.\S*)|(DPT=.\S*))'
Which gives me:
Mar 8 09:58:29
IN=br0
OUT=vlan2
SRC=192.168.1.8
DST=3.26.211.8
PROTO=UDP
SPT=22008
DPT=51413
The problem:
I don't seem to be able to have grep giving me the result on a single line
The wanted result:
Mar 8 09:58:29 IN=br0 OUT=vlan2 SRC=192.168.1.8 DST=3.26.211.8 PROTO=UDP SPT=22008 DPT=51413
Here is an awk
awk '{printf "%s %s %s ",$1,$2,$3;for (i=4;i<=NF;i++) if ($i~/(IN|OUT|SRC|DST|PROTO|SPT|DPT)=/) printf "%s ",$i;print ""}' file
Mar 8 09:58:29 IN=br0 OUT=vlan2 SRC=192.168.1.8 DST=3.26.211.8 PROTO=UDP SPT=22008 DPT=51413
Just change whats in the if test to select fields you like.
Related
In the below phrase i need to extract /testcard-directory/100 where the number at the last can vary between 0 to 999.
[Tue Jun 27 02:40:25.207457 2020] [:error] [pid 4085] [client
23.23.23.32:34223] Server IP: 172.17.0.3\nUser agent: Amazon CloudFront\nServer time: 2020-06-27 19:40:25\nRequestURI:
/testcard-directory/100\nMessage: mongodb:27017: Read timed out after reading 0 bytes, waited for 30.000000 seconds\n\nTrace:\n#0
I tried this
\/flashcard-directory\/[1-9][0-9]
but the last number is not getting picked up.
Try :
/testcard-directory/[0-9]*
Demo :
$estcard-directory/100\nMessage: mongodb:27017: Read timed out after reading 0 bytes, waited for 30.000000 seconds\n\nTrace:\n#0' | grep -o '/testcard-directory/[0-9]* <
> '
/testcard-directory/100
$
regex: \/testcard-directory\/\d{0,3}
https://regex101.com/r/DDuKsX/1
Use: ^\/testcard-directory\/\d{1,3}$
Here is a Javascript implementation easily portable:
var test = [
'/testcard-directory/',
'/testcard-directory/0',
'/testcard-directory/123',
'/testcard-directory/1234',
];
console.log(test.map(function (a) {
return a+' : '+/^\/testcard-directory\/\d{1,3}$/.test(a);
}));
Demo & explanation
I wrote a simple rrdtool database to graph Wi-Fi signal strength and modulation. The signal strength works, but when I try to update the db with MCS information, I get:
ERROR: ./somefile.rrd: expected 2 data source readings (got 1) from mcsul15
Here's my update code:
rssi=`snmpget -v 2c -c communityname 1.2.3.4 .1.3.6.1.4.1.17713.21.1.2.3.0 | awk -v x=4 '{print $x}' | tr -d -`
noisefloor=`snmpget -v 2c -c communityname 1.2.3.4 .1.3.6.1.4.1.17713.21.1.2.20.1.9.1 | awk -v x=4 '{print $x}' | tr -d -`
ulmcs14=`snmpget -v 2c -c communityname 1.2.3.4 CAMBIUM-PMP80211-MIB::ulWLanMCS14Packets.0 | awk -v x=4 '{print $x}'`
ulmcs15=`snmpget -v 2c -c communityname 1.2.3.4 CAMBIUM-PMP80211-MIB::ulWLanMCS15Packets.0 | awk -v x=4 '{print $x}'`
echo $rssi
echo $noisefloor
echo $ulmcs14
echo $ulmcs15
rrdtool update ./somefile.rrd --template \
rssi:noisefloor N:$rssi:$noisefloor \
mcsul15:mcsul14 N:$ulmcs15:$ulmcs14
Which gives me:
68
94
143679
17602658
ERROR: ./somefile.rrd: expected 2 data source readings (got 1) from mcsul15
What am I missing?
Assuming that somefile.rrd has 4 DS defined in it with those 4 names, you should give all four together when updating. You can only specify one template for the update, and the other parameters should be in that format.
Also, check the names of your DS are correct as your variable is called $ulmcs15 but the DS is being named mcsul15.
rrdtool update ./somefile.rrd --template \
rssi:noisefloor:mcsul15:mcsul14 \
N:$rssi:$noisefloor:$ulmcs15:$ulmcs14
The error message is because in your original commandline, mcsul15:mcsul14 is being taken as an update vector, not a template. Thus it is one timestamp and one value, where two were expected. It would have been a better error message to say something like "timestamp not recognised in 'mcsul15'" but that's a different issue...
I have a input file which looks like
1S6290615260715DUTCH-ALDI ROTTERDAM, EUDOKIAPLEIN 8 00002961999
20000010019149GRANEN 0000000100000001590 0000111
20000010019592ALASKA KOOLVISFILET 0000001270000024003 0000111
20000010022614PAPRIKA 3 ST 0000000460000005934 0000111
1S6290615260715DUTCH-ALDI BERGEN NH, JAN OLDENBURGLAAN 00002962888
20000000000404BLEEKMIDDEL 0000000900000003150 0000222
20000000005197FRUIT 0000000430000005977 0000222
20000000006013ROOIBOSTHEE 0000000140000001246 0000222
1S6290615260715DUTCH-ALDI DWINGELOO, HEUVELENWEG 00002963777
20000000006469PITABROODJES 0000000610000004209 0000333
20000000007372SCHENKSTROOP 0000000210000001869 0000333
20000000007545HUISVUILZAKKEN 0000001080000012852 0000333
1S6290615260715DUTCH-ALDI BARNEVELD, CATHARIJNESTEEG 00002964666
20000000005197FRUIT + GRANEN BISCUITS 0000000720000010008 0000444
20000000005209IJSASSORTI MINIMIX 0000000190000003781 0000444
20000000006013ROOIBOSTHEE 0000000210000001869 0000444
I need the break this file into multiple files based on the pattern match. In this file the pattern line begins with 1S6290615260715, based on that I need to create multiple files like
File 1:
1S6290615260715DUTCH-ALDI ROTTERDAM, EUDOKIAPLEIN 8 00002961999
20000010019149GRANEN 0000000100000001590 0000111
20000010019592ALASKA KOOLVISFILET 0000001270000024003 0000111
20000010022614PAPRIKA 3 ST 0000000460000005934 0000111
File 2
1S6290615260715DUTCH-ALDI BERGEN NH, JAN OLDENBURGLAAN 00002962888
20000000000404BLEEKMIDDEL 0000000900000003150 0000222
20000000005197FRUIT 0000000430000005977 0000222
20000000006013ROOIBOSTHEE 0000000140000001246 0000222
and so on.
Using awk i tried this command
awk '/^1S/f++ {print $0 > "file"f}' input.txt
with this each file is created with single line.
Please suggest the faster processing way either with sed or awk, because I need to do this for very larger files like 15GB to 20GB and provide these split files to hadoop framework for further processing.
You can use this awk:
awk '/^1S/{if (f) close(f); f = "file" ++i} {print > f}' file
i would like to gzip log files but i cannot work out how to run a regex expression in my command.
My Log file look like this, they roll every hour.
-rw-r--r-- 1 aus nds 191353 Sep 28 01:59 fubar.log.20150928-01
-rw-r--r-- 1 aus nds 191058 Sep 28 02:59 fubar.log.20150928-02
-rw-r--r-- 1 aus nds 190991 Sep 28 03:59 fubar.log.20150928-03
-rw-r--r-- 1 aus nds 191388 Sep 28 04:59 fubar.log.20150928-04
script.
FUBAR_DATE=$(date -d "days ago" +"%Y%m%d ")
fubar_file="/apps/fubar/logs/fubar.log."$AUS_DATE"-^[0-9]"
/bin/gzip $fubar_file
i have tried a few varients on using the regex but without success, can you see the simple error in my code.
Thanks in advace
I did:
$ fubar_file="./fubar.log."${FUBAR_DATE%% }"-[0-9][0-9]"
and it worked for me.
Why not make fubar_file an array to hold the matching log file names, and then use a loop to gzip them individually. Then presuming AUS_DATE contains 20150928:
# FUBAR_DATE=$(date -d "days ago" +"%Y%m%d ") # not needed for gzip
fubar_file=( /apps/fubar/logs/fubar.log.$AUS_DATE-[0-9][0-9] )
for i in ${fubar_file[#]}; do
gzip "$i"
done
or if you do not need to preserve the filenames in the array for later use, just gzip the files with a for loop:
for i in /apps/fubar/logs/fubar.log.$AUS_DATE-[0-9][0-9]; do
gzip "$i"
done
or, simply use find to match the files and gzip them:
find /apps/fubar/logs -type f -name "fubar.log.$AUS_DATE-[0-9][0-9]" -execdir gzip '{}' +
Note: all answers presume AUS_DATE contains 20150928.
I have to parse and filter in linux command line only some log files.
after applying some awk and sed commands
awk -v RS='+++ ' '!/Diameter|REPT OM BLOCKED|REPT OM STARTING/ { print f $0 } {f=RT}' ./snmplog* | grep -v '+++' | grep -v '++-' | sed -e 's/^\s*//g' | sed -e '/^$/d'
I got an output like this which look like an xml file:
<Alarm>
<AlarmIndex>8865</AlarmIndex>
<ObjectName>0-0-1#RcvTCAPabortRatio^0-0-3</ObjectName>
<SpecificProblem>KPI OUTSIDE OF CRITICAL THRESHOLD</SpecificProblem>
<ProbableCause>ThresholdCrossed</ProbableCause>
<NotificationIdentifier>8865</NotificationIdentifier>
<Severity>Cleared</Severity>
<AlarmType>QualityOfServiceAlarm</AlarmType>
<AdditionalText></AdditionalText>
<OMText>REPT MEAS KPI
(RcvTCAPabortRatio^0-0-3 = 20) OUTSIDE OF CRITICAL ALARM THRESHOLD (10)</O
MText>
<AlarmCode>922044</AlarmCode>
<AlarmSource>PLATFORM</AlarmSource>
<AlarmTime>Wed Mar 11 00:15:10 2015</AlarmTime>
<RepeatCount>0</RepeatCount>
<OMDBKey>/MS044</OMDBKey>
<AutoClear>1</AutoClear>
</Alarm>
<Alarm>
<AlarmIndex>8928</AlarmIndex>
<ObjectName>0-0-1#RcvTCAPabortRatio^0-0-11</ObjectName>
<SpecificProblem>KPI OUTSIDE OF CRITICAL THRESHOLD</SpecificProblem>
<ProbableCause>ThresholdCrossed</ProbableCause>
<NotificationIdentifier>8928</NotificationIdentifier>
<Severity>Cleared</Severity>
<AlarmType>QualityOfServiceAlarm</AlarmType>
<AdditionalText></AdditionalText>
<OMText>REPT MEAS KPI
(RcvTCAPabortRatio^0-0-11 = 19) OUTSIDE OF CRITICAL ALARM THRESHOLD (10)</
OMText>
<AlarmCode>922044</AlarmCode>
<AlarmSource>PLATFORM</AlarmSource>
<AlarmTime>Wed Mar 11 00:15:10 2015</AlarmTime>
<RepeatCount>0</RepeatCount>
<OMDBKey>/MS044</OMDBKey>
<AutoClear>1</AutoClear>
</Alarm>
<Alarm>
<AlarmIndex>8771</AlarmIndex>
<ObjectName>0-0-1#SuccStandaloneISDRatio</ObjectName>
<SpecificProblem>ZERO DENOMINATOR</SpecificProblem>
<ProbableCause>CorruptData</ProbableCause>
<NotificationIdentifier>8771</NotificationIdentifier>
<Severity>Cleared</Severity>
<AlarmType>ProcessingErrorAlarm</AlarmType>
<AdditionalText></AdditionalText>
<OMText>REPT MEAS KPI
CALCULATION OF (SuccStandaloneISDRatio) FAILED FOR ZERO DENOMINATOR</OMText>
<AlarmCode>922041</AlarmCode>
<AlarmSource>PLATFORM</AlarmSource>
<AlarmTime>Wed Mar 11 01:00:10 2015</AlarmTime>
<RepeatCount>0</RepeatCount>
<OMDBKey>/MS041</OMDBKey>
<AutoClear>1</AutoClear>
</Alarm>
I would like to have after threatment something like this:
<Alarm><AlarmIndex>8771</AlarmIndex>...<OMText>REPT MEAS KPI
CALCULATION OF (SuccStandaloneISDRatio) FAILED FOR ZERO DENOMINATOR</OMText><AlarmCode>922041</AlarmCode>...</Alarm>
I have to remove all new line after > and keep new line between tags.
As you can see in my log I have an issue in the tag </OMText> in which I can have a new line also and it should be removed.
I already try with many sed regex found here, but without success
How can I do this?
[Edit]
As requested, please find below the original log file:
+++ FE01 2015-03-11 00:25:35 SNMP /SNM001 #310852 0-0-1 >
<Alarm>
<AlarmIndex>1119</AlarmIndex>
<ObjectName>0-0-3#destMMENotAvail</ObjectName>
<SpecificProblem>CLR error,Diameter Peer:p3.mmeccd.3gppnetwork.org</SpecificProblem>
<ProbableCause>CommunicationsSubsystemFailure</ProbableCause>
<NotificationIdentifier>1119</NotificationIdentifier>
<Severity>Minor</Severity>
<AlarmType>CommunicationAlarm</AlarmType>
<AdditionalText>The destination MME is not reachable</AdditionalText>
<OMText>CLR error,Diameter Peer:p3.mmeccd.3gppne
twork.org</OMText>
<AlarmCode>50906</AlarmCode>
<AlarmSource>SDM#RTLTE</AlarmSource>
<AlarmTime>Wed Mar 11 00:25:35 2015</AlarmTime>
<RepeatCount>0</RepeatCount>
<OMDBKey></OMDBKey>
<AutoClear>1</AutoClear>
</Alarm>
END OF REPORT #310852++-
+++ FE01 2015-03-11 00:25:58 SNMP /SNM001 #310853 0-0-1 >
<Alarm>
<AlarmIndex>8914</AlarmIndex>
<ObjectName>0-0-14#2AILogger.C!81</ObjectName>
<SpecificProblem>OM BLOCKED AILogger.C</SpecificProblem>
<ProbableCause>QueueSizeExceeded</ProbableCause>
<NotificationIdentifier>8914</NotificationIdentifier>
<Severity>Minor</Severity>
<AlarmType>QualityOfServiceAlarm</AlarmType>
<AdditionalText></AdditionalText>
<OMText>REPT OM BLOCKED FOR PROCESS PDLSU1
612 MESSAGES DISCARD
OM IDENTITY :
CRERROR BEING BLOCKED; FILE : AILogger.C LINE NUMBER : 81
</OMText>
<AlarmCode>906065</AlarmCode>
<AlarmSource>PLATFORM</AlarmSource>
<AlarmTime>Wed Mar 11 00:25:58 2015</AlarmTime>
<RepeatCount>0</RepeatCount>
<OMDBKey>/CR065</OMDBKey>
<AutoClear>1</AutoClear>
</Alarm>
END OF REPORT #310853++-
First I have to discared messages which contains within tags: "Diameter", "REPT OM BLOCKED" "REPT OM STARTING" then keeping only the message between the tags ...
awk '
/<Alarm>/,/<\/Alarm>/ {
sub(/^[[:blank:]]+/, "") # trim leading blanks
sub(/[[:blank:]]+$/, "") # trim trailing blanks
if (/>$/) # if the line ends with a tag
printf "%s", $0 # print it with no newline
else
print
}
/<\/Alarm>/ {print ""} # add a newline after each Alarm block
' log.file
outputs
<Alarm><AlarmIndex>1119</AlarmIndex><ObjectName>0-0-3#destMMENotAvail</ObjectName><SpecificProblem>CLR error,Diameter Peer:p3.mmeccd.3gppnetwork.org</SpecificProblem><ProbableCause>CommunicationsSubsystemFailure</ProbableCause><NotificationIdentifier>1119</NotificationIdentifier><Severity>Minor</Severity><AlarmType>CommunicationAlarm</AlarmType><AdditionalText>The destination MME is not reachable</AdditionalText><OMText>CLR error,Diameter Peer:p3.mmeccd.3gppne
twork.org</OMText><AlarmCode>50906</AlarmCode><AlarmSource>SDM#RTLTE</AlarmSource><AlarmTime>Wed Mar 11 00:25:35 2015</AlarmTime><RepeatCount>0</RepeatCount><OMDBKey></OMDBKey><AutoClear>1</AutoClear></Alarm>
<Alarm><AlarmIndex>8914</AlarmIndex><ObjectName>0-0-14#2AILogger.C!81</ObjectName><SpecificProblem>OM BLOCKED AILogger.C</SpecificProblem><ProbableCause>QueueSizeExceeded</ProbableCause><NotificationIdentifier>8914</NotificationIdentifier><Severity>Minor</Severity><AlarmType>QualityOfServiceAlarm</AlarmType><AdditionalText></AdditionalText><OMText>REPT OM BLOCKED FOR PROCESS PDLSU1
612 MESSAGES DISCARD
OM IDENTITY :
CRERROR BEING BLOCKED; FILE : AILogger.C LINE NUMBER : 81
</OMText><AlarmCode>906065</AlarmCode><AlarmSource>PLATFORM</AlarmSource><AlarmTime>Wed Mar 11 00:25:58 2015</AlarmTime><RepeatCount>0</RepeatCount><OMDBKey>/CR065</OMDBKey><AutoClear>1</AutoClear></Alarm>
To pipe (should be modified from original file like you post later)
sed '
# don t care out of section
/<Alarm>/,\#</Alarm># !d
# in section
/<Alarm>/,\#</Alarm># {
# keep line in hold buffer
H
# if not the end, loop (cycle to next line and start of script)
\#</Alarm># !b
# clean current buffer
s/.*//
# exchange buffer (current/hold)
x
# remove first new line (extra due to first keep)
s/\n//
# remove first new line
s/\n//
# reformat first part until OMText
s#\(</AlarmIndex>\).*\(<OMText>\)#\1...\2#
# reformat between AlarmCode and /Alarm
s#\(</AlarmCode>\).*\(</Alarm>\)#\1...\2#
# print result at output
}' YourFile
Self explain, posix version