I am trying to replace a hex value in one of my files.
However, I am not sure how to write it back out in its normal form. I am not sure if the below is the right way to do it. I am changing the hex 25 to hex 15.
The file example_2018-02-02-14-51-47_US.txt is in UTF-8.
cat example_2018-02-02-14-51-47_US.txt | sed 's/\x25\x15/ /g' | od -x > example_2018-02-02-14-51-47_US_Convert.txt
Here is what the end of my example file looks like when I do xxd. I am trying to replace the hex 25 at the end with hex 15:
05ff020: a289 9585 a2a2 1ed3 f4c2 d7f5 f8d3 f1f3 ................
05ff030: e7c4 1e95 a493 931e d985 98a4 85a2 a396 ................
05ff040: 991e 95a4 9393 1e95 a493 931e 95a4 9393 ................
05ff050: 1ef2 f0f2 f2f2 f2f4 f0f4 f3f1 1ed2 d3e8 ................
05ff060: e6f8 f7f8 c2e6 c1d2 1ee2 8599 a589 8389 ................
05ff070: 9587 1ec2 9696 9240 9686 40c2 a4a2 8995 .......#..#.....
05ff080: 85a2 a21e c689 9985 1ec2 a4a2 8995 85a2 ................
05ff090: a240 d6a6 9585 99a2 1e95 a493 931e 95a4 .#..............
05ff0a0: 9393 1e95 a493 931e f0f0 f0f5 1ec9 d31e ................
05ff0b0: f0f0 f0f0 f0f0 f0f0 f0f0 f0f0 f3f5 f825 ...............%
EDIT: now the question reflect changing a binary file, so I've changed the answer accordingly:
We can replace bytes in a binary file using \x notation directly. I made a testfile:
$ echo -ne "\xf0\xf0\xf3\xf5\xf8\x25" > binfile
$ cat binfile | xxd
00000000: f0f0 0c33 0c35 0c38 0c32 35 ...3.5.8.25
Ok, so, we just pipe this through sed and it works:
$ cat binfile | sed -e 's/\x25/\x15/g' > newbinfile
$ cat newbinfile | xxd
00000000: f0f0 f3f5 f815 ......
This is the approach I ended up taking that worked.
tr '\045' '\025' < example_2018-02-02-14-51-47_US.txt > example_2018-02-02-14-51-47_US_replaced.txt
Related
I wrote a simple rrdtool database to graph Wi-Fi signal strength and modulation. The signal strength works, but when I try to update the db with MCS information, I get:
ERROR: ./somefile.rrd: expected 2 data source readings (got 1) from mcsul15
Here's my update code:
rssi=`snmpget -v 2c -c communityname 1.2.3.4 .1.3.6.1.4.1.17713.21.1.2.3.0 | awk -v x=4 '{print $x}' | tr -d -`
noisefloor=`snmpget -v 2c -c communityname 1.2.3.4 .1.3.6.1.4.1.17713.21.1.2.20.1.9.1 | awk -v x=4 '{print $x}' | tr -d -`
ulmcs14=`snmpget -v 2c -c communityname 1.2.3.4 CAMBIUM-PMP80211-MIB::ulWLanMCS14Packets.0 | awk -v x=4 '{print $x}'`
ulmcs15=`snmpget -v 2c -c communityname 1.2.3.4 CAMBIUM-PMP80211-MIB::ulWLanMCS15Packets.0 | awk -v x=4 '{print $x}'`
echo $rssi
echo $noisefloor
echo $ulmcs14
echo $ulmcs15
rrdtool update ./somefile.rrd --template \
rssi:noisefloor N:$rssi:$noisefloor \
mcsul15:mcsul14 N:$ulmcs15:$ulmcs14
Which gives me:
68
94
143679
17602658
ERROR: ./somefile.rrd: expected 2 data source readings (got 1) from mcsul15
What am I missing?
Assuming that somefile.rrd has 4 DS defined in it with those 4 names, you should give all four together when updating. You can only specify one template for the update, and the other parameters should be in that format.
Also, check the names of your DS are correct as your variable is called $ulmcs15 but the DS is being named mcsul15.
rrdtool update ./somefile.rrd --template \
rssi:noisefloor:mcsul15:mcsul14 \
N:$rssi:$noisefloor:$ulmcs15:$ulmcs14
The error message is because in your original commandline, mcsul15:mcsul14 is being taken as an update vector, not a template. Thus it is one timestamp and one value, where two were expected. It would have been a better error message to say something like "timestamp not recognised in 'mcsul15'" but that's a different issue...
Suppose I've written the following:
enum class Color { Red, Green, Blue, };
template <Color c> Color foo() { return c; }
template Color foo<Color::Green>();
and compiled it. When I look at an objdump of my compiled code, I get:
[einpoklum#myhost /tmp]$ objdump -t f.o | grep "\.text\." | sed 's/^.*\.text\.//;' | c++filt
Color foo<(Color)1>()
Color foo<(Color)1>() 000000000000000b Color foo<(Color)1>()
And if I use abi::__cxa_demangle() for <cxxabi.h> (GCC; maybe it's different with your compiler), it's also similar - (Color)0 or Color)1 are the template parameters, not Red or Green nor Color::Red or Color::Green.
Obviously, I can't have names mangled the way I like them. But - I would really like to be able to obtain (or write?) a variant of the demangling call which instead of "Color foo<(Color)1>()" returns "Color foo<(Color:Green>()" (or "Color foo<(Green>()". Is this doable?
It might be possible for object files with debug info - section .debug_info contains info about enum class Color, it requires some tool to read ELF debug info, parse data semantically and apply/pass info to the c++filt. I don't know if such tools exist or not (maybe, in the GDB it is all glued together)
It is pretty much impossible in general with object files compiled with optimization, or with stripped debug info - information about enum class Color is just NOT there...
From optimized build
objdump -s aaa.o
aaa.o: file format pe-x86-64
Contents of section .text$_Z3fooIL5Color1EES0_v:
0000 554889e5 b8010000 005dc390 90909090 UH.......]......
Contents of section .xdata$_Z3fooIL5Color1EES0_v:
0000 01040205 04030150 .......P
Contents of section .pdata$_Z3fooIL5Color1EES0_v:
0000 00000000 0b000000 00000000 ............
Contents of section .rdata$zzz:
0000 4743433a 20287838 365f3634 2d706f73 GCC: (x86_64-pos
0010 69782d73 65682d72 6576302c 20427569 ix-seh-rev0, Bui
0020 6c742062 79204d69 6e47572d 57363420 lt by MinGW-W64
0030 70726f6a 65637429 20352e33 2e300000 project) 5.3.0..
Debug build has partial contents of section .debug_info:
0070 00000000 00000000 00000002 436f6c6f ............Colo
0080 720004a3 00000001 01a30000 00035265 r.............Re
0090 64000003 47726565 6e000103 426c7565 d...Green...Blue
00a0 00020004 0405696e 74000566 6f6f3c28 ......int..foo<(
00b0 436f6c6f 7229313e 0001065f 5a33666f Color)1>..._Z3fo
00c0 6f494c35 436f6c6f 72314545 53305f76 oIL5Color1EES0_v
00d0 007b0000 00000000 00000000 000b0000 .{..............
00e0 00000000 00019c06 63007b00 00000100 ........c.{.....
00f0 00
I have a input file which looks like
1S6290615260715DUTCH-ALDI ROTTERDAM, EUDOKIAPLEIN 8 00002961999
20000010019149GRANEN 0000000100000001590 0000111
20000010019592ALASKA KOOLVISFILET 0000001270000024003 0000111
20000010022614PAPRIKA 3 ST 0000000460000005934 0000111
1S6290615260715DUTCH-ALDI BERGEN NH, JAN OLDENBURGLAAN 00002962888
20000000000404BLEEKMIDDEL 0000000900000003150 0000222
20000000005197FRUIT 0000000430000005977 0000222
20000000006013ROOIBOSTHEE 0000000140000001246 0000222
1S6290615260715DUTCH-ALDI DWINGELOO, HEUVELENWEG 00002963777
20000000006469PITABROODJES 0000000610000004209 0000333
20000000007372SCHENKSTROOP 0000000210000001869 0000333
20000000007545HUISVUILZAKKEN 0000001080000012852 0000333
1S6290615260715DUTCH-ALDI BARNEVELD, CATHARIJNESTEEG 00002964666
20000000005197FRUIT + GRANEN BISCUITS 0000000720000010008 0000444
20000000005209IJSASSORTI MINIMIX 0000000190000003781 0000444
20000000006013ROOIBOSTHEE 0000000210000001869 0000444
I need the break this file into multiple files based on the pattern match. In this file the pattern line begins with 1S6290615260715, based on that I need to create multiple files like
File 1:
1S6290615260715DUTCH-ALDI ROTTERDAM, EUDOKIAPLEIN 8 00002961999
20000010019149GRANEN 0000000100000001590 0000111
20000010019592ALASKA KOOLVISFILET 0000001270000024003 0000111
20000010022614PAPRIKA 3 ST 0000000460000005934 0000111
File 2
1S6290615260715DUTCH-ALDI BERGEN NH, JAN OLDENBURGLAAN 00002962888
20000000000404BLEEKMIDDEL 0000000900000003150 0000222
20000000005197FRUIT 0000000430000005977 0000222
20000000006013ROOIBOSTHEE 0000000140000001246 0000222
and so on.
Using awk i tried this command
awk '/^1S/f++ {print $0 > "file"f}' input.txt
with this each file is created with single line.
Please suggest the faster processing way either with sed or awk, because I need to do this for very larger files like 15GB to 20GB and provide these split files to hadoop framework for further processing.
You can use this awk:
awk '/^1S/{if (f) close(f); f = "file" ++i} {print > f}' file
i would like to gzip log files but i cannot work out how to run a regex expression in my command.
My Log file look like this, they roll every hour.
-rw-r--r-- 1 aus nds 191353 Sep 28 01:59 fubar.log.20150928-01
-rw-r--r-- 1 aus nds 191058 Sep 28 02:59 fubar.log.20150928-02
-rw-r--r-- 1 aus nds 190991 Sep 28 03:59 fubar.log.20150928-03
-rw-r--r-- 1 aus nds 191388 Sep 28 04:59 fubar.log.20150928-04
script.
FUBAR_DATE=$(date -d "days ago" +"%Y%m%d ")
fubar_file="/apps/fubar/logs/fubar.log."$AUS_DATE"-^[0-9]"
/bin/gzip $fubar_file
i have tried a few varients on using the regex but without success, can you see the simple error in my code.
Thanks in advace
I did:
$ fubar_file="./fubar.log."${FUBAR_DATE%% }"-[0-9][0-9]"
and it worked for me.
Why not make fubar_file an array to hold the matching log file names, and then use a loop to gzip them individually. Then presuming AUS_DATE contains 20150928:
# FUBAR_DATE=$(date -d "days ago" +"%Y%m%d ") # not needed for gzip
fubar_file=( /apps/fubar/logs/fubar.log.$AUS_DATE-[0-9][0-9] )
for i in ${fubar_file[#]}; do
gzip "$i"
done
or if you do not need to preserve the filenames in the array for later use, just gzip the files with a for loop:
for i in /apps/fubar/logs/fubar.log.$AUS_DATE-[0-9][0-9]; do
gzip "$i"
done
or, simply use find to match the files and gzip them:
find /apps/fubar/logs -type f -name "fubar.log.$AUS_DATE-[0-9][0-9]" -execdir gzip '{}' +
Note: all answers presume AUS_DATE contains 20150928.
after some investigation I have managed to find my wanted regular expression to work on /bin/sh - busybox:
INPUT:
Mar 8 09:58:29 mysuperhost kern.alert kernel: Rejected OUT -- IN=br0 OUT=vlan2 SRC=192.168.1.8 DST=3.26.211.8 LEN=95 TOS=0x00 PREC=0x00 TTL=127 ID=648 PROTO=UDP SPT=22008 DPT=51413 LEN=75
REGEXP:
grep -o -E '((^.{0,16})|(IN=.\S*)|(IN=.\S*)|(OUT=.\S*)|(SRC=.\S*)|(DST=.\S*)|(PROTO=.\S*)|(SPT=.\S*)|(DPT=.\S*))'
Which gives me:
Mar 8 09:58:29
IN=br0
OUT=vlan2
SRC=192.168.1.8
DST=3.26.211.8
PROTO=UDP
SPT=22008
DPT=51413
The problem:
I don't seem to be able to have grep giving me the result on a single line
The wanted result:
Mar 8 09:58:29 IN=br0 OUT=vlan2 SRC=192.168.1.8 DST=3.26.211.8 PROTO=UDP SPT=22008 DPT=51413
Here is an awk
awk '{printf "%s %s %s ",$1,$2,$3;for (i=4;i<=NF;i++) if ($i~/(IN|OUT|SRC|DST|PROTO|SPT|DPT)=/) printf "%s ",$i;print ""}' file
Mar 8 09:58:29 IN=br0 OUT=vlan2 SRC=192.168.1.8 DST=3.26.211.8 PROTO=UDP SPT=22008 DPT=51413
Just change whats in the if test to select fields you like.