I'm want to replace some bytes in binary file to another.
Created sample (6 bytes long) file via
echo -ne '\x8f\x15\x42\x02\x24\xc2' > test
Then tried to replace bytes \x15\x42\x02 to \x12\x12\x02 via sed:
sed 's \x15\x42\x02 \x12\x12\x02 g' test > test1
sed replaced bytes:
cat test test1 | xxd -c 6
0000000: 8f15 4202 24c2 ..B.$.
0000006: 8f12 1202 24c2 ....$.
^^ ^^^^
Tried then replace bytes \x42\x02\x24 to \x12\x02\x24:
sed 's \x42\x02\x24 \x12\x02\x24 g' test > test2
sed NOT replaced bytes:
cat test test2 | xxd -c 6
0000000: 8f15 4202 24c2 ..B.$.
0000006: 8f15 4202 24c2 ..B.$.
^^^^ ^^
What's wrong? I have sed (GNU sed) 4.2.2 (Kubuntu 13.10)
Thank You.
It is because 0x24 is the hex code for the $ character. Note that ^ and $ have special meaning in regex (which sed uses for matching), and literally means "at the beginning of the line", and "at the end of the line", respectively. As such, those characters are ignored unless escaped. Therefore,
echo "this and that" | sed 's/this$/cat/'
will leave the string unchanged because it you're telling sed to look for 'this' at the end of the line -- since 'that' is at the end, it doesn't match. However,
echo "It costs $50.00." | sed 's/\$50\.00/47.00 Swiss Francs/'
Will change the line as expected because the $ was escaped. For binary data, just include the hex code for the \ character (\x5c) just prior to \x24 in the regex section, and it should work just fine.
Cheers.
You can try this,
hexdump -ve '1/1 "%.2X"' file1 | sed 's/420224/121224/g' | xxd -r -p > new_updated
Test:
sat:~# xxd -c 6 file1
0000000: 8f15 4202 24c2 ..B.$.
sat:~# hexdump -ve '1/1 "%.2X"' file1 | sed 's/420224/121224/g' | xxd -r -p > new_updated
sat:~# xxd -c 6 new_updated
0000000: 8f15 1212 24c2 ....$.
the sed statement should be like below
sed 's/\x15\x42\x02/\x12\x12\x02/g' test > test1
their should be / between the search pattern and the pattern that should replace it
and the result should be like below
[root#localhost ~]# cat test test1 | xxd -c 6
0000000: 8f15 4202 24c2 ..B.$.
0000006: 8f12 1202 24c2 ....$.
Related
I need to get a number of version from file. My version file looks like this:
#define MINOR_VERSION_NUMBER 1
I try to use sed command:
VERSION_MINOR=`sed -i -e 'MINOR_VERSION_NUMBER\s+\([0-9]+\).*/\1/p' $WORKSPACE/project/common/version.h`
but I get error:
sed: -e expression #1, char 2: extra characters after command
The "address" that selects matching lines needs to be enclosed in /.../ (or \X...X for any X).
sed -ne '/MINOR_VERSION_NUMBER/{ s/.*\([0-9]\).*/\1/;p }'
Don't use -i, it changes the file in place and doesn't output anything.
The more common way would be to use awk to find the line and extract the wanted column:
awk '(/MINOR_VERSION_NUMBER/){print$3}'
using grep
grep MINOR_VERSION_NUMBER | grep -o '[0-9]*$'
Demo :
$echo "#define MINOR_VERSION_NUMBER 1" | grep -o '[0-9]*$'
1
$echo "#define MINOR_VERSION_NUMBER 1123" | grep -o '[0-9]*$'
1123
$
Here is a correction of your attempt. Change your line:
VERSION_MINOR=`sed -i -e 'MINOR_VERSION_NUMBER\s+\([0-9]+\).*/\1/p' $WORKSPACE/project/common/version.h`
into:
VERSION_MINOR=`sed -n -e '/^#define\s\+MINOR_VERSION_NUMBER\s\+\([0-9]\+\).*/ s//\1/p' $WORKSPACE/project/common/version.h`
This can be made more readable with GNU sed's -r option:
VERSION_MINOR=`sed -n -r -e '/^#define\s+MINOR_VERSION_NUMBER\s+([0-9]+).*/ s//\1/p' $WORKSPACE/project/common/version.h`
As stated by choroba, awk would be more suited than sed for this kind of processing (see his answer).
However, here is another solution using bash's read builtin, together with GNU grep:
read x x VERSION_MINOR x < <(grep -F -w -m1 MINOR_VERSION_NUMBER $WORKSPACE/project/common/version.h)
VERSION_MINOR=$(echo "#define MINOR_VERSION_NUMBER 1" | tr -s ' ' | cut -d' ' -f3)
I'm trying to use grep -P to find specific byte sequences in potentially large binary files. However, it sometimes matches where it shouldn't - for example, here's a golfed-down case where it appears to simply "match over" a wrong \xc2 byte:
➜ alias bin2hex='xargs echo -n | od -An -tx1'
➜ echo -e '\x3e\x1f\xc2\x9d\xa0' > test.bin
➜ cat test.bin | bin2hex
3e 1f c2 9d a0
➜ grep -P '\x1f\x9d' test.bin
Binary file test.bin matches
➜ grep -Pao '\x1f\x9d' test.bin | bin2hex
1f c2 9d
Why does this happen?
And can it be avoided?
This command:
grep -P '\x1f\x9d' <<< $(echo -e '\x3e\x1f\xc2\x9d\xa0') | xargs echo -n | od -An -tx1
prints nothing with grep versions:
GNU grep 2.5.1
GNU grep 2.6.3
GNU grep 2.21
Are you sure your grep is not aliased to anything wrong (type grep)?
UPDATE: converting comments into answer
I can reproduce your problem with a different LANG value:
LANG=en_US.UTF-8; grep -P '\x1f\x9d' <<< $(echo -e '\x3e\x1f\xc2\x9d\xa0')
Binary file (standard input) matches
The problem is not reproduced with:
LANG=en_US; grep -P '\x1f\x9d' <<< $(echo -e '\x3e\x1f\xc2\x9d\xa0')
I have an input file, an example of which is shown below : (?U0 ?U2 ?U9 ?U11 ?U21) I want to extract all the numbers after ?U to an output file as: 0 2 9 11 21 Please help me in this regard, I am new to it.
Thanks
You could use grep but it produces output in each per line.
grep -oP '\?U\K\d+' file
or
$ echo '(?U0 ?U2 ?U9 ?U11 ?U21)' | grep -oP '\?U\K\d+' | paste -s -d " " -
0 2 9 11 21
Using sed you can do:
s='(?U0 ?U2 ?U9 ?U11 ?U21)'
sed 's/?U\([0-9]\+\)/\1/g; s/[()]//g' <<< "$s"
0 2 9 11 21
simple sed
echo "(?U0 ?U2 ?U9 ?U11 ?U21)" | sed 's/[()?U]//g'
output
0 2 9 11 21
deleting all unneeded characters, you can put in set [...] if needed another characters
or more universal
echo "(?U0 ?U2 ?U9 ?U11 ?U21)" | sed 's/[^0-9 ]*//g'
deleting all non-digit characters (and not space)
With grep:
out="$(grep -oP '(?<=\?U)\d+' filepath |tr -s '\n' ' ')"
out="${out% }"
echo "$out" >outfilepath
I want to count the number of matches there is on one single line (or all lines as there always will be only one line).
I want to count not just one match per line as in
echo "123 123 123" | grep -c -E "123" # Result: 1
Better example:
echo "1 1 2 2 2 5" | grep -c -E '([^ ])( \1){1}' # Result: 1, expected: 2 or 3
You could use grep -o then pipe through wc -l:
$ echo "123 123 123" | grep -o 123 | wc -l
3
Maybe below:
echo "123 123 123" | sed "s/123 /123\n/g" | wc -l
( maybe ugly, but my bash fu is not that great )
Maybe you should convert spaces to newlines first:
$ echo "1 1 2 2 2 5" | tr ' ' $'\n' | grep -c 2
3
Why not use awk?
You could use awk '{print gsub(your_regex,"&")}'
to print the number of matches on each line, or
awk '{c+=gsub(your_regex,"&")}END{print c}'
to print the total number of matches. Note that relative speed may vary depending on which awk implementation is used, and which input is given.
This might work for you:
sed -n -e ':a' -e 's/123//p' -e 'ta' file | sed -n '$='
GNU sed could be written:
sed -n ':;s/123//p;t' file | sed -n '$='
Is there a way to tell sed to output only captured groups?
For example, given the input:
This is a sample 123 text and some 987 numbers
And pattern:
/([\d]+)/
Could I get only 123 and 987 output in the way formatted by back references?
The key to getting this to work is to tell sed to exclude what you don't want to be output as well as specifying what you do want. This technique depends on knowing how many matches you're looking for. The grep command below works for an unspecified number of matches.
string='This is a sample 123 text and some 987 numbers'
echo "$string" | sed -rn 's/[^[:digit:]]*([[:digit:]]+)[^[:digit:]]+([[:digit:]]+)[^[:digit:]]*/\1 \2/p'
This says:
don't default to printing each line (-n)
exclude zero or more non-digits
include one or more digits
exclude one or more non-digits
include one or more digits
exclude zero or more non-digits
print the substitution (p) (on one line)
In general, in sed you capture groups using parentheses and output what you capture using a back reference:
echo "foobarbaz" | sed 's/^foo\(.*\)baz$/\1/'
will output "bar". If you use -r (-E for OS X) for extended regex, you don't need to escape the parentheses:
echo "foobarbaz" | sed -r 's/^foo(.*)baz$/\1/'
There can be up to 9 capture groups and their back references. The back references are numbered in the order the groups appear, but they can be used in any order and can be repeated:
echo "foobarbaz" | sed -r 's/^foo(.*)b(.)z$/\2 \1 \2/'
outputs "a bar a".
If you have GNU grep:
echo "$string" | grep -Po '\d+'
It may also work in BSD, including OS X:
echo "$string" | grep -Eo '\d+'
These commands will match any number of digit sequences. The output will be on multiple lines.
or variations such as:
echo "$string" | grep -Po '(?<=\D )(\d+)'
The -P option enables Perl Compatible Regular Expressions. See man 3 pcrepattern or man 3 pcresyntax.
Sed has up to nine remembered patterns but you need to use escaped parentheses to remember portions of the regular expression.
See here for examples and more detail
you can use grep
grep -Eow "[0-9]+" file
run(s) of digits
This answer works with any count of digit groups. Example:
$ echo 'Num123that456are7899900contained0018166intext' \
| sed -En 's/[^0-9]*([0-9]{1,})[^0-9]*/\1 /gp'
123 456 7899900 0018166
Expanded answer.
Is there any way to tell sed to output only captured groups?
Yes. replace all text by the capture group:
$ echo 'Number 123 inside text' \
| sed 's/[^0-9]*\([0-9]\{1,\}\)[^0-9]*/\1/'
123
s/[^0-9]* # several non-digits
\([0-9]\{1,\}\) # followed by one or more digits
[^0-9]* # and followed by more non-digits.
/\1/ # gets replaced only by the digits.
Or with extended syntax (less backquotes and allow the use of +):
$ echo 'Number 123 in text' \
| sed -E 's/[^0-9]*([0-9]+)[^0-9]*/\1/'
123
To avoid printing the original text when there is no number, use:
$ echo 'Number xxx in text' \
| sed -En 's/[^0-9]*([0-9]+)[^0-9]*/\1/p'
(-n) Do not print the input by default.
(/p) print only if a replacement was done.
And to match several numbers (and also print them):
$ echo 'N 123 in 456 text' \
| sed -En 's/[^0-9]*([0-9]+)[^0-9]*/\1 /gp'
123 456
That works for any count of digit runs:
$ str='Test Num(s) 123 456 7899900 contained as0018166df in text'
$ echo "$str" \
| sed -En 's/[^0-9]*([0-9]{1,})[^0-9]*/\1 /gp'
123 456 7899900 0018166
Which is very similar to the grep command:
$ str='Test Num(s) 123 456 7899900 contained as0018166df in text'
$ echo "$str" | grep -Po '\d+'
123
456
7899900
0018166
About \d
and pattern: /([\d]+)/
Sed does not recognize the '\d' (shortcut) syntax. The ascii equivalent used above [0-9] is not exactly equivalent. The only alternative solution is to use a character class: '[[:digit:]]`.
The selected answer use such "character classes" to build a solution:
$ str='This is a sample 123 text and some 987 numbers'
$ echo "$str" | sed -rn 's/[^[:digit:]]*([[:digit:]]+)[^[:digit:]]+([[:digit:]]+)[^[:digit:]]*/\1 \2/p'
That solution only works for (exactly) two runs of digits.
Of course, as the answer is being executed inside the shell, we can define a couple of variables to make such answer shorter:
$ str='This is a sample 123 text and some 987 numbers'
$ d=[[:digit:]] D=[^[:digit:]]
$ echo "$str" | sed -rn "s/$D*($d+)$D+($d+)$D*/\1 \2/p"
But, as has been already explained, using a s/…/…/gp command is better:
$ str='This is 75577 a sam33ple 123 text and some 987 numbers'
$ d=[[:digit:]] D=[^[:digit:]]
$ echo "$str" | sed -rn "s/$D*($d+)$D*/\1 /gp"
75577 33 123 987
That will cover both repeated runs of digits and writing a short(er) command.
Give up and use Perl
Since sed does not cut it, let's just throw the towel and use Perl, at least it is LSB while grep GNU extensions are not :-)
Print the entire matching part, no matching groups or lookbehind needed:
cat <<EOS | perl -lane 'print m/\d+/g'
a1 b2
a34 b56
EOS
Output:
12
3456
Single match per line, often structured data fields:
cat <<EOS | perl -lape 's/.*?a(\d+).*/$1/g'
a1 b2
a34 b56
EOS
Output:
1
34
With lookbehind:
cat <<EOS | perl -lane 'print m/(?<=a)(\d+)/'
a1 b2
a34 b56
EOS
Multiple fields:
cat <<EOS | perl -lape 's/.*?a(\d+).*?b(\d+).*/$1 $2/g'
a1 c0 b2 c0
a34 c0 b56 c0
EOS
Output:
1 2
34 56
Multiple matches per line, often unstructured data:
cat <<EOS | perl -lape 's/.*?a(\d+)|.*/$1 /g'
a1 b2
a34 b56 a78 b90
EOS
Output:
1
34 78
With lookbehind:
cat EOS<< | perl -lane 'print m/(?<=a)(\d+)/g'
a1 b2
a34 b56 a78 b90
EOS
Output:
1
3478
I believe the pattern given in the question was by way of example only, and the goal was to match any pattern.
If you have a sed with the GNU extension allowing insertion of a newline in the pattern space, one suggestion is:
> set string = "This is a sample 123 text and some 987 numbers"
>
> set pattern = "[0-9][0-9]*"
> echo $string | sed "s/$pattern/\n&\n/g" | sed -n "/$pattern/p"
123
987
> set pattern = "[a-z][a-z]*"
> echo $string | sed "s/$pattern/\n&\n/g" | sed -n "/$pattern/p"
his
is
a
sample
text
and
some
numbers
These examples are with tcsh (yes, I know its the wrong shell) with CYGWIN. (Edit: For bash, remove set, and the spaces around =.)
Try
sed -n -e "/[0-9]/s/^[^0-9]*\([0-9]*\)[^0-9]*\([0-9]*\)[^0-9]*\([0-9]*\)[^0-9]*\([0-9]*\)[^0-9]*\([0-9]*\)[^0-9]*\([0-9]*\)[^0-9]*\([0-9]*\)[^0-9]*\([0-9]*\)[^0-9]*\([0-9]*\).*$/\1 \2 \3 \4 \5 \6 \7 \8 \9/p"
I got this under cygwin:
$ (echo "asdf"; \
echo "1234"; \
echo "asdf1234adsf1234asdf"; \
echo "1m2m3m4m5m6m7m8m9m0m1m2m3m4m5m6m7m8m9") | \
sed -n -e "/[0-9]/s/^[^0-9]*\([0-9]*\)[^0-9]*\([0-9]*\)[^0-9]*\([0-9]*\)[^0-9]*\([0-9]*\)[^0-9]*\([0-9]*\)[^0-9]*\([0-9]*\)[^0-9]*\([0-9]*\)[^0-9]*\([0-9]*\)[^0-9]*\([0-9]*\).*$/\1 \2 \3 \4 \5 \6 \7 \8 \9/p"
1234
1234 1234
1 2 3 4 5 6 7 8 9
$
You need include whole line to print group, which you're doing at the second command but you don't need to group the first wildcard. This will work as well:
echo "/home/me/myfile-99" | sed -r 's/.*myfile-(.*)$/\1/'
It's not what the OP asked for (capturing groups) but you can extract the numbers using:
S='This is a sample 123 text and some 987 numbers'
echo "$S" | sed 's/ /\n/g' | sed -r '/([0-9]+)/ !d'
Gives the following:
123
987
I want to give a simpler example on "output only captured groups with sed"
I have /home/me/myfile-99 and wish to output the serial number of the file: 99
My first try, which didn't work was:
echo "/home/me/myfile-99" | sed -r 's/myfile-(.*)$/\1/'
# output: /home/me/99
To make this work, we need to capture the unwanted portion in capture group as well:
echo "/home/me/myfile-99" | sed -r 's/^(.*)myfile-(.*)$/\2/'
# output: 99
*) Note that sed doesn't have \d
You can use ripgrep, which also seems to be a sed replacement for simple substitutions, like this
rg '(\d+)' -or '$1'
where ripgrep uses -o or --only matching and -r or --replace to output only the first capture group with $1 (quoted to be avoid intepretation as a variable by the shell) two times due to two matches.