I would like to print directly with sed a HEX value translation by isolating the HEX values in capture groups. This works:
echo bbb3Accc | sed -n 's/3A/\x3A/p'
bbb:ccc
...but this doesn't work:
echo bbb3Accc | sed 's/\(3A\)/\x\1/'
bbbx3Accc
...or an actual capture group REGEX matching based on URL encoded strings:
echo bbb%3Accc | sed 's/%\([A-Za-z0-9]\)/\x\1/'
bbbx3Accc
Apparently sed no longer interprets and translates the HEX value if it is constructed from a REGEX capture group, together with the \x escape.
But I am wondering if there's a workaround that I am not aware of, to make this work only with sed. Note that I am aware that I can do a bash command substitution and wrap the sed syntax in a echo -e but I would like to avoid that.
Your question isn't clear but maybe this is what you're trying to do using GNU awk for multi-char RS, RT, and strtonum():
$ echo 'bbb%3Accc%21ddd' |
gawk -v RS='%[[:xdigit:]]{2}' 'sub(/%/,"0x",RT){RT=sprintf("%c",strtonum(RT))} {ORS=RT} 1'
bbb:ccc!ddd
As mentioned in the comments, \xAB is interpreted by sed's parser, rather than as an expression, so \x won't work in the way you were trying.
sed is pretty primitive and your example is beyond what it is intended for, so you'd be better off using something more general purpose. For example, in Perl:
$ echo bbb3Accc | perl -ple 's/([0-9A-F]{2})/chr(hex($1))/ge'
bbb:ccc
Related
I'm trying swap words around with sed, not replace because that's what I keep finding on Google search.
I don't know if it's the regex that I'm getting wrong. I did a search for everything before a char and everything after a char, so that's how I got the regex.
echo xxx,aaa | sed -r 's/[^,]*/[^,]*$/'
or
echo xxx/aaa | sed -r 's/[^\/]*/[^\/]*$/'
I am getting this output:
[^,]*$,aaa
or this:
[^,/]*$/aaa
What am I doing wrong?
For the first sample, you should use:
echo xxx,aaa | sed 's/\([^,]*\),\([^,]*\)/\2,\1/'
For the second sample, simply use a character other than slash as the delimiter:
echo xxx/aaa | sed 's%\([^/]*\)/\([^/]*\)%\2/\1%'
You can also use \{1,\} to formally require one or more:
echo xxx,aaa | sed 's/\([^,]\{1,\}\),\([^,]\{1,\}\)/\2,\1/'
echo xxx/aaa | sed 's%\([^/]\{1,\}\)/\([^/]\{1,\}\)%\2/\1%'
This uses the most portable sed notation; it should work anywhere. With modern versions that support extended regular expressions (-r with GNU sed, -E with Mac OS X or BSD sed), you can lose some of the backslashes and use + in place of * which is more precisely what you're after (and parallels \{1,\} much more succinctly):
echo xxx,aaa | sed -E 's/([^,]+),([^,]+)/\2,\1/'
echo xxx/aaa | sed -E 's%([^/]+)/([^/]+)%\2/\1%'
With sed it would be:
sed 's#\([[:alpha:]]\+\)/\([[:alpha:]]\+\)#\2,\1#' <<< 'xxx/aaa'
which is simpler to read if you use extended posix regexes with -r:
sed -r 's#([[:alpha:]]+)/([[:alpha:]]+)#\2/\1#' <<< 'xxx/aaa'
I'm using two sub patterns ([[:alpha:]]+) which can contain one or more letters and are separated by a /. In the replacement part I reassemble them in reverse order \2/\1. Please also note that I'm using # instead of / as the delimiter for the s command since / is already the field delimiter in the input data. This saves us to escape the / in the regex.
Btw, you can also use awk for that, which is pretty easy to read:
awk -F'/' '{print $2,$1}' OFS='/' <<< 'xxx/aaa'
I'm writing a simple script that will take URLs pointing to Evernote notes online, and convert them to the evernote:/// protocol. The regex I'm using matches and modifies the URL correctly when I try it out in a regex tester (I'm using Patterns for OS X). However, when I use it with sed, it just returns the original string.
echo "https://www.evernote.com/shard/s2/nl/227468/1875e55a-e512-4cf9-9b18-9e93c6a27359/" | sed 's#https?:_/_/www_.evernote_.com_/shard_/(..)_/nl_/(......)_/(.+_/)#evernote:_/_/_/view_/$2_/$1_/$3$3#'
Any idea why this isn't working? Thanks!
fort
[Edit: In case anyone's interested, this was for the AppleScript bit of a Keyboard Maestro macro:
set theURL to the clipboard
set ENcode to "echo \"" & theURL & "\" | sed -E 's#https?://www.evernote.com/shard/(..)/nl/(.*)/(.+/)#evernote:///view/\\2/\\1/\\3\\3#' | pbcopy"
do shell script ENcode
Thanks to #DreadPirateShawn for helping me fix the regex.
]
Using the extended regex flag -E, removing the underscores, and replacing each $1 pattern with \1 yields a functional regex here:
$ echo "https://www.evernote.com/shard/s2/nl/227468/1875e55a-e512-4cf9-9b18-9e93c6a27359/" | sed -E 's#https?://www\.evernote\.com/shard/(..)/nl/(......)/(.+/)#evernote:///view/\2/\1/\3\3#'
evernote:///view/227468/s2/1875e55a-e512-4cf9-9b18-9e93c6a27359/1875e55a-e512-4cf9-9b18-9e93c6a27359/
(Confirmed on Ubuntu 12.04 and OS X.)
If you don't use -E, then you also need to change s? to [s]? and escape the grouping parentheses:
$ echo "https://www.evernote.com/shard/s2/nl/227468/1875e55a-e512-4cf9-9b18-9e93c6a27359/" | sed 's#http[s]*://www\.evernote\.com/shard/\(.*\)/nl/\(.*\)/\(.*/\)#evernote:///view/\2/\1/\3\3#'
evernote:///view/227468/s2/1875e55a-e512-4cf9-9b18-9e93c6a27359/1875e55a-e512-4cf9-9b18-9e93c6a27359/
In the latter example, I also replaced each (....)-type sequence with (.*) -- unless you're absolutely positive of the length of each sequence (and even then perhaps), the (.*) approach will be a bit more flexible.
I think you're trying this:
echo "https://www.evernote.com/shard/s2/nl/227468/1875e55a-e512-4cf9-9b18-9e93c6a27359/" | sed -re 's#https://www.evernote.com/shard/(..)/nl/(......)/(.+)/#evernote://view/\2/\1/\3#'
evernote://view/227468/s2/1875e55a-e512-4cf9-9b18-9e93c6a27359
Making no use of Extended regex:
echo "https://www.evernote.com/shard/s2/nl/227468/1875e55a-e512-4cf9-9b18-9e93c6a27359/" | sed 's#https://www.evernote.com/shard/\(..\)/nl/\(......\)/\(.\+\)/#evernote://view/\2/\1/\3#'
evernote://view/227468/s2/1875e55a-e512-4cf9-9b18-9e93c6a27359
Everyone!!
I want to get specific substring from stdout of command.
stdout:
{"response":
{"id":"110200dev1","success":"true","token":"09ad7cc7da1db13334281b84f2a8fa54"},"success":"true"}
I need to get a hex string after token without quotation marks, the length of hex string is 32 letters.I suppose it can be done by sed or egrep. I don't want to use awk here. Because the stdout is being changed very often.
This is an alternate gnu-awk solution when grep -P isn't available:
awk -F: '{gsub(/"/, "")} NF==2&&$1=="token"{print $2}' RS='[{},]' <<< "$string"
09ad7cc7da1db13334281b84f2a8fa54
grep's nature is extracting things:
grep -Po '"token":"\K[^"]+'
-P option interprets the pattern as a Perl regular expression.
-o option shows only the matching part that matches the pattern.
\K throws away everything that it has matched up to that point.
Or an option using sed...
sed 's/.*"token":"\([^"]*\)".*/\1/'
With sed:
your-command | sed 's/.*"token":"\([^"]*\)".*/\1/'
YourStreamOrFile | sed -n 's/.*"token":"\([a-f0-9]\{32\}\)".*/\1/p'
doesn not return a full string if not corresponding
I would like to capture special character from a line:
var=`echo "#this is comment" | grep "[^a-zA-Z0-9 \t]"`
echo $var
Expected Output: #
But getting: #this is comment
Can someone help me out.
It seems like you want something more like:
var=`echo "#this is comment" | sed 's/[^a-zA-Z0-9 \t]//g;'`
Using sed will replace the characters; using grep was only searching for the characters.
Edit: Note that the \t construct is not guaranteed to be portable to all systems or locales; I believe if your sed supports POSIX regular expressions, using [:space:] may work better. (thanks #ghoti!)
string="#this is comment"
var=$(echo "$string" | sed 's/[a-zA-Z0-9 ]//g')
echo "$var"
I've removed \t as it's not portable.
If you want to do this with awk, as your tag suggests, you can use something like:
var=$(echo "$string" | awk '{gsub(/[a-zA-Z0-9 ]/, "")} 1')
Note that these are probably not good ways to achieve whatever it is that you're trying to do. If you post more of your code, showing us some context, we can help you avoid an XY problem.
Of course, you can also do substitutions like this directly in bash, if you want.
var=${string//[A-Za-z0-9 ]}
You'll save CPU and time by avoiding the call to an extra program when you don't really need it.
sed can be used for this, but tr is a better choice:
echo "#this is comment" | tr -d 'a-zA-Z0-9 \t'
tr also supports character classes such as [:space:] and [:alpha:]
I'm dealing with a specific filenames, and need to extract information from them.
The structure of the filename is similar to: "20100613_M4_28007834.005_F_RANDOMSTR.raw.gz"
with RANDOMSTR a string of max 22 chars, and which may contain a substring (or not) with the format "-W[0-9].[0-9]{2}.[0-9]{3}". This substring also has the unique feature of starting with "-W".
The information I need to extract is the substring of RANDOMSTR without this optional substring.
I want to implement this in a bash script, and so far the best option I found is to use gawk with a regular expression. My best attempt so far fails:
gawk --re-interval '{match ($0,"([0-9]{8})_(M[0-9])_([0-9]{8}\\.[0-9]{3})_(.)_(.*)(-W.*)?.raw.gz",arr); print arr[5]}' <<< "20100613_M4_28007834.005_F_OTHER-STRING-W0.40+045.raw.gz"
OTHER-STRING-W0.40+045
The expected results are:
gawk --re-interval '{match ($0,$regexp,arr); print arr[5]}' <<< "20100613_M4_28007834.005_F_SOME-STRING.raw.gz"
SOME-STRING
gawk --re-interval '{match ($0,$regexp,arr); print arr[5]}' <<< "20100613_M4_28007834.005_F_OTHER-STRING-W0.40+045.raw.gz"
OTHER-STRING
How can I get the desired effect.
Thanks.
You need to be able to use look-arounds and I don't think awk/gawk supports that, but grep -P does.
$ pat='(?<=[0-9]{8}_M[0-9]_[0-9]{8}\.[0-9]{3}_._)(.*?)(?=(-W.*)?\.raw\.gz)'
$ echo "20100613_M4_28007834.005_F_SOME-STRING.raw.gz" | grep -Po "$pat"
SOME-STRING
$ echo "20100613_M4_28007834.005_F_OTHER-STRING-W0.40+045.raw.gz" | grep -Po "$pat"
OTHER-STRING
While the grep solution is very nice indeed, the OP didn't mention an operating system, and the -P option only seems to be available in Linux. It's also pretty simple to do this in awk.
$ awk -F_ '{sub(/(-W[0-9].[0-9]+.[0-9]+)?\.raw\.gz$/,"",$NF); print $NF}' <<EOT
> 20100613_M4_28007834.005_F_SOME-STRING.raw.gz
> 20100613_M4_28007834.005_F_OTHER-STRING-W0.40+045.raw.gz
> EOT
SOME-STRING
OTHER-STRING
$
Note that this breaks on "20100613_M4_28007834.005_F_OTHER-STRING-W0_40+045.raw.gz". If this is a risk, and -W only shows up in the place shown above, it might be better to use something like:
$ awk -F_ '{sub(/(-W[0-9.+]+)?\.raw\.gz$/,"",$NF); print $NF}'
The difficulty here seems to be the fact that the (.*) before the optional (-W.*)? gobbles up the latter text. Using a non-greedy match doesn't help either. My regex-fu is unfortunately too weak to combat this.
If you don't mind a multi-pass solution, then a simpler approach would be to first sanitise the input by removing the trailing .raw.gz and possible -W*.
str="20100613_M4_28007834.005_F_OTHER-STRING-W0.40+045.raw.gz"
echo ${str%.raw.gz} | # remove trailing .raw.gz
sed 's/-W.*$//' | # remove trainling -W.*, if any
sed -nr 's/[0-9]{8}_M[0-9]_[0-9]{8}\.[0-9]{3}_._(.*)/\1/p'
I used sed, but you can just as well use gawk/awk.
Wasn't able to get reluctant quantifiers going, but running through two regexes in sequence does the job:
sed -E -e 's/^.{27}(.*).raw.gz$/\1/' << FOO | sed -E -e 's/-W[0-9.]+\+[0-9.]+$//'
20100613_M4_28007834.005_F_SOME-STRING.raw.gz
20100613_M4_28007834.005_F_OTHER-STRING-W0.40+045.raw.gz
FOO