How to remove certain prefix in every word separate by space? which I want to remove the prefix of abc and def from the beginning of the string. I have the sed statement which make it so long. Don't know if can make it shorter and simplier
Sed: sed -e 's/, /,/g' -e 's/'.yaml$'//g' -e 's/^abc_//g' -e 's/^def_//g' -e 's/,abc_/,/g' -e 's/,def_/,/g'
Input: abc_mscp_def.yaml_v1, def_mscp_abc.yaml_v2, abc_mscp_abc.yaml_v2, def_mscp_def.yaml_v2
Output: mscp_def_v1,mscp_abc_v2,mscp_abc_v2,mscp_def_v2
You may use
sed -E 's/(^|,) ?(abc|def)_|(,) |\.yaml/\1\3/g'
See the online demo:
s="abc_mscp_def.yaml_v1, def_mscp_abc.yaml_v2, abc_mscp_abc.yaml_v2, def_mscp_def.yaml_v2"
sed -E 's/(^|,) ?(abc|def)_|(,) |\.yaml/\1\3/g' <<< "$s"
# => mscp_def_v1,mscp_abc_v2,mscp_abc_v2,mscp_def_v2
Details
-E option enables POSIX ERE syntax and alternation
(^|,) ?(abc|def)_|(,) |\.yaml - matches:
(^|,) ?(abc|def)_ - Group 1: start of string or comma, then an optional space, and then Group 2: either abc or def
| - or
(,) - Group 3: a comma, and then a space
| - or
\.yaml - .yaml substring.
The replacement is \1\3, i.e. the values of Group 1 and 3 concatenated.
Related
I am trying to replace part of the string, but can not find a proper regex for sed to execute it properly.
I have a string
/abc/foo/../bar
And I would like to achive the following result:
/abc/bar
I have tried to do it using this command:
echo $string | sed 's/\/[^:-]*\..\//\//'
But as result I am getting just /bar.
I understand that I must use group, but I just do not get it.
Could you, please, help me to find out this group that could be used?
You can use
#!/bin/bash
string='/abc/foo/../bar'
sed -nE 's~^(/[^/]*)(/.*)?/\.\.(/[^/]*).*~\1\3~p' <<< "$string"
See the online demo. Details:
-n - suppresses default line output
E - enables POSIX ERE regex syntax
^ - start of string
(/[^/]*) - Group 1: a / and then zero or more chars other than /
(/.*)? - an optional group 2: a / and then any text
/\.\. - a /.. fixed string
(/[^/]*) - Group 3: a / and then zero or more chars other than /
.* - the rest of the string.
\1\3 replaces the match with Group 1 and 3 values concatenated
p only prints the result of successful substitution.
You can use a capture group for the first part and then match until the last / to remove.
As you are using / to match in the pattern, you can opt for a different delimiter.
#!/bin/bash
string="/abc/foo/../bar"
sed 's~\(/[^/]*/\)[^:-]*/~\1~' <<< "$string"
The pattern in parts:
\( Capture group 1
/[^/]*/ Match from the first till the second / with any char other than / in between
\) Close group 1
[^:-]*/ Match optional chars other than : and - then match /
Output
/abc/bar
Using sed
$ sed 's#^\(/[^/]*\)/.*\(/\)#\1\2#' input_file
/abc/bar
or
$ sed 's#[^/]*/[^/]*/##2' input_file
/abc/bar
Using awk
string='/abc/foo/../bar'
awk -F/ '{print "/"$2"/"$NF}' <<< "$string"
#or
awk -F/ 'BEGIN{OFS=FS}{print $1,$2,$NF}' <<< "$string"
/abc/bar
Using bash
string='/abc/foo/../bar'
echo "${string%%/${string#*/*/}}/${string##*/}"
/abc/bar
Using any sed:
$ echo "$string" | sed 's:\(/[^/]*/\).*/:\1:'
/abc/bar
I have strings like these:
/my/directory/file1_AAA_123_k.txt
/my/directory/file2_CCC.txt
/my/directory/file2_KK_45.txt
So basically, the number of underscores is not fixed. I would like to extract the string between the first underscore and the dot. So the output should be something like this:
AAA_123_k
CCC
KK_45
I found this solution that works:
string='/my/directory/file1_AAA_123_k.txt'
tmp="${string%.*}"
echo $tmp | sed 's/^[^_:]*[_:]//'
But I am wondering if there is a more 'elegant' solution (e.g. 1 line code).
With bash version >= 3.0 and a regex:
[[ "$string" =~ _(.+)\. ]] && echo "${BASH_REMATCH[1]}"
You can use a single sed command like
sed -n 's~^.*/[^_/]*_\([^/]*\)\.[^./]*$~\1~p' <<< "$string"
sed -nE 's~^.*/[^_/]*_([^/]*)\.[^./]*$~\1~p' <<< "$string"
See the online demo. Details:
^ - start of string
.* - any text
/ - a / char
[^_/]* - zero or more chars other than / and _
_ - a _ char
\([^/]*\) (POSIX BRE) / ([^/]*) (POSIX ERE, enabled with E option) - Group 1: any zero or more chars other than /
\. - a dot
[^./]* - zero or more chars other than . and /
$ - end of string.
With -n, default line output is suppressed and p only prints the result of successful substitution.
With your shown samples, with GNU grep you could try following code.
grep -oP '.*?_\K([^.]*)' Input_file
Explanation: Using GNU grep's -oP options here to print exact match and to enable PCRE regex respectively. In main program using regex .*?_\K([^.]*) to get value between 1st _ and first occurrence of .. Explanation of regex is as follows:
Explanation of regex:
.*?_ ##Matching from starting of line to till first occurrence of _ by using lazy match .*?
\K ##\K will forget all previous matched values by regex to make sure only needed values are printed.
([^.]*) ##Matching everything till first occurrence of dot as per need.
A simpler sed solution without any capturing group:
sed -E 's/^[^_]*_|\.[^.]*$//g' file
AAA_123_k
CCC
KK_45
If you need to process the file names one at a time (eg, within a while read loop) you can perform two parameter expansions, eg:
$ string='/my/directory/file1_AAA_123_k.txt.2'
$ tmp="${string#*_}"
$ tmp="${tmp%%.*}"
$ echo "${tmp}"
AAA_123_k
One idea to parse a list of file names at the same time:
$ cat file.list
/my/directory/file1_AAA_123_k.txt.2
/my/directory/file2_CCC.txt
/my/directory/file2_KK_45.txt
$ sed -En 's/[^_]*_([^.]+).*/\1/p' file.list
AAA_123_k
CCC
KK_45
Using sed
$ sed 's/[^_]*_//;s/\..*//' input_file
AAA_123_k
CCC
KK_45
This is easy, except that it includes the initial underscore:
ls | grep -o "_[^.]*"
I have a string, for example home/JOHNSMITH-4991-common-task-list, and I want to take out the uppercase part and the numbers with the hyphen between them. I echo the string and pipe it to sed like so, but I keep getting all the hyphens I don't want, e.g.:
echo home/JOHNSMITH-4991-common-task-list | sed 's/[^A-Z0-9-]//g'
gives me:
JOHNSMITH-4991---
I need:
JOHNSMITH-4991
How do I ignore all but the first hyphen?
You can use
sed 's,.*/\([^-]*-[^-]*\).*,\1,'
POSIX BRE regex details:
.* - any zero or more chars
/ - a / char
\([^-]*-[^-]*\) - Group 1: any zero or more chars other than -, a hyphen, and then again zero or more chars other than -
.* - any zero or more chars
The replacement is the Group 1 placeholder, \1, to restore just the text captured.
See the online demo:
#!/bin/bash
s="home/JOHNSMITH-4991-common-task-list"
sed 's,.*/\([^-]*-[^-]*\).*,\1,' <<< "$s"
# => JOHNSMITH-4991
1st solution: With awk it will be much easier and we could keep it simple, please try following, written and tested with your shown samples.
echo "echo home/JOHNSMITH-4991-common-task-list" | awk -F'/|-' '{print $2"-"$3}'
Explanation: Simple explanation would be, setting field separator as / OR - and printing 2nd field - and 3rd field of current line.
2nd solution: Using match function of awk program here.
echo "echo home/JOHNSMITH-4991-common-task-list" |
awk '
match($0,/\/[^-]*-[^-]*/){
print substr($0,RSTART+1,RLENGTH-1)
}'
3rd solution: Using GNU grep solution here. Using -oP option of grep here, to print matched values with o option and to enable ERE(extended regular expression) with P option. Then in main program of grep using .*/ followed by \K to ignore previous matched part and then mentioning [^-]*-[^-]* to make sure to get values just before 2nd occurrence of - in matched line.
echo "echo home/JOHNSMITH-4991-common-task-list" | grep -oP '.*/\K[^-]*-[^-]*'
Here is a simple alternative solution using cut with bash string substitution:
s='home/JOHNSMITH-4991-common-task-list'
cut -d- -f1-2 <<< "${s##*/}"
JOHNSMITH-4991
You could match until the first occurrence of the /, then clear the match buffer with \K and then repeat the character class 1+ times with a hyphen in between to select at least characters before and after the hyphen.
[^/]*/\K[A-Z0-9]+-[A-Z0-9]+
If supported, using gnu grep:
echo "echo home/JOHNSMITH-4991-common-task-list" | grep -oP '[^/]*/\K[A-Z0-9]+-[A-Z0-9]+'
Output
JOHNSMITH-4991
If gnu awk is an option, using the same pattern but with a capture group:
echo "home/JOHNSMITH-4991-common-task-list" | awk 'match($0, /[^\/]*\/([A-Z0-9]+-[A-Z0-9]+)/, a) {print a[1]}'
If the desired output is always the first match where the character class with a hyphen matches:
echo "home/JOHNSMITH-4991-common-task-list" | awk -v FPAT="[A-Z0-9]+-[A-Z0-9]+" '$0=$1'
Output
JOHNSMITH-4991
Assumptions:
could be more than one fwd slash in string
(after the last fwd slash) there are 2 or more hyphens in the string
desired output is between last fwd slash and 2nd hyphen
One idea using parameter substitutions:
$ string='home/dir/JOHNSMITH-4991-common-task-list'
$ string1="${string##*/}"
$ typeset -p string1
declare -- string1="JOHNSMITH-4991-common-task-list"
$ string1="${string1%%-*}"
$ typeset -p string1
declare -- string1="JOHNSMITH"
$ string2="${string#*-}"
$ typeset -p string2
declare -- string2="4991-common-task-list"
$ string2="${string2%%-*}"
$ typeset -p string2
declare -- string2="4991"
$ newstring="${string1}-${string2}"
$ echo "${newstring}"
JOHNSMITH-4991
NOTES:
typeset commands added solely to show progression of values
a bit of typing but if doing this a lot of times in bash the overall performance should be good compared to other solutions that require spawning a sub-process
if there's a need to parse a large number of strings best performance will come from streaming all strings at once (via a file?) to one of the other solutions (eg, a single awk call that processes all strings will be faster than running the set of strings through a bash loop and performing all of these parameter substitutions)
I have the string like this feature/test-111-test-test.
I need to extract string till the second dash and change forward slash to dash as well.
I have to do it in Makefile using shell syntax and there for me doesn't work some regular expression which can help or this case
Finally I have to get smth like this:
input - feature/test-111-test-test
output - feature-test-111- or at least feature-test-111
feature/test-111-test-test | grep -oP '\A(?:[^-]++-??){2}' | sed -e 's/\//-/g')
But grep -oP doesn't work in my case. This regexp doesn't work as well - (.*?-.*?)-.*.
Another sed solution using a capture group and regex/pattern iteration (same thing Socowi used):
$ s='feature/test-111-test-test'
$ sed -E 's/\//-/;s/^(([^-]*-){3}).*$/\1/' <<< "${s}"
feature-test-111-
Where:
-E - enable extended regex support
s/\//-/ - replace / with -
s/^....*$/ - match start and end of input line
(([^-]-){3}) - capture group #1 that consists of 3 sets of anything not - followed by -
\1 - print just the capture group #1 (this will discard everything else on the line that's not part of the capture group)
To store the result in a variable:
$ url=$(sed -E 's/\//-/;s/^(([^-]*-){3}).*$/\1/' <<< "${s}")
$ echo $url
feature-test-111-
You can use awk keeping in mind that in Makefile the $ char in awk command must be doubled:
url=$(shell echo 'feature/test-111-test-test' | awk -F'-' '{gsub(/\//, "-", $$1);print $$1"-"$$2"-"}')
echo "$url"
# => feature-test-111-
See the online demo. Here, -F'-' sets the field delimiter as -, gsub(/\//, "-", $1) replaces / with - in Field 1 and print $1"-"$2"-" prints the value of --separated Field 1 and 2.
Or, with a regex as a field delimiter:
url=$(shell echo 'feature/test-111-test-test' | awk -F'[-/]' '{print $$1"-"$$2"-"$$3"-"}')
echo "$url"
# => feature-test-111-
The -F'[-/]' option sets the field separator to - and /.
The '{print $1"-"$2"-"$3"-"}' part prints the first, second and third value with a separating hyphen.
See the online demo.
To get the nth occurrence of a character C you don't need fancy perl regexes. Instead, build a regex of the form "(anything that isn't C, then C) for n times":
grep -Eo '([^-]*-){2}' | tr / -
With sed and cut
echo feature/test-111-test-test| cut -d'-' -f-2 |sed 's/\//-/'
Output
feature-test-111
echo feature/test-111-test-test| cut -d'-' -f-2 |sed 's/\//-/;s/$/-/'
Output
feature-test-111-
You can use the simple BRE regex form of not something then that something which is [^-]*- to get all characters other than - up to a -.
This works:
echo 'feature/test-111-test-test' | sed -nE 's/^([^/]*)\/([^-]*-[^-]*-).*/\1-\2/p'
feature-test-111-
Another idea using parameter expansions/substitutions:
s='feature/test-111-test-test'
tail="${s//\//-}" # replace '/' with '-'
# split first field from rest of fields ('-' delimited); do this 3x times
head="${tail%%-*}" # pull first field
tail="${tail#*-}" # drop first field
head="${head}-${tail%%-*}" # pull first field; append to previous field
tail="${tail#*-}" # drop first field
head="${head}-${tail%%-*}-" # pull first field; append to previous fields; add trailing '-'
$ echo "${head}"
feature-test-111-
A short sed solution, without extended regular expressions:
sed 's|\(.*\)/\([^-]*-[^-]*\).*|\1-\2|'
I need to edit following string
UA399_GTTTCG_L002_R1_001.file.gz
to
UA399_GTTTCG_L002
I tried sed 's/^\(.*\)_.*/\1/' but that produces
UA399_GTTTCG_L002_R1
Can someone help please?
You may use either sed or awk or cut like:
s='UA399_GTTTCG_L002_R1_001.file.gz'
awk -F'_' '{print $1"_"$2"_"$3}' <<< "$s"
sed 's/^\([^_]*\(_[^_]*\)\{2\}\).*/\1/' <<< "$s"
cut -d_ -f1,2,3 <<< "$s"
See the online demo. In both cases, the result is UA399_GTTTCG_L002.
sed details
^ - start of a line
\([^_]*\(_[^_]*\)\{2\}\) - Capturing group 1 matching:
[^_]* - 0+ chars other than _
\(_[^_]*\)\{2\} - two consecutive occurrences (\{2\}) of:
_ - an underscore
[^_]* - 0+ chars other than _
.* - the rest of the line.
The replacement is just the placeholder that contains the Group 1 text.
awk details
-F'_' splits the record into underscore separated fields
print $1"_"$2"_"$3 prints only the first, second and third fields and concats them with _ char.
cut details
-d_ sets the field delimiter to _ char
-f1,2,3 selects only the specified set of fields (it may also accept a range of fields as in Kent's solution).
you can simply use cut for this problem:
cut -d'_' -f1-3 <<<STRING
With your example:
kent$ cut -d'_' -f1-3 <<<"UA399_GTTTCG_L002_R1_001.file.gz"
UA399_GTTTCG_L002
With GNU sed:
sed -E 's/((_*[^_]*){3}).*/\1/' file
This might work for you (GNU sed):
sed 's/_[^_]*//3g' file
Delete the third (or more) set(s) of characters beginning with an underscore and followed by zero or more non-underscores.