I have a file where you want to delete line matching by pattern and remove strings above and below.
By example:
FFFFIFIBBFFFFFFFFFFFFFBBBBFBBBBFBBBB77<<BBBBBB7B<BBBBBB<B<
#HISEQ:102:h9u5badxx:1:1101:13002:2147 1:N:0:CTGT
GATCCCCGTCTATCAGATACACGTTACTCAGCTAGTGCGAATGCGAACGCGAAATTTT
+
FFFFFFFFBBFFFFFFFFFFFFFBFBFFFFFFFFFBFFFBFFFFFBFFFFFFFFFBFB
#HISEQ:102:h9u5badxx:1:1101:15368:2194 1:N:0:CTGT
+
FFIFBFFIFFBBBFFFFFFFBBFFBFFBBBFFFBB7BBBBBBFFFBB700<7770<BBB0<0<BFFBFBFFFFF
#HISEQ:102:h9u5badxx:1:1101:19167:2169 1:N:0:CTGT
GATCTCATATAGGGCAGCGTGGTCGCGGC
I want to remove second block which does not contain the nucleotide sequence.
The end result:
`FFFFIFIBBFFFFFFFFFFFFFBBBBFBBBBFBBBB77<<BBBBBB7B<BBBBBB<B<
#HISEQ:102:h9u5badxx:1:1101:13002:2147 1:N:0:CTGT
GATCCCCGTCTATCAGATACACGTTACTCAGCTAGTGCGAATGCGAACGCGAAATTTT
+
FFIFBFFIFFBBBFFFFFFFBBFFBFFBBBFFFBB7BBBBBBFFFBB700<7770<BBB0<0<BFFBFBFFFFF
#HISEQ:102:h9u5badxx:1:1101:19167:2169 1:N:0:CTGT
GATCTCATATAGGGCAGCGTGGTCGCGGC
`
Pattern which matched this block
'^.+$(\n)^(#HISEQ).*$(\n)^\+'
works in perl and javascript, but not sed.
Because sed does not work with line break.
I found the solution
sed -e ':a;N;$!ba;s/\n/ /' test
But this code replace line break to space. If insert to this code my regexp:
sed -e ':a;N;$!ba;/^.+$(\n)^(#HISEQ).*$(\n)^\+/d' test
this does not work.
Can you help me find the solution of this problem?
I'm just stupid. I misunderstood the file format.
Input:
#HWI-ST383:199:D1L73ACXX:3:1101:1309:1956 1:N:0:ACAGTGA
+
JJJHIIJFIJJJJ=BFFFFFEEEEEEDDDDDDDDDDBD
#HWI-ST383:199:D1L73ACXX:3:1101:3437:1952 1:N:0:ACAGTGA
GATCTCGAAGCAAGAGTACGACGAGTCGGGCCCCTCCA
+
IIIIFFF<?6?FAFEC#=C#1AE###############
How to edit the regular exp to get what you want
output:
#HWI-ST383:199:D1L73ACXX:3:1101:3437:1952 1:N:0:ACAGTGA
GATCTCGAAGCAAGAGTACGACGAGTCGGGCCCCTCCA
+
IIIIFFF<?6?FAFEC#=C#1AE###############
If I understand you correctly, then
sed ':loop; N; /\n+/ ! { $ ! b loop }; /\n#HISEQ[^\n]\+\n+/ d' foo.txt
will work. This is as follows:
:loop # in a loop
N # fetch more lines
/\n+/ ! { $ ! b loop } # until one starts with + or is the last line
/\n#HISEQ[^\n]\+\n+/ d # if the penultimate line of all that begins with #HISEQ,
# discard the lot.
That last pattern is using the fact that it is checked right after the first line that begins with + is found, so the \n+ at the end of it uniquely matches the start of the last line in the block.
To remove the second block, you can just do:
awk 'NR!=2' RS=+ ORS=+ input
But I would suspect you want something more like:
awk '/[GATC]{5,}\n/' RS=+ ORS=+ input
or
awk '/\n[GATC]*\n/' RS=+ ORS=+ input
Easier to do this parsing using awk:
awk -v RS=+ -v ORS=+ '!/\n#HISEQ[^\n]*\n$/' file
FFFFIFIBBFFFFFFFFFFFFFBBBBFBBBBFBBBB77<<BBBBBB7B<BBBBBB<B<7BBBBFFFBBBBFBBBBBBBFBFFFFB<<
#HISEQ:102:h9u5badxx:1:1101:13002:2147 1:N:0:CTGT
GATCCCCGTCTATCAGATACACGTTACTCAGCTAGTGCGAATGCGAACGCGAAATTTT
+
FFIFBFFIFFBBBFFFFFFFBBFFBFFBBBFFFBB7BBBBBBFFFBB700<7770<BBB0<0<BFFBFBFFFFF<B<7<<BBBBFB0
#HISEQ:102:h9u5badxx:1:1101:19167:2169 1:N:0:CTGT
GATCTCATATAGGGCAGCGTGGTCGCGGC
+
sed '/FFFFFFFFBBFFFFFFFFFFFFFBFBFFFFFFFFFBFFFBFFFFFBFFFFFFFFFBFB/,/\+/ d' YourFile
should be enough
Related
I'm trying to create a script to append oracleserver to /etc/hosts as an alias of localhost. Which means I need to:
Locate the line that ^127.0.0.1 and NOT oracleserver$
Then, append oracleserver to this line
I know the best practice is probably using negative look ahead. However, sed does not have look around feature: What's wrong with my lookahead regex in GNU sed?. Can anyone provide me some possible solutions?
sed -i '/oracleserver$/! s/^127\.0\.0\.1.*$/& oracleserver/' filename
/oracleserver$/! - on lines not ending with oracleserver
^127\.0\.0\.1.*$ - replace the whole line if it is starting with 127.0.0.1
& oracleserver - with the line plus a space separator ' ' (required) and oracleserver after that
Just use awk with && to combine the two conditions:
awk '/^127\.0\.0\.1/ && !/oracleserver$/ { $0 = $0 "oracleserver" } 1' file
This appends the string when the first pattern is matched but the second one isn't. The 1 at the end is always true, so awk prints each line (the default action is { print }).
I wouldn't use sed but instead perl:
Locate the line that ^127.0.0.1 and NOT oracleserver$
perl -pe 'if ( m/^127\.0\.0\.1/ and not m/oracleserver$/ ) { s/$/oracleserver/ }'
Should do the trick. You can add -i.bak to inplace edit too.
I have a simple awk script named "script.awk" that contains:
/\/some_simple_string/ { print $0;}
I'm using it to parse some file that contains:
(by using: cat file | awk -f script.awk)
14 catcat one_two/some_thing
15 catcat one_three/one_more_some_simple_string
16 dogdog one_two/some_simple_string_again
17 dogdog one_four/some_simple_string
18 qweqwe firefire/ppp
I want the script to only print the stroke that fully reflect "/some_simple_string[END_OF_LINE]" but not 2 or 3.
Is there any simple way to do it?
I think, the most appropriate way is to add end-of-line sigh to the regular expression.
So it will parse only strokes that starting with "/some.." and have a new line at the end of "..string[END_OF_LINE]"
Desired output:
17 dogdog one_four/some_simple_string
Sorry for confusion, I was asking for END OF LINE sign in regular expressions.
The correct answer is:
/\/some_simple_string$/ { print $0;}
You can always use:
/\/some_simple_string$/ { print $0 }
I.e. match not only "some_simple_string" but match "/some_simple_string" followed by the end of the line ($ is end of line in regex)
grep '\some_simple_string$' file | tail -n 1 should do the trick.
Or if you really want to use awk do awk '/\/some_simple_string/{x = $0}END{print x}'
To return just the last of a group of matches, ...
Store the line in a variable and print it in the END block.
/some_simple_string/ { x = $0 }
END{ print x }
To print all the matches that end with the string /some_simple_string using regular expression you need to anchor to the the end of the line using $. The most suitable tool for this job is grep:
$ grep '/some_simple_string$' file
In awk the command is much the same:
$ awk '/[/]some_simple_string$/' file
To print all lines after the matching you would do:
$ awk 'print_flag{print;f=0} /[/]some_simple_string$/{print_flag=1}' file
Or just combine grep and tail if it makes it clearer using context option -A to print the following lines:
$ grep -A1 '/some_simple_string$' file | tail -n 1
I sometimes find that the input records can have a trailing carriage return (\r).
Yes, I deal with both Windows and Linux text files.
So I add the following 'pre-processor' to my awk scripts:
1 == 1 { # preprocess all records
res = gsub("\r", "") # remove unwanted trailing char
if(res>0 && NR<100) { print "(removed stuff)" > "/dev/stderr" } # optional
}
more optimally, let FS do the work instead of having it perform unnecessary and unrelated field splitting (adding the \r bit for Windows/DOS completeness):
mawk '!_<NF' FS='[/]some_simple_string[\r]?$'
17 dogdog one_four/some_simple_string
I am trying to filter out text between two patterns, I've seen a dozen examples but didn't manage to get exactly what I want:
Sample input:
START LEAVEMEBE text
data
START DELETEME text
data
more data
even more
START LEAVEMEBE text
data
more data
START DELETEME text
data
more
SOMETHING that doesn't start with START
# sometimes it starts with characters that needs to be escaped...
I want to stay with:
START LEAVEMEBE text
data
START LEAVEMEBE text
data
more data
SOMETHING that doesn't start with START
# sometimes it starts with characters that needs to be escaped...
I tried running sed with:
sed 's/^START DELETEME/,/^[^ ]/d'
And got an inclusive removal, I tried adding "exclusions" (not sure if I really understand this syntax well):
sed 's/^START DELETEME/,/^[^ ]/{/^[^ ]/!d}'
But my "START DELETEME" line is still there (yes, I can grep it out, but that's ugly :) and besides - it DOES remove the empty line in this sample as well and I'd like to leave empty lines if they are my end pattern intact )
I am wondering if there is a way to do it with a single sed command.
I have an awk script that does this well:
BEGIN { flag = 0 }
{
if ($0 ~ "^START DELETEME")
flag=1
else if ($0 !~ "^ ")
flag=0
if (flag != 1)
print $0
}
But as you know "A is for awk which runs like a snail". It takes forever.
Thanks in advance.
Dave.
Using a loop in sed:
sed -n '/^START DELETEME/{:l n; /^[ ]/bl};p' input
GNU sed
sed '/LEAVEMEBE/,/DELETEME/!d;{/DELETEME/d}' file
I would stick with awk:
awk '
/LEAVE|SOMETHING/{flag=1}
/DELETE/{flag=0}
flag' file
But if you still prefer sed, here's another way:
sed -n '
/LEAVE/,/DELETE/{
/DELETE/b
p
}
' file
I have a pattern like
Fixed pattern
text which can change(world)
I want to replace this with
Fixed pattern
text which can change(hello world)
What I am trying to use
cat myfile | sed -e "s#\(Fixed Pattern$A_Z_a_z*\(\)#\1 hello#g > newfile
UPDATE:
The above word world is also a variable and will change
Basically add hello after the first parenthesis encountered after the expression.
Thanks in advance.
Assuming your goal is to add 'hello ' inside of every opening parentheses on the line after 'Fixed pattern', here is a solution that should work:
sed -e '/^Fixed pattern$/!b' -e 'n' -e 's/(/(hello /' myfile
Here is an explanation of each portion:
/^Fixed pattern$/!b # skip all of the following commands if 'Fixed pattern'
# doesn't match
n # if 'Fixed pattern' did match, read the next line
s/(/(hello / # replace '(' with '(hello '
To do this with sed, use n:
sed '/Fixed pattern/{n; s/world/hello world/}' myfile
You may need to be more careful, but this should work for most situations. Whenever sed sees the Fixed pattern (you may want to use line anchors ^ and $), it will read the next line and then apply the substitution to it.
I'm trying to automagically remove all lines from a text file that contains a letter "T" that is not immediately followed by a "H". I've been using grep and sending the output to another file, but I can't come up with the magic regex that will help me do this.
I don't mind using awk, sed, or some other linux tool if grep isn't the right tool to be using.
That should do it:
grep -v 'T[^H]'
-v : print lines not matching
[^H]: matches any character but H
You can do:
grep -v 'T[^H]' input
-v is the inverse match option of grep it does not list the lines that match the pattern.
The regex used is T[^H] which matches any lines that as a T followed by any character other than a H.
Read lines from file exclude EMPTY Lines and Lines starting with #
grep -v '^$\|^#' folderlist.txt
folderlist.txt
# This is list of folders
folder1/test
folder2
# This is comment
folder3
folder4/backup
folder5/backup
Results will be:
folder1/test
folder2
folder3
folder4/backup
folder5/backup
Adding 2 awk solutions to the mix here.
1st solution(simpler solution): With simple awk and any version of awk.
awk '!/T/ || /TH/' Input_file
Checking 2 conditions:
If a line doesn't contain T OR
If a line contains TH then:
If any of above condition is TRUE then print that line simply.
2nd solution(GNU awk specific): Using GNU awk using match function where mentioning regex (T)(.|$) and using match function's array creation capability.
awk '
!/T/{
print
next
}
match($0,/(T)(.|$)/,arr) && arr[1]=="T" && arr[2]=="H"
' Input_file
Explanation: firstly checking if a line doesn't have T then print that simply. Then using match function of awk to match T followed by any character OR end of the line. Since these are getting stored into 2 capturing groups so checking if array arr's 1st element is T and 2nd element is H then print that line.