Why is this sed command only working on every other match? - regex

Question

Here's a sed command, works great, just on every other line (simplified for your convenience):
cat testfile.txt | sed -E "/PATTERN/,/^>/{//!d;}"
if my testfile.txt is
>PATTERN
1
2
3
>PATTERN
a
b
c
>PATTERN
1
2
3
>PATTERN
a
b
c
>asdf
1
2
3
>asdf
a
b
c
Expected output:
>PATTERN
>PATTERN
>PATTERN
>PATTERN
>asdf
1
2
3
>asdf
a
b
c
actual output:
>PATTERN
>PATTERN
a
b
c
>PATTERN
>PATTERN
a
b
c
>asdf
1
2
3
>asdf
a
b
c
-An aisde-
(The actual goal is to find a one of a group of patterns then delete the stuff that comes after it until the next occurence of a ">" symbol {also delete that line which I can do by piping to a grep -v})
I more or less got guidance by following what I found here. I've had this work for me. Here's an exact example (not that you have the file to look at it)
for line in $(cat bad_results.txt)
do
echo "removing $line"
cat 16S.fasta | sed "/$line/,/^>/{//!d;}" | grep $line -v > temp_stor.fasta
done

Answer 1

/PATTERN/,/^>/ will match from a line containing PATTERN to a line starting with > (which can be a line containing PATTERN). You should instead match an empty line, like so:
$ sed '/PATTERN/,/^$/{/PATTERN/!d}' ip.txt
>PATTERN
>PATTERN
>PATTERN
>PATTERN
>asdf
1
2
3
>asdf
a
b
c
Your aside isn't very clear to me, but if you want to delete the line with PATTERN as well, you can simplify it to:
$ sed '/PATTERN/,/^$/d' ip.txt
>asdf
1
2
3
>asdf
a
b
c
You can also use:
awk -v RS= -v ORS='\n\n' '!/PATTERN/'
but it will have an extra empty line at the end of the output. The advantage is that instead of your for loop, you can do this:
awk 'BEGIN{FS="\n"; ORS="\n\n"}
NR==FNR{a[">" $0]; next}
!($1 in a)' bad_results.txt RS= 16S.fasta
The above code stores each line of bad_results.txt in an associative array, with > character prefixed. And then, contents of 16S.fasta will be printed only if entire line starting with > isn't present in bad_results.txt.
If you want a partial match:
awk 'BEGIN{FS="\n"; ORS="\n\n"}
NR==FNR{a[$0]; next}
{for (k in a) if(index($1, k)) next; print}' bad_results.txt RS= 16S.fasta

Answer 2

In your range pattern match, the second element 'consumes' the line so that the start of the range no longer sees that block as a match. This is why you apparently have 'skipping.' This can be fixed by using a lookahead that does not consume characters to match. Unfortunately, sed lacks lookaheads.
Perl is really a better choice than sed for complex multi line matches involving lookaheads.
Here is a Perl that reads the file and applies the regex /(?:^>PATTERN)|(?:^>[\s\S]*?)(?=\v?^>|\z)/ (Demo) to it:
$ perl -0777 -lnE 'while(/(?:^>PATTERN)|(?:^>[\s\S]*?)(?=\v?^>|\z)/gm) { say $& }' file
>PATTERN
>PATTERN
>PATTERN
>PATTERN
>asdf
1
2
3
>asdf
a
b
c
Aside: Please read Looping through the content of a file in Bash. The way you are doing it is not idea. Specifically, read here on the side effects of using cat in a Bash loop.

Answer 3

This might work for you (GNU sed):
sed -E '/PATTERN/{p;:a;$!{N;/\n>/!s/\n//;ta};D}' file
As has been already stated, the range operator matches from PATTERN to a line beginning >. The latter line may also contain PATTERN but is not matched, hence the alternating pattern.
The solution above, does not use the range operator but instead gathers the lines from the first containing PATTERN to the line before a line beginning >.
If a line contains PATTERN it is printed, then subsequent lines are collected until the end-of-file or a line begins >.
Within this collection, newlines are removed - essentially making the first line in the pattern space the concatenation of one or more lines.
On a match (or end-of-file) this long line is removed and any line still in the pattern space is processed as if it had been read in as part of the normal sed cycle.
N.B. The difference between the d and the D commands is the d command deletes the pattern space and immediately begins the next sed cycle which involves reading in the next line of input. Whereas the D command removes everything up to and including the first newline in the pattern space and then begins the next sed cycle. However if the pattern space is not empty, the reading in of the next line from the input is forgone and then the sed cycle resumed.
An alternative:
sed '/^>/{h;/^>PATTERN/p};G;/\n>PATTERN/!P;d' file

Answer 4

To answer the question as to why it seemed to be skipping every other occurrence
(as fleshed out in the comments of Sundeep's answer. See his answer to work around this)
The apparent skipping was just an illusion. sed is greedy; it found the first occurrence of PATTERN and up to and including the next line starting with a >. It then deletes everything between (as instructed). sed then continues where it left off and as such doesn't "see" that last line as a new occurrence
to be clear:
>PATTERN <--- sed see's the first occurrence here------------------|
a |(this whole
a |chunk is
a |considered
|by sed)
>PATTERN <--- then matches up to here (the next occurence of ">")--|
b <--- then continues from here "missing" the match of PATTERN above
b
b
>PATTERN
c
c
c