Hi I am new using perl and I would like to know how to use it to replace regex in multline mode. And if possible to also make the "." match with break lines.
I am using the following expression:
perl -pe 's/text.*end/textChanged/g' myFile.txt
The expression above replace in single line mode. It does not consider break lines.
Using:
Windows
Stranberry Perl
Note that Perl one-liner uses -p or -n switch which wraps a while loop behind the scenes.
And the while loop uses scalar context which reads line by line, so you will not see any changes in your output unless the text.*end appears in a single line.
Here is a sample
$ cat a.txt
abc
text 1 2
2 3 4
ab end
hello
here
$ perl -pe 's/text.*end/textChanged/g' a.txt # Nothing happens - while reads line by line
abc
text 1 2
2 3 4
ab end
hello
here
Now, you can do like setting the Record separator variable to undef.
$ perl -pe ' BEGIN { $/=undef } s/text.*end/textChanged/g' a.txt # Nothing happens
abc
text 1 2
2 3 4
ab end
hello
here
But, when you add the /s modifier, the substitution takes place.
$ perl -pe ' BEGIN { $/=undef } s/text.*end/textChanged/gs ' a.txt
abc
textChanged
hello
here
$
Reading the entire file using slurp mode and again nothing happens with your substitution.
$ perl -0777 -pe ' s/text.*end/textChanged/g ' a.txt
abc
text 1 2
2 3 4
ab end
hello
here
$
Now you use the /s flag so that dot can match the newline as well and the substitution takes place.
$ perl -0777 -pe ' s/text.*end/textChanged/gs ' a.txt
abc
textChanged
hello
here
$
Thanks #ikegami... for the bundle options, like below
$ perl -0777pe ' s/text.*end/textChanged/gs ' a.txt
So when you want the dot to match newlines, you need to add the /s modifier in the regex.
Related
Given a file, for example:
potato: 1234
apple: 5678
potato: 5432
grape: 4567
banana: 5432
sushi: 56789
I'd like to grep for all lines that start with potato: but only pipe the numbers that follow potato:. So in the above example, the output would be:
1234
5432
How can I do that?
grep 'potato:' file.txt | sed 's/^.*: //'
grep looks for any line that contains the string potato:, then, for each of these lines, sed replaces (s/// - substitute) any character (.*) from the beginning of the line (^) until the last occurrence of the sequence : (colon followed by space) with the empty string (s/...// - substitute the first part with the second part, which is empty).
or
grep 'potato:' file.txt | cut -d\ -f2
For each line that contains potato:, cut will split the line into multiple fields delimited by space (-d\ - d = delimiter, \ = escaped space character, something like -d" " would have also worked) and print the second field of each such line (-f2).
or
grep 'potato:' file.txt | awk '{print $2}'
For each line that contains potato:, awk will print the second field (print $2) which is delimited by default by spaces.
or
grep 'potato:' file.txt | perl -e 'for(<>){s/^.*: //;print}'
All lines that contain potato: are sent to an inline (-e) Perl script that takes all lines from stdin, then, for each of these lines, does the same substitution as in the first example above, then prints it.
or
awk '{if(/potato:/) print $2}' < file.txt
The file is sent via stdin (< file.txt sends the contents of the file via stdin to the command on the left) to an awk script that, for each line that contains potato: (if(/potato:/) returns true if the regular expression /potato:/ matches the current line), prints the second field, as described above.
or
perl -e 'for(<>){/potato:/ && s/^.*: // && print}' < file.txt
The file is sent via stdin (< file.txt, see above) to a Perl script that works similarly to the one above, but this time it also makes sure each line contains the string potato: (/potato:/ is a regular expression that matches if the current line contains potato:, and, if it does (&&), then proceeds to apply the regular expression described above and prints the result).
Or use regex assertions: grep -oP '(?<=potato: ).*' file.txt
grep -Po 'potato:\s\K.*' file
-P to use Perl regular expression
-o to output only the match
\s to match the space after potato:
\K to omit the match
.* to match rest of the string(s)
sed -n 's/^potato:[[:space:]]*//p' file.txt
One can think of Grep as a restricted Sed, or of Sed as a generalized Grep. In this case, Sed is one good, lightweight tool that does what you want -- though, of course, there exist several other reasonable ways to do it, too.
This will print everything after each match, on that same line only:
perl -lne 'print $1 if /^potato:\s*(.*)/' file.txt
This will do the same, except it will also print all subsequent lines:
perl -lne 'if ($found){print} elsif (/^potato:\s*(.*)/){print $1; $found++}' file.txt
These command-line options are used:
-n loop around each line of the input file
-l removes newlines before processing, and adds them back in afterwards
-e execute the perl code
You can use grep, as the other answers state. But you don't need grep, awk, sed, perl, cut, or any external tool. You can do it with pure bash.
Try this (semicolons are there to allow you to put it all on one line):
$ while read line;
do
if [[ "${line%%:\ *}" == "potato" ]];
then
echo ${line##*:\ };
fi;
done< file.txt
## tells bash to delete the longest match of ": " in $line from the front.
$ while read line; do echo ${line##*:\ }; done< file.txt
1234
5678
5432
4567
5432
56789
or if you wanted the key rather than the value, %% tells bash to delete the longest match of ": " in $line from the end.
$ while read line; do echo ${line%%:\ *}; done< file.txt
potato
apple
potato
grape
banana
sushi
The substring to split on is ":\ " because the space character must be escaped with the backslash.
You can find more like these at the linux documentation project.
Modern BASH has support for regular expressions:
while read -r line; do
if [[ $line =~ ^potato:\ ([0-9]+) ]]; then
echo "${BASH_REMATCH[1]}"
fi
done
grep potato file | grep -o "[0-9].*"
Why is there a dollar sign at the starting of line 2 and line 3?
➜ echo -e "hello\nworld" | perl -pe 's/$/\$/g'
hello$
$world$
$%
Above, I am trying to add a dollar sign at the end of each line, but somehow it's appending a dollar sign at the beginning too. It does that when global flag is enabled. But when I remove the global flag, it works fine:
➜ echo -e "hello\nworld" | perl -pe 's/$/\$/'
hello$
world$
Can anyone explain what's happening? Maybe it has something to do with '\r\n' characters?
EDIT : Adding the lookbehind case
It's not just breaking in this cases, but other cases as well. Consider the following:
➜ echo -e "A\nB\nC\nD" | perl -pe 's/(?<!A)$/\$/'
A
$B$
C$
D$
Above, I want to mark rows which don't end in "A" with $.
The extra dollar sign in line 2 shouldn't be there. I'm not even using global flag.
SOLUTION : Okay got it now. The solution for second one is like this (for explanation, refer to Wiktor Stribiżew's answer)
➜ echo -e "A\nB\nC\nD" | perl -pe 's/(?<!A|\n)$/\$/'
A
B$
C$
D$
But beware, if you try with more than single characters, it will throw
Variable length lookbehind not implemented in regex. For example:
➜ echo -e "AA\nBB\nCC\nDD" | perl -pe 's/(?<!AA|\n)$/\$/'
Variable length lookbehind not implemented in regex m/(?<!AA|\n)$/ at -e line 1.
To solve this, add the appropriate number of . before newline.
➜ echo -e "AA\nBB\nCC\nDD" | perl -pe 's/(?<!AA|.\n)$/\$/'
AA
BB$
CC$
DD$
The point is that $ is a zero-width assertion and it can match before a final newline. Perl reads a line with a trailing \n, so $ matches twice: before and after that.
Your string basically goes to Perl as two lines:
hello\n
world\n
And the $ can match both before a final newline and at the very end of the string. Thus, there are two matches in both lines ("strings" in this context).
If you want to match the very end of string, use \z:
perl -pe 's/\z/\$/g'
since \z only matches the very end of the string, but it is not likely anyone would want to use that since it will effectively insert a $ at the start of the second and subsequent lines, adding it as the final line as well.
To only insert $ before the last \n and stop, use your perl -pe 's/$/\$/', with no g modifier.
If you really want to use it with the global replace, you can use the following command:
echo -e "hello\nworld" | perl -pe 's/^(.*)$/\1\$/g'
hello$
world$
or without back-references you can use:
echo -e "hello\nworld" | perl -pe 's/\n$/\$\n/g'
hello$
world$
you might need to replace \n by \r\n if you manipulate a file from windows or just use dos2unix to remove Windows EOL chars \r.
Question
Let's say I have one line of text with a number placed somewhere (it could be at the beginning, in the middle or at the end of the line).
How to match and keep the first number found in a line using sed?
Minimal example
Here is my attempt (following this page of a tutorial on regular expressions) and the output for different positions of the number:
$echo "SomeText 123SomeText" | sed 's:.*\([0-9][0-9]*\).*:\1:'
3
$echo "123SomeText" | sed 's:.*\([0-9][0-9]*\).*:\1:'
3
$echo "SomeText 123" | sed 's:.*\([0-9][0-9]*\).*:\1:'
3
As you can only the last digit is kept in the process whereas the desired output should be 123...
Using sed:
echo "SomeText 123SomeText 456" | sed -r 's/^[^0-9]*([0-9]+).*$/\1/'
123
You can also do this in gnu awk:
echo "SomeText 123SomeText 456" | awk '{print gensub(/^[^0-9]*([0-9]+).*$/, "\\1", $0)}'
123
To complement the sed solutions, here's an awk alternative (assuming that the goal is to extract the 1st number on each line, if any (i.e., ignore lines without any numbers)):
awk -F'[^0-9]*' '/[0-9]/ { print ($1 != "" ? $1 : $2) }'
-F'[^0-9]*' defines any sequence of non-digit chars. (including the empty string) as the field separator; awk automatically breaks each input line into fields based on that separator, with $1 representing the first field, $2 the second, and so on.
/[0-9]/ is a pattern (condition) that ensures that output is only produced for lines that contain at least one digit, via its associated action (the {...} block) - in other words: lines containing NO number at all are ignored.
{ print ($1!="" ? $1 : $2) } prints the 1st field, if nonempty, otherwise the 2nd one; rationale: if the line starts with a number, the 1st field will contain the 1st number on the line (because the line starts with a field rather than a separator; otherwise, it is the 2nd field that contains the 1st number (because the line starts with a separator).
You can also use grep, which is ideally suited to this task. sed is a Stream EDitor, which is only going to indirectly give you what you want. With grep, you only have to specify the part of the line you want.
$ cat file.txt
SomeText 123SomeText
123SomeText
SomeText 123
$ grep -o '[0-9]\+' file.txt
123
123
123
grep -o prints only the matching parts of a line, each on a separate line. The pattern is simple: one or more digits.
If your version of grep is compatible with the -P switch, you can use Perl-style regular expressions and make the command even shorter:
$ grep -Po '\d+' file.txt
123
123
123
Again, this matches one or more digits.
Using grep is a lot simpler and has the advantage that if the line doesn't match, nothing is printed:
$ echo "no number" | grep -Po '\d+' # no output
$ echo "yes 123number" | grep -Po '\d+'
123
edit
As pointed out in the comments, one possible problem is that this won't only print the first matching number on the line. If the line contains more than one number, they will all be printed. As far as I'm aware, this can't be done using grep -o.
In that case, I'd go with perl:
perl -lne 'print $1 if /.*?(\d+).*/'
This uses lazy matching (the question mark) so only non-digit characters are consumed by the .* at the start of the pattern. The $1 is a back reference, like \1 in sed. If there are more than one number on the line, this only prints the first. If there aren't any at all, it doesn't print anything:
$ echo "no number" | perl -ne 'print "$1\n" if /.*?(\d+).*/'
$ echo "yes123number456" | perl -lne 'print $1 if /.*?(\d+).*/'
123
If for some reason you still really want to use sed, you can do this:
sed -n 's/^[^0-9]*\([0-9]\{1,\}\).*$/\1/p'
unlike the other answers, this is compatible with all version of sed and will only print lines that contain a match.
Try this sed command,
$echo "SomeText 123SomeText" | sed -r '/[^0-9]*([0-9][0-9]*)[^0-9]*/ s//\1 /g'
123
Another example,
$ echo "SomeText 123SomeText 456" | sed -r '/[^0-9]*([0-9][0-9]*)[^0-9]*/ s//\1 /g'
123 456
It prints all the numbers in a file and the captured numbers are separated by spaces while printing.
I have a file something like:
# ID 1
blah blah
blah blah
$ description 1
blah blah
# ID 2
blah
$ description 2
blah blah
blah blah
How can I use a sed command to delete all lines between the # and $ line? So the result will become:
# ID 1
$ description 1
blah blah
# ID 2
$ description 2
blah blah
blah blah
Can you please kindly give an explanation as well?
Use this sed command to achieve that:
sed '/^#/,/^\$/{/^#/!{/^\$/!d}}' file.txt
Mac users (to prevent extra characters at the end of d command error) need to add semicolons before the closing brackets
sed '/^#/,/^\$/{/^#/!{/^\$/!d;};}' file.txt
OUTPUT
# ID 1
$ description 1
blah blah
# ID 2
$ description 2
blah blah
blah blah
Explanation:
/^#/,/^\$/ will match all the text between lines starting with # to lines starting with $. ^ is used for start of line character. $ is a special character so needs to be escaped.
/^#/! means do following if start of line is not #
/^$/! means do following if start of line is not $
d means delete
So overall it is first matching all the lines from ^# to ^\$ then from those matched lines finding lines that don't match ^# and don't match ^\$ and deleting them using d.
$ cat test
1
start
2
end
3
$ sed -n '1,/start/p;/end/,$p' test
1
start
end
3
$ sed '/start/,/end/d' test
1
3
In general form, if you have a file with contents of form abcde, where section a precedes pattern b, then section c precedes pattern d, then section e follows, and you apply the following sed commands, you get the following results.
In this demonstration, the output is represented by => abcde, where the letters show which sections would be in the output. Thus, ae shows an output of only sections a and e, ace would be sections a, c, and e, etc.
Note that if b or d appear in the output, those are the patterns appearing (i.e., they're treated as if they're sections in the output).
Also don't confuse the /d/ pattern with the command d. The command is always at the end in these demonstrations. The pattern is always between the //.
sed -n -e '/b/,/d/!p' abcde => ae
sed -n -e '/b/,/d/p' abcde => bcd
sed -n -e '/b/,/d/{//!p}' abcde => c
sed -n -e '/b/,/d/{//p}' abcde => bd
sed -e '/b/,/d/!d' abcde => bcd
sed -e '/b/,/d/d' abcde => ae
sed -e '/b/,/d/{//!d}' abcde => abde
sed -e '/b/,/d/{//d}' abcde => ace
Another approach with sed:
sed '/^#/,/^\$/{//!d;};' file
/^#/,/^\$/: from line starting with # up to next line starting with $
//!d: delete all lines except those matching the address patterns
I did something like this long time ago and it was something like:
sed -n -e "1,/# ID 1/ p" -e "/\$ description 1/,$ p"
Which is something like:
-n suppress all output
-e "1,/# ID 1/ p" execute from the first line until your pattern and p (print)
-e "/\$ description 1/,$ p" execute from the second pattern until the end and p (print).
I might be wrong with some of the escaping on the strings, so please double check.
The example below removes lines between "if" and "end if".
All files are scanned, and lines between the two matching patterns are removed ( including them ).
IFS='
'
PATTERN_1="^if"
PATTERN_2="end if"
# Search for the 1st pattern in all files under the current directory.
GREP_RESULTS=(`grep -nRi "$PATTERN_1" .`)
# Go through each result
for line in "${GREP_RESULTS[#]}"; do
# Save the file and line number where the match was found.
FILE=${line%%:*}
START_LINE=`echo "$line" | cut -f2 -d:`
# Search on the same file for a match of the 2nd pattern. The search
# starts from the line where the 1st pattern was matched.
GREP_RESULT=(`tail -n +${START_LINE} $FILE | grep -in "$PATTERN_2" | head -n1`)
END_LINE="$(( $START_LINE + `echo "$GREP_RESULT" | cut -f1 -d:` - 1 ))"
# Remove lines between first and second match from file
sed -e "${START_LINE},${END_LINE}d;" $FILE > $FILE
done
How do I display data from the beginning of a file until the first occurrence of a regular expression?
For example, if I have a file that contains:
One
Two
Three
Bravo
Four
Five
I want to start displaying the contents of the file starting at line 1 and stopping when I find the string "B*". So the output should look like this:
One
Two
Three
perl -pe 'last if /^B/' source.txt
An explanation: the -p switch adds a loop around the code, turning it into this:
while ( <> ) {
last if /^B.*/; # The bit we provide
print;
}
The last keyword exits the surrounding loop immediately if the condition holds - in this case, /^B/, which indicates that the line begins with a B.
if its from the start of the file
awk '/^B/{exit}1' file
if you want to start from specific line number
awk '/^B/{exit}NR>=10' file # start from line 10
sed -n '1,/^B/p'
Print from line 1 to /^B/ (inclusive). -n suppresses default echo.
Update: Opps.... didn't want "Bravo", so instead the reverse action is needed ;-)
sed -n '/^B/,$!p'
/I3az/
sed '/^B/,$d'
Read that as follows: Delete (d) all lines beginning with the first line that starts with a "B" (/^B/), up and until the last line ($).
Some of the sed commands given by others will continue to unnecessarily process the input after the regex is found which could be quite slow for large input. This quits when the regex is found:
sed -n '/^Bravo/q;p'
in Perl:
perl -nle '/B.*/ && last; print; ' source.txt
Just sharing some answers I've received:
Print data starting at the first line, and continue until we find a match to the regex, then stop:
<command> | perl -n -e 'print "$_" if 1 ... /<regex>/;'
Print data starting at the first line, and continue until we find a match to the regex, BUT don't display the line that matches the regular expression:
<command> | perl -pe '/<regex>/ && exit;'
Doing it in sed:
<command> | sed -n '1,/<regex>/p'
Your problem is a variation on an answer in perlfaq6: How can I pull out lines between two patterns that are themselves on different lines?.
You can use Perl's somewhat exotic .. operator (documented in perlop):
perl -ne 'print if /START/ .. /END/' file1 file2 ...
If you wanted text and not lines, you would use
perl -0777 -ne 'print "$1\n" while /START(.*?)END/gs' file1 file2 ...
But if you want nested occurrences of START through END, you'll run up against the problem described in the question in this section on matching balanced text.
Here's another example of using ..:
while (<>) {
$in_header = 1 .. /^$/;
$in_body = /^$/ .. eof;
# now choose between them
} continue {
$. = 0 if eof; # fix $.
}
Here is a perl one-liner:
perl -pe 'last if /B/' file
If Perl is a possibilty, you could do something like this:
% perl -0ne 'if (/B.*/) { print $`; last }' INPUT_FILE
one liner with basic shell commands:
head -`grep -n B file|head -1|cut -f1 -d":"` file