I have multiple xml files that look like this: <TEST><TEST><TEST><TEST><TEST><TEST><TEST><TEST><TEST><TEST>
I would like to break into a new like for every '<' and get rid of every '>'.
I want to do this via regex since what I'm working on is for *nix.
There is no need for regex to do such a simple search & replace. You want to replace < with \n< and > with an empty string.
Assuming your content is in file input.txt, this simple sed command line can do the job:
sed 's/</\n</g;s/>//g' input.txt
How it works
There are two sed commands separated by ;:
s/</\n</g
s/>//g
Both commands are s (search and replace). The s command requires the search regex (no regex here), the replacement string and some optional flag, separated by /.
The first s searches for < and replaces it with \n<. \n is the usual notation for a newline character in regex and many Unix tools (even when no regex is involved).
The second s searches for > and replaces it with nothing.
Both s commands use the g (global) flag that tells them to do all the replacements they can do on each line. sed runs each command for every line of the input and by default, s stops after the first replacement (on a line).
Related
How can I parse every line in a .txt file to remove everything after the second occurrence of a / after a given flag jdk on each line of a file.
For example
/usr/lib/jvm/jdk-1.7.0/2.0/zi/etc/GMT
/usr/lib/jvm/jdk1.7.2/3.0/zi/etc/GMT
/usr/share/servertool-java-openjdk/4.0/jce.jar
becomes
/usr/lib/jvm/jdk-1.7.0/2.0/
/usr/lib/jvm/jdk1.7.2/3.0/
/usr/share/servertool-java-openjdk/4.0/
Note, I can't just split on jdk, because it may be jdk-1.*.*/ etc.
My end goal is to find all the unique paths on a highly restricted SeLinux box that has the output of a locate jdk stored in a output.txt file
Update: my attempt so far, to get closer is
cat output.txt | awk -F '\\jdk' '{print $1"jdk"}' | sort -u
This just chops everything after jdk, and removes dupes.
sed is a very appropriate tool for this job. You'll use the s/// command to remove the part of the line you want to delete.
Note the slashes in the s/// command can be changed to other characters so that any slashes you have in the pattern or replacement parts don't need to be escaped.
Your pattern will be:
in capturing parentheses:
"jdk" followed by zero or more non-slashes
followed by a slash
followed by one or more non-slashes
followed by a slash
followed by any number of characters
The replacement will be the text that was captured.
You'll want to refer to the sed manual
3.3 The s Command
5 Regular Expressions: selecting text
5.2 Basic (BRE) and extended (ERE) regular expression
if you want to replace in the same file, you can use below script
#!/bin/bash
cat output.txt | while read line
do
x=${line#/*jdk*/*/}
replace=${line%${x}}
sed -i "s|$line|$replace|g" output.txt
done
While looking for a way to format 'ifconfig' output and display only the network interfaces names, I found a regular expression that worked like a charm for OS X.
ifconfig -a | sed -E 's/[[:space:]:].*//;/^$/d'
How can I breakdown this regular expression so I can understand it?
Here is the sed command
s/[[:space:]:].*//;/^$/d
There is a semicolon in the middle, so it's actually two commands:
s/[[:space:]:].*//
/^$/d
First command is a substitution. What to substitute? It's between the 1st 2 slashes.
[[:space:]:].*
Character class [] of any kind of whitespace or a colon, followed by zero or more * of any character .. This matches everything in a line after the first whitespace or colon.
Substitute with what? Between the 2nd two slashes: s/...//: Nothing. The matched strings are deleted from each line.
This leaves the interface names which start their lines, the other lines remain too, but they are empty, as they start with whitespace.
How to remove these empty lines? That's the second command:
/^$/d
Find empty lines that match regex with nothing between start of line ^ and end of line $. Then delete them with command d.
All that's left are the interface names.
This is more a sequence of commands than it is a regular expression, but I suppose breaking the sequence down may be instructive.
Read the manpage on ifconfig to find this
Optionally, the -a flag may be used instead of an interface name. This
flag instructs ifconfig to display information about all interfaces in
the system. The -d flag limits this to interfaces that are down, and
-u limits this to interfaces that are up. When no arguments are given,
-a is implied.
That's one part done. The pipe (|) sends what ifconfig would normally print to the standard output to the standard input of sed instead.
You're passing sed the option -E. Again, man sed is your friend and tells you that this option means
Interpret regular expressions as extended (modern) regular
expressions rather than basic regular expressions (BRE's). The
re_format(7) manual page fully describes both formats.
This isn't all you need though... The first string that you're giving sed lets it know which operation to perform.
Search the same manual for the word "substitute" to reach this
paragraph:
[2addr]s/regular expression/replacement/flags
Substitute the replacement string for the first instance of
the regular expression in the pattern space. Any character other than
backslash or newline can be used instead of a slash to delimit the RE
and the replacement. Within the RE and the replacement, the RE
delimiter itself can be used as a literal character if it is preceded
by a backslash.
Now we can run man 7 re_format to decode the first command s/[[:space:]:].*// which means "for each line passed to standard input, substitute the part matching the extended regular expression [[:space:]:].* with the empty string"
[[:space:]:] = match either a : or any character in the character class [:space:]
.* = match any character (.), zero or more times (*)
To understand the second command look for the [2addr]d part of the sed manual page.
[2addr]d
Delete the pattern space and start the next cycle.
Let's then look at the next command /^$/d which says "for each line passed to standard input, delete it if it corresponds to the extended regex ^$"
^$ = a line that contains no characters between its start (^) and its end ($)
We've discussed how to start with man pages and follow the clues to "decode" commands you see in everyday life.
Thanks Benjamin and Xufox for the resources. After taking a look, this is my conclusion:
s/[[:space:]:].*//;
[[:space:]:] this will search for spaces and/or : and begin the execution of the command, and this and anything that comes afterwards(hence the '.*') will be substituted by nothing (because the next thing is //, which in between should be what we would want to substitute for, which in this case is nothing.).
;
marks the end of the first command
and then we have
/^$/d
where ^$ means search for all empty spaces and d to delete them.
This is half wrong. Take a look at the other answer which gives you the complete and correct response! Thanks guys.
Using grep or another command line tool I need to filter a list so that every line containing one or more of the following characters are excluded:
.
/
-
'
[space]
I'm having a hard time escaping special characters while searching for multiple expesseions.
This isn't working:
grep -v '(.|/|-|'| )' input > output
By default, the grep command uses "Basic" regular expression format. The regex you've written is in "Extended" format. You can tell grep to use extended format with the -E option.
You've included a dot in your regex. Remember that a dot matches "any" character. To escape its normal behaviour you can either escape it with a backslash (\.) or by putting it in a range ([.]). I prefer the latter notation because I find that backslashes make things more difficult to read. The choice is yours.
You have a single quote in your expression. As you've written it, the command line won't work because the embedded single quote exits the string begun with the first single quote. You can get around this by wrapping your regex in double quotes.
You also don't need the outer brackets with this regex.
So... You could write the whole thing in Basic notation:
grep -v "[.]\|/\|-\|'\| " input > output
Or you could write it in Extended notation:
grep -Ev "[.]|/|-|'| " input > output
Or alternately, you could put ALL these characters into a range, which is written the same way in Basic and Extended:
grep -v "[./' -]" input > output
Note that the hyphen has moved to the END of the range so that it won't be interpreted as "the range of characters between a forward slash and a single quote". Note also that since this range is also compatible with Basic RE notation, I've removed the -E option.
See man re_format(7) for details.
This question already has answers here:
How can I search for a multiline pattern in a file?
(11 answers)
Closed 1 year ago.
I'm running a grep to find any *.sql file that has the word select followed by the word customerName followed by the word from. This select statement can span many lines and can contain tabs and newlines.
I've tried a few variations on the following:
$ grep -liIr --include="*.sql" --exclude-dir="\.svn*" --regexp="select[a-zA-Z0-
9+\n\r]*customerName[a-zA-Z0-9+\n\r]*from"
This, however, just runs forever. Can anyone help me with the correct syntax please?
Without the need to install the grep variant pcregrep, you can do a multiline search with grep.
$ grep -Pzo "(?s)^(\s*)\N*main.*?{.*?^\1}" *.c
Explanation:
-P activate perl-regexp for grep (a powerful extension of regular expressions)
-z Treat the input as a set of lines, each terminated by a zero byte (the ASCII NUL character) instead of a newline. That is, grep knows where the ends of the lines are, but sees the input as one big line. Beware this also adds a trailing NUL char if used with -o, see comments.
-o print only matching. Because we're using -z, the whole file is like a single big line, so if there is a match, the entire file would be printed; this way it won't do that.
In regexp:
(?s) activate PCRE_DOTALL, which means that . finds any character or newline
\N find anything except newline, even with PCRE_DOTALL activated
.*? find . in non-greedy mode, that is, stops as soon as possible.
^ find start of line
\1 backreference to the first group (\s*). This is a try to find the same indentation of method.
As you can imagine, this search prints the main method in a C (*.c) source file.
I am not very good in grep. But your problem can be solved using AWK command.
Just see
awk '/select/,/from/' *.sql
The above code will result from first occurence of select till first sequence of from. Now you need to verify whether returned statements are having customername or not. For this you can pipe the result. And can use awk or grep again.
Your fundamental problem is that grep works one line at a time - so it cannot find a SELECT statement spread across lines.
Your second problem is that the regex you are using doesn't deal with the complexity of what can appear between SELECT and FROM - in particular, it omits commas, full stops (periods) and blanks, but also quotes and anything that can be inside a quoted string.
I would likely go with a Perl-based solution, having Perl read 'paragraphs' at a time and applying a regex to that. The downside is having to deal with the recursive search - there are modules to do that, of course, including the core module File::Find.
In outline, for a single file:
$/ = "\n\n"; # Paragraphs
while (<>)
{
if ($_ =~ m/SELECT.*customerName.*FROM/mi)
{
printf file name
go to next file
}
}
That needs to be wrapped into a sub that is then invoked by the methods of File::Find.
I can't seem to understand regular expression at all. How can I match a character which resides between a START and END string. For Example
#START-EDIT
#ValueA=0
#ValueB=1
#END-EDIT
I want to match any # which is between the #START-EDIT and #END-EDIT.
Specifically I want to use sed to replace the matches # values with nothing (delete them) on various files which may or may not have multiple START-EDIT and END-EDIT sections.
^#START-EDIT.*(#) *. *#END-EDIT$
sed is line based. you can easily search, replace based on regex in one line. But there is no really easy way to search/replace on multilines. AWK might do the trick.
If you have the regex on one line, the following command could be what you are looking for
sed -e "/^#START-EDIT.*#END-EDIT$//" myInput.txt