I have a XML file that I wish to extract all occurrences of some tag AB. The file is one long line with ~500 000 chars.
Now I do know about regexp and such, but when I try it with sed and try to extract only the characters within the tags I am totally lost regarding the result :).
Here's my command:
sed -r 's/(.*)<my_tag>([A-Z][A-Z])<\/my_tag>(.*)/hello\2/g' myfile.out
transforms the entire file with only "helloAB" e.g. While the expected should at least contain 100+ matches.
So I'm thinking around the concepts of greedy matching and such but not getting anywhere. Maybe awk is a better idea?
If you have python (2.6+), this should be fairly trivial:
import xml.dom.minidom as MD
tree = MD.parse("yourfile.xml")
for e in tree.getElementsByTagName("AB"):
print e.toprettyxml()
In general, trying to parse XML by hand should be avoided as there are much simpler solutions like these. Not to mention, these kinds of libraries will give you easy access to attributes and values without further parsing.
Thank your for your answers.
I tried #MannyD's suggestion and unfortunately the XML didn't seem to be well formed, thus the parsing failed. Since I cannot anticipate only well formed XML's I made grep solution, which does the job.
grep -o "<my_tag>[A-Z][A-Z]</my_tag>" myfile.out | sort -u
The -o option flag will print each match on a new line, from there I just sort and print the unique matches from the file.
Related
Before I Start, this isn't exactly how it seems and I did search the web for a while before coming here. Basically I have a script where the user passes in a string and stores it in a variable. I then have to take that word and search for all the subwords that could be made from it in a dictionary file. The problem I am having is I need to make sure the words are at least 4 characters long. I do not have the best grasp on regular expressions. I'm aware of the techniques you can use just logically can't piece it together sometimes. I will show you the line of code and explain my reasoning behind why I think it should be this way. Then, could someone correct me on my logic? I am not looking for someone to send me the working line of code but perhaps correct my logic so I can understand better and derive the answer on my own.
words=$(grep -iE '(["$text"]{4,})' /usr/dict/words)
echo "$words"
For example if I pass in string college I should get output like
cell
cello
clee
cleg
etc.....
I am storing the command in another variable to echo. I am not sure why exactly, It just seems from what I saw online most people were rather fond of this. Using grep with -i for ignore case and -E for regular expression or (egrep) I believe the expression needs to be enclosed in single quote parenthesis for expressions. $text is the variable I stored the users input in. I know $ usually signifies the ending in and [] is a range and "" makes it read the variable rather than print what is there. Then {4,} meaning four or more characters. then the last part is the path to the file. Any input would be appreciated and again, I do not like being spoon fed answers it's an easy way to learn nothing. I would just like corrections on my logic if all possible. Thanks everyone!!
If by "subwords" you mean permutations of its letters, then your command is fine except for the quotes. Unfortunately you have to do it like this:
words=$(grep -iE '(['"$text"']{4,})' /usr/dict/words)
This way you pass to grep the single quoted string so that the shell doesn't interpret its special symbols. But at the same time you have to expand your $text var, thus you have to make a gap inside your single-quoted string, and in that gap place your variable in double quotes.
Hope I didn't spoil it for you.
We have a Samba server that is backing up to an S3 bucket. Come to find out that a large number of file names contain inappropriate characters and the AWS CLI won't allow the transfer of those files. Using the "worst offender" I build a quick regex check, tested in rubular against another file name to try and generate a list of files that need to be fixed:
([ä¸æ–‡ç½‘页我们的团队å™é¹â€“¦]+)
The command I'm running is:
find . -regextype awk -regex ".*/([ä¸æ–‡ç½‘页我们的团队å™é¹â€“¦]+)"
This brings back a small list of files that contain the above string, in order, not individual characters contained throughout the name. This leads me to believe that either my regextype is incorrect or something is wrong with the formatting of the list of characters. I've tried types emacs and egrep as they seem most similar to regex I've used outside of a Unix environment to no luck.
My test file name is: this-is-my€™s'-test-_ folder-name. which, according to my rubular tests, should be returned but isn't. Any help would be greatly appreciated.
Your regex .*/([ä¸æ–‡ç½‘页我们的团队å™é¹â€“¦]+) expects one of the special characters after the slash and your test file doesn't start with one of these characters.
You might try something more like .*[ä¸æ–‡ç½‘页我们的团队å™é¹â€“¦]+.* instead.
I'm desperately trying to search the following:
<texit info> author=MySelf title=MyTitle </texit>
and replace it with blank.
What I've tried so far is the following:
sed –I '1,5s/<texit//;s/info>//;s/author=MySelf//;s/title=MyTitle//' test.txt
But it doesn't work.
Don't edit XML with sed -- the right tool would be something like XMLStarlet, with a line like the following:
xmlstarlet ed -u //texit[#info] -v 'author=NewAuthor title=NewTitle'
...if your goal were to update the text within the tag.
Regular expressions are not expressive enough to correctly handle XML (even formally -- regular expressions are theoretically sufficient to parse regular languages; XML is not one). For instance, your original would be just as valid written with newlines, as:
< texit
info >author=MySelf title=MyTitle</texit>
...and writing a sed command to handle that case would not be fun. XML-native tools, on the other hand, can correctly handle all of XML's corner cases.
That said, the sed expression you gave does indeed "work", inasmuch as it does exactly what it's written to do.
sed -e '1,5s/<texit//;s/info>//;s/author=MySelf//;s/title=MyTitle//' \
<<<"<texit info>author=MySelf title=MyTitle foo bar</texit>"
returns the output
foo bar</texit>
which is exactly what it should do, as it's removing the <texit string, the info> string, the author=MySelf, title=MyTitle, but leaving the closing </texit> and any excess text, just as you asked. If you expect or desire it to do something different, you should explain what that is.
sed 's/<texit\s\+info>\s*author=MySelf\s\+title=MyTitle\s*<\/texit>//g' test.txt
You should generally not edit XML with a regex, but if you only want to strip these tags, the above will work. You don't need multiple s commands, just use a single pattern with correctly defined whitespace.
In my LaTeX files, I have literally thousands of occurrences of the following construct:
$\displaystyle{...math goes here...}$
I'd like to replace these with
\mymath{...math goes here...}
Note that the $'s disappear, but the curly braces remain---if not for the trailing $, this would be a basic find-and-replace. If only I knew any regex, I'm sure it would handle this with no problem. What's the regex I need to make this happen?
Many thanks in advance.
Edit: Some issues and questions have arisen, so let me clarify:
Yes, $\displaystyle{ ... }$ can occur multiple times on the same line.
No, nested }$'s (such as $\displaystyle{...{more math}$...}$) cannot occur. I mean, I suppose it could if you put it in an \mbox or something, but I can't imagine why anyone would ever do that inside a $\displaystlye{}$ construct, the purpose of which is to display math inline with text. At any rate, it's not something I've ever done or am likely to do.
I tried using the perl suggestion, but while the shell raised no objections, the files remained unaffected.
I tried using the sed suggestion, but the shell objected to an "unexpected token near `('". I've never used sed before (and "man sed" was obtuse), but here's what I did: navigated to a directory containing .tex files and typed "sed s/\$\\displaystyle({[^}]+})\$/\\mymath\1/g *.tex". No luck. How do I use sed to do what I want?
Again, many many thanks for all offered help.
Be very careful when using REGEX to do this type of substitution
because the theoretical answer is that
REGEX is incapable of matching this type of pattern.
REGEX is a finite state machine; it does not incorporate a pushdown stack so
it cannot work with nested structures such as "{...math goes here...}" if
there is any possibility of nesting such that something like "{more math}$"
can appear as part of a "math goes here" string. You need at a minimum a
context free grammar to describe this type of construct - a state machine
just doesn't cut it!
Now having said that, you may still be able to pull this off using REGEX
provided none of your "math goes here" strings are more complex than
what a state machine can handle.
Give it a shot.... but beware of the results!
sed:
s/\$\\displaystyle({[^}]+})\$/\\mymath\1/g
perl -pi -e 's/$\\displaystyle({.*)}\$/\\mymath$1}/g' *.tex
if multiples }$ are on the same line you need a non greedy version:
perl -pi -e 's/$\\displaystyle({.*?)}\$/\\mymath$1}/g' *.tex
Using a regex, I am able to find a bunch of numbers that I want to replace. However, I want to replace the number with another number that is calculated using the original - captured - number.
Is that possible in notepad++ using a kind of expression in the replacement-part?
Edit: Maybe a strange thought, but could the calculation be done in the search part, generating a second captured number that would effectively be the result?
Even if it is possible, it will almost certainly be "messy" - why not do the replacements with a simple script instead? For example..
#!/usr/bin/env ruby
f = File.new("f1.txt", File::RDWR)
contents = f.read()
contents.gsub!(/\d+/){|m|
m.to_i + 1 # convert the current match to an integer, and add one
}
f.truncate(0) # empty the existing file
f.seek(0) # seek to the start of the file, before writing again
f.write(contents) # write modified file
f.close()
..and the output:
$ cat f1.txt
This was one: 1
This two two: 2
$ ruby replacer.rb
$ cat f1.txt
This was one: 2
This two two: 3
In reply to jeroen's comment,
I was actually interested if the possibility existed in the regular expression itself as they are so widespread
A regular expression is really just a simple pattern matching syntax. To do anything more advanced than search/replace with the matches would be up to the text-editors, but the usefulness of this is very limited, and can be achieved via scripting most editors allow (Notepad++ has a plugin system, although I've no idea how easy it is to use).
Basically, if regex/search-and-replace will not achieve what you want, I would say either use your editors scripting ability or use an external script.
Is that possible in notepad++ using a kind of expression in the replacement-part?
Interpolated evaluation of regular-expression matches is a relatively advanced feature that I probably would not expect to find in a general-purpose text editing application. I played around with Notepad++ a bit but was unable to get this to work, nor could I find anything in the documentation that suggests this is possible.
Hmmm... I'd have to recommend AWK to do this.
http://en.wikipedia.org/wiki/AWK
notepad++ has limited regular expressions built in. There are extensions that add a bit more to the regular expression find and replace, but I've found those hard to use. I would recommend writing a little external program to do it for you. Either Ruby, Perl or Python would be great for it. If you know those languages. I use Ruby and have had lots of success with it.