Extracting data from a page source. In the extracted data, need to display text after the ". Tried different options. Didn't work. Any suggestions
Page source text
enter image description here
input type name=loginForm_SUBMIT value="1" /input type=""name="faces.ViewState" id="faces.ViewState" value="9uiY/UWJ1/w3PQ==" /><
regular expression: value="[^"1" ].*\w==
Output: value="9uiY/UWJ1/w3PQ==
Expected Output: 9uiY/UWJ1/w3PQ==
Don't parse XML/HTML with regex, use a proper XML/HTML parser and a powerful xpath query.
theory :
According to the compiling theory, XML/HTML can't be parsed using regex based on finite state machine. Due to hierarchical construction of XML/HTML you need to use a pushdown automaton and manipulate LALR grammar using tool like YACC.
realLife©®™ everyday tool in a shell :
You can use one of the following :
xmllint often installed by default with libxml2, xpath1 (check my wrapper to have newlines delimited output
xmlstarlet can edit, select, transform... Not installed by default, xpath1
xpath installed via perl's module XML::XPath, xpath1
xidel xpath3
saxon-lint my own project, wrapper over #Michael Kay's Saxon-HE Java library, xpath3
or you can use high level languages and proper libs, I think of :
python's lxml (from lxml import etree)
perl's XML::LibXML, XML::XPath, XML::Twig::XPath, HTML::TreeBuilder::XPath
ruby nokogiri, check this example
php DOMXpath, check this example
Check: Using regular expressions with HTML tags
Example using xpath :
xmllint --html --xpath 'string(//input[#value][2]/#value)' file
Output :
9uiY/UWJ1/w3PQ==
You may try this
(?:value[^v]*value=\")([^\"]*)
The output you want is captured in group 1, and you can retrieve it by backreference \1 or $1. Demo
"value=" is occurred twice in your sample text, so you seemed use the regex(value="[^"1" ].*\w==) to avoid the first one and match second one.
But the regex is wrong because character class'[...]' means one character. If the character class is followed by the quantifier(repeater) *, +, or {min,max} etc, then it's possible the regex means the string which has multiple characters.
Related
I'm learning about Bash scripting, and need some help understanding regex's.
I have a variable that is basically the html of a webpage (exported using wget):
currentURL = "https://www.example.com"
currentPage=$(wget -q -O - $currentURL)
I want to get the id's of all linked photos in this page. I just need help figuring out what the RegEx should be.
I started with this, but I need to modify the regex:
Test string (this is what currentURL contains, there can be zero to many instances of this):
<img src="./download/file.php?id=123456&t=1">
Current Regex:
.\/download\/file.php\?id=[0-9]{6}\&mode=view
Here's the regex I created, but it doesn't seem to work in bash.
The best solution would be to have the ID of each file. In this case, simply 123456. But if we can start with getting the /download/file.php?id=123456, that'd be a good start.
Don't parse XML/HTML with regex, use a proper XML/HTML parser.
theory :
According to the compiling theory, HTML can't be parsed using regex based on finite state machine. Due to hierarchical construction of HTML you need to use a pushdown automaton and manipulate LALR grammar using tool like YACC.
realLife©®™ everyday tool in a shell :
You can use one of the following :
xmllint often installed by default with libxml2, xpath1
xmlstarlet can edit, select, transform... Not installed by default, xpath1
xpath installed via perl's module XML::XPath, xpath1
xidel xpath3
saxon-lint my own project, wrapper over #Michael Kay's Saxon-HE Java library, xpath3
or you can use high level languages and proper libs, I think of :
python's lxml (from lxml import etree)
perl's XML::LibXML, XML::XPath, XML::Twig::XPath, HTML::TreeBuilder::XPath
Check: Using regular expressions with HTML tags
Example using xidel:
xidel -s "$currentURL" -e '//a/extract(#href,"id=(\d+)",1)'
Let's first clarify a couple of misunderstandings.
I'm learning about Bash scripting, and need some help understanding regex's.
You seem to be implying some sort of relation between Bash and regex.
As if Bash was some sort of regex engine.
It isn't. The [[ builtin is the only thing I recall in Bash that supports regular expressions, but I think you mean something else.
There are some common commands executed in Bash that support some implementation of regular expressions such as grep or sed and others. Maybe that's what you meant. It's good to be specific and accurate.
I want to get the id's of all linked photos in this page. I just need help figuring out what the RegEx should be.
This suggests an underlying assumption that if you want to extract content from an HTML, then regex is the way to go. That assumption is incorrect.
Although it's best to extract content from HTML using an XML parser (using one of the suggestions in Gilles' answer),
and trying to use regex for it is not a good reflect,
for simple cases like yours it might just be good enough:
grep -oP '\./download/file\.php\?id=\K\d+(?=&mode=view)' file.html
Take note that you escaped the wrong characters in the regex:
/ and & don't have a special meaning and don't need to be escaped
. and ? have special meaning and need to be escaped
Some extra tricks in the above regex are good to explain:
The -P flag of grep enables Perl style (powerful) regular expressions
\K is a Perl specific symbol, it means to not include in the match the content before the \K
The (?=...) is a zero-width positive lookahead assertion. For example, /\w+(?=\t)/ matches a word followed by a tab, without including the tab in the match.
The \K and the lookahead trickery is to work with grep -o, which outputs only the matched part. But without these trickeries the matched part would be for example ./download/file.php?id=123456&mode=view, which is more than what you want.
I am trying to replace the contents of a string which contains xml tags as follows
I want to replace the entirety of the below statement, where ABCDEF could be any random value
<originalFileName>ABCDEF</originalFileName>
How would I do this?
Solution
You can try this for your purpose:
(<originalFileName>[\w]*</originalFileName>)
Not recommended
However, note that it is not recommended.
Regular expressions are a tool that is insufficient to understand the constructs employed by XML/HTML/XHTML. XML/HTML/XHTML is not a regular language and hence cannot be parsed by regular expressions. Regex queries are not equipped to break down XML/HTML/XHTML into its meaningful parts. Even enhanced irregular regular expressions as used by Perl are not up to the task of parsing XML/HTML/XHTML.XML/HTML/XHTML is a language of sufficient complexity that it cannot be parsed by regular expressions.
Further details : RegEx match open tags except XHTML self-contained tags
This could help you:
EDITED (this new one will also accept non-characters between your tags and I have scaped the / symbol which can give some errors if not):
(<originalFileName>.*<\/originalFileName>)
Check it here.
I am new to regular expression. I want to know is there any way to batch up many 'find and replace' regular expressions together and is there any specific tool which could make this task easy?
In details-
I mean- Find one regular expression and replace with other regular expression, then find another regular expression and replace it with another different regular expression, then find third and replace it with some another, so on .. so on .. may be upto 20 search and replace. And in automated way as compared to manually doing search and replace singly upto 20 times.
Chaining Replacements
You can chain replacements in any language that gives you access to a regex engine.
Python and PHP are good choices if you are starting out and want to do a bit of scripting
Any of the .NET languages, Java, Ruby, Perl... You name it.
In Java
In the comments, you mention that you use Java. To chain replacements, you can do things like this:
String result1 = subjectString.replaceAll(myregex, myreplacement);
String result2 = result1.replaceAll(myregex2, myreplacement2);
String result3 = result2.replaceAll(myregex3, myreplacement3);
GUI Tools
I can think of three GUI tools that allow regex chaining:
PowerGrep (commercial, by Jan Goyvaerts, the author of the famous RegexBuddy)
TextDistil (free at the moment, .NET regex flavor)
TextPipe Pro (commercial)
In addition, regex chaining is available in applications with a narrow focus, for instance:
Directory Opus (powerful File Manager for Windows)
A Better Finder Rename and Name Mangler (file renamers for OSX)
In PHP you can do this with preg_replace(). If the pattern and replacement arguments are both arrays, each regexp in the pattern argument will be replaced with the corresponding element of the replacement argument.
I use an application to find specific text patterns in free text fields in XML records. It uses regex to identify the pattern and then it is tagged in the XML. For a specific project, it would be a great time saver (I am working with about 18 million records) if I could add 2 characters 27 in front of one of the pattern I have to use.
Can this be done or am I just going to have to go the long way around?
No, you can't have a regex match text that isn't there. A regex will only be able to return text that is part of the original text.
However, if you matched into groups, you could potentially use the group name for extra information about what you're matching.
Regex is not the right tool if you'd like to edit an XML file. Instead, use a modern language like Python, Perl, Ruby, PHP, Java with a proper XML parser module. If you work in Unix like shell, I recommend xmlstarlet
That said, if you'd like to go ahead with a substitution, you can try sed (at your own risks) :
sed -i -r 's/987654/27&/g' files*.xml
(use only -i switch only to modify in-place)
Using a regex, I am able to find a bunch of numbers that I want to replace. However, I want to replace the number with another number that is calculated using the original - captured - number.
Is that possible in notepad++ using a kind of expression in the replacement-part?
Edit: Maybe a strange thought, but could the calculation be done in the search part, generating a second captured number that would effectively be the result?
Even if it is possible, it will almost certainly be "messy" - why not do the replacements with a simple script instead? For example..
#!/usr/bin/env ruby
f = File.new("f1.txt", File::RDWR)
contents = f.read()
contents.gsub!(/\d+/){|m|
m.to_i + 1 # convert the current match to an integer, and add one
}
f.truncate(0) # empty the existing file
f.seek(0) # seek to the start of the file, before writing again
f.write(contents) # write modified file
f.close()
..and the output:
$ cat f1.txt
This was one: 1
This two two: 2
$ ruby replacer.rb
$ cat f1.txt
This was one: 2
This two two: 3
In reply to jeroen's comment,
I was actually interested if the possibility existed in the regular expression itself as they are so widespread
A regular expression is really just a simple pattern matching syntax. To do anything more advanced than search/replace with the matches would be up to the text-editors, but the usefulness of this is very limited, and can be achieved via scripting most editors allow (Notepad++ has a plugin system, although I've no idea how easy it is to use).
Basically, if regex/search-and-replace will not achieve what you want, I would say either use your editors scripting ability or use an external script.
Is that possible in notepad++ using a kind of expression in the replacement-part?
Interpolated evaluation of regular-expression matches is a relatively advanced feature that I probably would not expect to find in a general-purpose text editing application. I played around with Notepad++ a bit but was unable to get this to work, nor could I find anything in the documentation that suggests this is possible.
Hmmm... I'd have to recommend AWK to do this.
http://en.wikipedia.org/wiki/AWK
notepad++ has limited regular expressions built in. There are extensions that add a bit more to the regular expression find and replace, but I've found those hard to use. I would recommend writing a little external program to do it for you. Either Ruby, Perl or Python would be great for it. If you know those languages. I use Ruby and have had lots of success with it.