I'm trying to extract the part of url (To be more specific, I'm trying to extract the value of page_info parameter in the url which is next to rel="next"
String testUrl = "<https://demo.myshopify.com/admin/api/2022-01/products.json?limit=10&page_info=eyJkaXJlY3Rpb24iOiJwcmV2IiwibGFzdF9pZCI6NjczMDU4MDcyMTc1NCwibGFzdF92YWx1ZSI6IjE4SyBHb2xkIFBsYXRlZCBCcmFjZWxldCJ9>; rel='previous', <https://demo.myshopify.com/admin/api/2022-01/products.json?limit=10&page_info=eyJkaXJlY3Rpb24iOiJuZXh0IiwibGFzdF9pZCI6NjczMDIyNzcxMjA5MCwibGFzdF92YWx1ZSI6IjE4SyBHb2xkIFBsYXRlZCBIZWFydCBQZW5kYW50IE5lY2tsYWNlIn0>; rel='next'";
List<String> splitUrl = testUrl.split("=");
print(splitUrl[5]);
// this is what it prints out
eyJkaXJlY3Rpb24iOiJuZXh0IiwibGFzdF9pZCI6NjczMDIyNzcxMjA5MCwibGFzdF92YWx1ZSI6IjE4SyBHb2xkIFBsYXRlZCBIZWFydCBQZW5kYW50IE5lY2tsYWNlIn0>; rel
// this is what I'm trying to extract
eyJkaXJlY3Rpb24iOiJuZXh0IiwibGFzdF9pZCI6NjczMDIyNzcxMjA5MCwibGFzdF92YWx1ZSI6IjE4SyBHb2xkIFBsYXRlZCBIZWFydCBQZW5kYW50IE5lY2tsYWNlIn0
// value for rel="next"
I tried to split the url by using split function on String but that would also bring the angle bracket with it. I'm trying to extract only page_info= parameter value which is for rel="next"
I know this has to do something with regex but I'm not really good at it! Any help would be really appreciated
I grabbed that url from header response (paginated REST API), it returns two page_info parameters (one for next and other one for previous page) I'm trying to extract value for next page. Splitting the url didn't help me
thank you
An alternative approach is to use Uri.parse to parse the URL:
void main() {
String testUrl = "<https://demo.myshopify.com/admin/api/2022-01/products.json?limit=10&page_info=eyJkaXJlY3Rpb24iOiJwcmV2IiwibGFzdF9pZCI6NjczMDU4MDcyMTc1NCwibGFzdF92YWx1ZSI6IjE4SyBHb2xkIFBsYXRlZCBCcmFjZWxldCJ9>; rel='previous', <https://demo.myshopify.com/admin/api/2022-01/products.json?limit=10&page_info=eyJkaXJlY3Rpb24iOiJuZXh0IiwibGFzdF9pZCI6NjczMDIyNzcxMjA5MCwibGFzdF92YWx1ZSI6IjE4SyBHb2xkIFBsYXRlZCBIZWFydCBQZW5kYW50IE5lY2tsYWNlIn0>; rel='next'";
// Extract just the URL.
var match = RegExp(r'<([^>]*)>').firstMatch(testUrl);
if (match != null) {
var uri = Uri.parse(match.group(1)!);
print(uri.queryParameters['page_info']); // Prints: eyJkaXJlY3Rpb24iOiJ...
}
}
Note that the above wouldn't need any of the RegExp code if testUrl were a proper URL without the angle brackets and rel='next' junk.
the regEx pattern page_info=([\w]+)
gives you
eyJkaXJlY3Rpb24iOiJwcmV2IiwibGFzdF9pZCI6NjczMDU4MDcyMTc1NCwibGFzdF92YWx1ZSI6IjE4SyBHb2xkIFBsYXRlZCBCcmFjZWxldCJ9
eyJkaXJlY3Rpb24iOiJuZXh0IiwibGFzdF9pZCI6NjczMDIyNzcxMjA5MCwibGFzdF92YWx1ZSI6IjE4SyBHb2xkIFBsYXRlZCBIZWFydCBQZW5kYW50IE5lY2tsYWNlIn0
https://regexr.com/6qj1h
I looking for regular expression to use in my javascript code, which give me last part of url without parameters if they exists - here is example - with and without parameters:
https://scontent-fra3-1.xx.fbcdn.net/v/t1.0-9/14238253_132683573850463_7287992614234853254_n.jpg?oh=fdbf6800f33876a86ed17835cfce8e3b&oe=599548AC
https://scontent-fra3-1.xx.fbcdn.net/v/t1.0-9/14238253_132683573850463_7287992614234853254_n.jpg
In both cases as result I want to get:
14238253_132683573850463_7287992614234853254_n.jpg
Here is this regexp
.*\/([^?]+)
and JS code:
let lastUrlPart = /.*\/([^?]+)/.exec(url)[1];
let lastUrlPart = url => /.*\/([^?]+)/.exec(url)[1];
// TEST
let t1 = "https://scontent-fra3-1.xx.fbcdn.net/v/t1.0-9/14238253_132683573850463_7287992614234853254_n.jpg?oh=fdbf6800f33876a86ed17835cfce8e3b&oe=599548AC"
let t2 = "https://scontent-fra3-1.xx.fbcdn.net/v/t1.0-9/14238253_132683573850463_7287992614234853254_n.jpg"
console.log(lastUrlPart(t1));
console.log(lastUrlPart(t2));
May be there are better alternatives?
You could always try doing it without regex. Split the URL by "/" and then parse out the last part of the URL.
var urlPart = url.split("/");
var img = urlPart[urlPart.length-1].split("?")[0];
That should get everything after the last "/" and before the first "?".
I'm trying to scrape the inside ownership from Morningstar at this url:
http://investors.morningstar.com/ownership/shareholders-overview.html?t=TWTR®ion=usa&culture=en-US
This is the code I'm using:
Sub test()
Dim appIE As Object
Set appIE = CreateObject("InternetExplorer.Application")
With appIE
.Navigate "http://investors.morningstar.com/ownership/shareholders-overview.html?t=TWTR®ion=usa&culture=en-US"
.Visible = True
End With
While appIE.Busy
DoEvents
Wend
Set allRowOfData = appIE.Document.getElementById("currentInsiderVal")
Debug.Print allRowOfData
Dim myValue As String: myValue = allRowOfData.Cells(0).innerHTML
appIE.Quit
Set appIE = Nothing
Range("A30").Value = myValue
End Sub
I get run-time error 13 at line
Set allRowOfData = appIE.Document.getElementById("currentInsiderVal")
but I can't see any mismatch. What is going on?
You can just do it with XHR and RegEx instead of cumbersome IE:
Sub Test()
Dim sContent
With CreateObject("MSXML2.XMLHTTP")
.Open "GET", "http://investors.morningstar.com/ownership/shareholders-overview.html?t=TWTR®ion=usa&culture=en-US", False
.Send
sContent = .ResponseText
End With
With CreateObject("VBScript.RegExp")
.Pattern = ",""currInsiderVal"":(.*?),"
Range("A30").Value = .Execute(sContent).Item(0).SubMatches(0)
End With
End Sub
Here is the description how the code works:
First of all MSXML2.XMLHTTP ActiveX instance is created. GET request opened with target URL in synchronous mode (execution interrupts until response received).
Then VBScript.RegExp is created. By default .IgnoreCase, .Global and .MultiLine properties are False. The pattern is ,"currInsiderVal":(.*?),, where (.*?) is a capturing group, . means any character, .* - zero or more characters, .*? - as few as possible characters (lazy matching). Other characters in pattern to be found as is. .Execute method returns a collection of matches, there is only one match object in it since .Global is False. This match object has a collection of submatches, there is only one submatch in it since the pattern contains the only capturing group.There are some helpful MSDN articles on regex:
Microsoft Beefs Up VBScript with Regular Expressions
Introduction to Regular Expressions
Here is the description how I created the code:
First I found an element containing the target value on the webpage DOM using browser:
The corresponding node is:
<td align="right" id="currrentInsiderVal">143.51</td>
Then I made XHR and found this node in the response HTML, but it didn't contain the value (you can find response in the browser developer tools on network tab after you refresh the page):
<td align="right" id="currrentInsiderVal">
</td>
Such behavior is typical for DHTML. Dynamic HTML content is generated by scripts after the webpage loaded, either after retrieving a data from web via XHR or just processing already loaded withing webpage data. Then I just searched for the value 143.51 in the response, the snippet ,"currInsiderVal":143.51, located within JS function:
fundsArr = {"fundTotalHistVal":132.61,"mutualFunds":[[1,89,"#a71620"],[2,145,"#a71620"],[3,152,"#a71620"],[4,198,"#a71620"],[5,155,"#a71620"],[6,146,"#a71620"],[7,146,"#a71620"],[8,132,"#a71620"]],"insiderHisMaxVal":3.535,"institutions":[[1,273,"#283862"],[2,318,"#283862"],[3,351,"#283862"],[4,369,"#283862"],[5,311,"#283862"],[6,298,"#283862"],[7,274,"#283862"],[8,263,"#283862"]],"currFundData":[2,2202,"#a6001d"],"currInstData":[1,4370,"#283864"],"instHistMaxVal":369,"insiders":[[5,0.042,"#ff6c21"],[6,0.057,"#ff6c21"],[7,0.057,"#ff6c21"],[8,3.535,"#ff6c21"],[5,0],[6,0],[7,0],[8,0]],"currMax":4370,"histLineQuars":[[1,"Q2"],[2,"Q3"],[3,"Q4"],[4,"Q1<br>2015"],[5,"Q2"],[6,"Q3"],[7,"Q4"],[8,"Q1<br>2016"]],"fundHisMaxVal":198,"currInsiderData":[3,143,"#ff6900"],"currFundVal":2202.85,"quarters":[[1,"Q2"],[2,""],[3,""],[4,"Q1<br>2015"],[5,""],[6,""],[7,""],[8,"Q1<br>2016"]],"insiderTotalHistVal":3.54,"currInstVal":4370.46,"currInsiderVal":143.51,"use10YearData":"false","instTotalHistVal":263.74,"maxValue":369};
So the regex pattern created based on that it should find the snippet ,"currInsiderVal":<some text>, where <some text> is our target value.
Had a look on the site and the element you are trying to retrieve has a typo in it; instead of currentInsiderVal try using currrentInsiderVal and you should retrieve the data correctly.
Probably worth considering some error trapping to catch stuff like this for any other fields you retrieve?
After your comment I took a closer look. Your issue seemed like it was trying to trap the id of the individual cell rather than navigating down the object tree. I've modified the code to retrieve the row of the table you are after and then set myValue to be the correct cell within that row. Seemed to be working when I tried it out. Give this a shot?
Sub test()
Dim appIE As Object
Set appIE = CreateObject("internetexplorer.application")
With appIE
.Navigate "http://investors.morningstar.com/ownership/shareholders-overview.html?t=TWTR®ion=usa&culture=en-US"
.Visible = True
End With
While appIE.Busy
DoEvents
Wend
Set allRowOfData = appIE.Document.getelementbyID("tableTest").getElementsByTagName("tbody")(0).getElementsByTagName("tr")(5)
myValue = allRowOfData.Cells(2).innerHTML
appIE.Quit
Set appIE = Nothing
Range("A30").Value = myValue
End Sub
I need to find out, whether in an array there is a specific HTML code. The array contains HTML codes and I need to get a number, that is included in a link.
This would be what I am searching for (the number 10 ist the number I want):
class = "active" href = "http://www.example.com/something-10
So I tried the following using preg_match:
if(preg_match('/class = "active" href = "http://www.example.com/something-(.*)/',$array["crawler"],$arr)) { print_r($arr,true); }
Unfortunately this will give me nothing as result. So I guess, something is wrong with my preg_match. I allready checked all the manuals, but I still dont get what I am doing wrong.
Could someone help me with this? Thank you!
phpheini
Aside from advising you to not parse HTML using regular expressions, your particular regular expression needs different delimiters:
preg_match('~class = "active" href = "http://www\.example\.com/something-(\d+)~', ...)
Alternatively, you could have escaped the slashes within the regex, but that leads to LSS (leaning slash syndrome):
preg_match('/class = "active" href = "http:\/\/www\.example\.com\/something-(.*)/', ...)
And that's just ugly.
You should have gotten an error, if your error_reporting is turned on.
I'm working on WatiN automation tool. I'm having problem in regular expression. I've situation where i have to enter some text and click on a button in the popup window. I'm using AttachToIE method and URL attribute("http://192.168.25.10:215/admin/SelectUsers.aspx?Type=FeedbackID=ef5ad7ef5490-4656-9669-32464aeba7cd") of the popup to attach to the popup.
The problem is each time the popup appears the ID value in the URL changes. So i'm not able to access the popup. can anyone plz help with this by giving me Regular Expression for the changing value of ID in the below URL
("http://192.168.25.10:215/admin/SelectUsers.aspx?Type=FeedbackID=ef5ad7ef5490-4656-9669-32464aeba7cd")
thanking you
It appears that you have a URL with 2 query string parameters Type and ID and your pattern is:
"http://192.168.25.10:215/admin/SelectUsers.aspx?Type=Feedback&ID={some id}"
You can use the Find.ByUrl() attribute constraint method and pass it to AttachToIE() as shown below with the regex for matching that pattern.
string url = "http://192.168.25.10:215/admin/SelectUsers.aspx?Type=Feedback&ID="
Regex regex = new Regex(url + "[a-z0-9]+", RegexOptions.IgnoreCase);
IE ie = IE.AttachToIE(Find.ByUrl(regex));
string baseUrl ="http://192.168.25.10:215/admin/SelectUsers.aspx?Type=FeedbackID="
Regex urlIE= new Regex(baseUrl + "[\\wd]+", RegexOptions.IgnoreCase);
IE ie = IE.AttachToIE(Find.ByUrl(urlIE);
I'm not familiar with WatiN but it looks like it's runs on .Net so perhaps this might help?
var desiredId = "000000000000-0000-0000-000000000000";
var url = "http://192.168.25.10:215/admin/SelectUsers.aspx?Type=FeedbackID=ef5ad7ef5490-4656-9669-32464aeba7cd&someMoreStuff";
var pattern = #"(?i)(?<=FeedBackId=)[-a-z0-9]+";
var result = Regex.Replace(url, pattern, desiredId);
Console.WriteLine(result);
//Output: http://192.168.25.10:215/admin/SelectUsers.aspx?Type=FeedbackID=000000000000-0000-0000-000000000000&someMoreStuff
The following pattern should have the same affect but is more defensive. It should only match stuff in the query string, it requires the id to be 35 characters and won't match similar parameter names like "PreviousFeedBackId".
var pattern = #"(?i)(?<=\?.*\bFeedBackId=)[-a-z0-9]{35,35}\b";
If you just want to extract the id:
var id = Regex.Match(url, pattern).Value;
Console.WriteLine(id);
//output: ef5ad7ef5490-4656-9669-32464aeba7cd
WatiN has a feature where in we can use the url by neglecting the query string. Below is the code which is working fine for me.
string baseUrl = "http://192.168.25.10:215/admin/SelectUsers.aspx";
IE ie = IE.AttachToIE(Find.ByUrl(baseUrl,true));