I am trying to get the contact names from here and I'm facing a very strange problem. The content is visible in the browser but when I use selenium to find the element using xpath, I get no data. As soon as I click inspect element, selenium will find the data.
mydriver = webdriver.Chrome()
print 'Webdriver Started'
mydriver.get('http://listings.fta-companies-au.com/l/101662595/BNP-Paribas-in-Sydney-NSW')
contact_persons = mydriver.find_elements_by_xpath('//div[#class="data-block is-editable no-header"]//div[#class="srp-float-wrap flt-scroll-wrap"]//table[#class="srp-widget-table"]/tbody/tr')
for p in contact_persons:
print p.text
When I just load and try find the data, it will return an empty list but as soon as I click inspect element, I'll get the required data.
I've also tried using requests and lxml to parse but and they too return empty data.
Seems that the details table is only setup when you scroll to it. Try moving to the h2 tag with text -- Employees and Executives, using driver.moveto.... function. This should make the details available.
Related
The main objective I have is to read a table in a webpage and account for the total element it has. But since you have to scroll down to find another elements not 'chased' by this sentece
table_css=driver.find_elements_by_id('DeletButtn')
then I decided to zoom in down to 30% to catch them all. But no way I've used is working when trying to zoom in:
driver.execute_script("document.body.style_zoom='30%'")
driver.execute_script("document.body.style.zoom='30%'")
driver.execute_script("document.body.style.zoom='0.3'")
do not work
I have also tried to use key and send keys, but they have been worthless too:
html = driver.find_element_by_tag_name("body")
html.send_keys(Keys.CONTROL, Keys.SUBTRACT)
driver.Keys(html, "ctrl").Keys(html, "-").perform()
action = ActionChains(driver)
action.key_down(Keys.CONTROL).key_down(Keys.SUBTRACT).perform()
None of these combinations have worked up til now. They haven't show me any error so far, so I do not know what I am doing wrong.
I am using python 2.7, geckodriver and Firefox
Thanks for your help and greetings!
I'm trying to scrape google search results using python and selenium. I'm able to get only the first search result. Here is the code I'm using.
driver.get(url)
res = driver.find_elements_by_css_selector('div.g')
link = res[0].find_element_by_tag_name("a")
href = link.get_attribute("href")
How can I get all the search results?
Try to get list of links (from first page only. If you need to scrape more pages, you need to click "Next" button in a loop and append results from following pages) as below:
href = [link.get_attribute("href") for link in driver.find_elements_by_css_selector('div.g a')]
P.S. You also might use solutions from this question to get results as GET request response with requests lib
I use opencart version 2.1.0.1
Everytime I click admin > sales > order, it will pop up "error undefined." By closing that popup window, I can still edit order but cannot delete order (no response).
In my log, there is:
PHP Notice: Undefined variable: order_id in
/var/www/html/opencart2101/system/storage/modification/admin/view/template/sale/order_list.tpl on line 821
The line 821 is:
url: 'index.php?route=extension/openbay/addorderinfo&token=<?php echo $token; ?>&order_id=<?php echo $order_id; ?>&status_id=' + status_id,
However, I haven't installed any openbay related module. Also, line 821 is inside <!-- --> mark. It should have no effect.
Help!
Although this is now an older version of opencart, I still see this being reported a lot around and about.
The problem occurs due to the store front adding the http url rather than the https url to the order. So firstly you need to fix that. If you dont want to read all of my explanation, you can just hit up the bold points :)
Either way BACKUP EVERYTHING actually not really, back up the file you are going to edit and backup your whole database.
open:
catalog/controller/checkout/confirm.php at around line 100
Find:
$order_data['store_url'] = HTTP_SERVER;
Change to:
$order_data['store_url'] = HTTPS_SERVER;
Now you will want to fix your database because for reasons I cannot fathom, the domain name is placed in the order along with the stores id. and when editing orders it is the usage of that directly within your admin order page that throws up the undefined notice. Basically the browser blocks the request because its trying to make an insecure request from a secure page.
Crack open phpmyadmin or whatever database tool you have on hand.
locate the table, default is oc_orders
Browsing the table, look for the column that contains your store url (i cant remember the name off hand, i think its just store_url but it will be obvious anyway. if you are multi store you will need to run the query for each
I am sure somebody can come up with a clever way to automatically convert just the http into https with a single use sql query on the one column, but this works for me.
Run SQL: adjust as appropriate
UPDATE `oc_orders` SET `store_url` = 'https://example.com' WHERE store_id = 0;
I'm trying to get Selenium Webdriver to click x number of links in a table, and I can't get it to work. I can print the links like this:
links = driver.find_elements_by_xpath("//table[2]/tbody/tr/td/p/strong/a")
for i in range(0,len(links)):
print links[i].text
But when I try to do a links[i].click() instead of printing, python throws me an error.
The site uses JSP and the hrefs of the links looks like this "javascript:loadTestResult(169)"
This is a sub/sub-page and not possible to access by direct URL, and the table containing the links are very messy and large so instead of pasting the whole source here I saved the page on this url.
http://wwwe.aftonbladet.se/redaktion/martin/badplats.html
(I'm hunting the 12 blue links in the left column)
Any ideas?
Thanks
Martin
Sorry, to trigger happy.
Simple solution to my own problem:
linkList = driver.find_elements_by_css_selector("a[href*='loadTestResult']")
for i in range(0,len(linkList)):
links = driver.find_elements_by_css_selector("a[href*='loadTestResult']")
links[i].click()
I'm using Selenium to scrape / parse an awful web site (if it wasn't awful, I might not use Selenium, and yes, respecting robots.txt).
I'm reading a set of links from a table of unknown size, with sequential element ids, using find_element_by_id(). I'm catching NoSuchElementException to tell me that I'm at the end of the table and there are no other elements to pick up.
This smoothly walks through the elements that exist, but takes about 30 seconds to throw the error when I request the non-existent element that tells me I'm at the end of the table.
The file is not that huge - the html dump from DOM Inspector delivers a 81kb file. The last link in the table (which Selenium finds quickly) is 7/8s of the way through the file, so (assuming Selenium is parsing sequentially) file size alone doesn't seem to explain this.
Can I speed up the failure of finding the missing element? Or is there a more elegant way to know I am in the last row of the table with content?
You might want to do this using css selectors instead.
driver.findElements( By.cssSelector( '[id^=id_name]' ) )