Looping though a directory which has dynamic link text - python-2.7

Question

I'm trying to loop through the list of companies in the Link. The link of each company name is dynamic for example http://ae.bizdirlib.com/node/946273 - Text link 946273 keeps changing i.e its dynamic. I want open each of these links in the page in a browser I'm really confused on how to do this. I have tried this for now.
from selenium import webdriver
from selenium.common.exceptions import NoSuchElementException
from selenium.webdriver.common.keys import Keys
import time
browser = webdriver.Firefox() # Get local session of firefox
#wait until the pages are loaded
browser.implicitly_wait(3)
browser.get("http://ae.bizdirlib.com/taxonomy/term/1493") # Load page
browser.refresh()
page_source = browser.page_source
for node in page_source:
link = browser.find_element_by_link_text('node').click
On executing this code it gives a error
Traceback (most recent call last):
File "C:/Python27/automation scripts/ggulf/large data.py", line 29, in <module>
link = browser.find_element_by_link_text('node').click
File "C:\Python27\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 276, in find_element_by_link_text
return self.find_element(by=By.LINK_TEXT, value=link_text)
File "C:\Python27\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 684, in find_element
{'using': by, 'value': value})['value']
File "C:\Python27\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 195, in execute
self.error_handler.check_response(response)
File "C:\Python27\lib\site-packages\selenium\webdriver\remote\errorhandler.py", line 170, in check_response
raise exception_class(message, screen, stacktrace)
NoSuchElementException: Message: Unable to locate element: {"method":"link text","selector":"node"}
Stacktrace:
at FirefoxDriver.prototype.findElementInternal_ (file:///c:/users/akrakhan/appdata/local/temp/tmppveyk8/extensions/fxdriver#googlecode.com/components/driver-component.js:10299)
at fxdriver.Timer.prototype.setTimeout/<.notify (file:///c:/users/akrakhan/appdata/local/temp/tmppveyk8/extensions/fxdriver#googlecode.com/components/driver-component.js:603)

Answer 1

You are better off looking for something more specific rather than looking through the page source. All the company links are links inside a H2 tag. You can find them using a CSS selector h2 > a which is read find all A tags that are a child of (>) an h2 element.
browser.get("http://ae.bizdirlib.com/taxonomy/term/1493") # Load page
links = browser.find_elements_by_css_selector("h2 > a")
for link in links:
link.click
This isn't the final solution because clicking the link will take you off the main page but it's a parallel to what you were trying to accomplish. Probably a better approach would be to store the URLs of all the company links in a string array and then loop through that array navigating to each URL... or something like that. An exercise for the reader... :)