I am trying to click a tab (Regulatory Regional) on a webpage: https://www5.fdic.gov/idasp/advSearchLanding.asp
However, it does not recognize the command. Here, I have attached the code.
import urllib2
import urllib
from bs4 import BeautifulSoup
import subprocess
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import Select
from selenium.webdriver.common.keys import Keys
browser = webdriver.Chrome("/usr/local/bin/chromedriver")
import time
s1_url = 'https://www5.fdic.gov/idasp/advSearchLanding.asp'
browser.get(s1_url)
Problem: choose regulatory regional tab but it does not click it.
browser.find_element_by_xpath('//[#id="Banks_Regulatory_Tab"]/a').click()
Got an exception:
NoSuchElementException: Message: no such element: Unable to locate element: {"method":"xpath","selector":"//*[#id="Banks_Regulatory_Tab"]/a"}
Required element located inside an iframe. To be able to handle it you need to switch to that iframe:
browser.switch_to.frame("content")
browser.find_element_by_link_text("Regulatory Regional").click()
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import Select
from selenium.webdriver.support.select import Select
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
driver = webdriver.Chrome()
driver.get("https://www.amazon.com/")
driver.find_element_by_partial_link_text("Sign in").click()
driver.find_element_by_name("email").send_keys("** UR EMAILID **")
driver.find_element_by_name("password").send_keys(" ** UR PASSWORD **")
driver.find_element_by_id("signInSubmit").click()
driver.find_element_by_id("twotabsearchtextbox").send_keys("Mens shoes")
driver.find_element_by_css_selector("#nav-search > form > div.nav-right > div > input").click()
driver.find_element_by_partial_link_text("Fashion Sneakers").click()
dropdown = driver.find_element_by_css_selector("#native_dropdown_selected_size_name")
select = Select(dropdown)
select.select_by_value("4,B01CE7QQPY")
driver.find_element_by_xpath("//*[#id='add-to-cart-button']").click()
I'm able to login, go to mens shoes and select fashion sneakers and select a type of shoe with particular size. Despite selecting its size, I'm not allowed to add to cart. The page says "select the size form the left to add to shopping cart". There are no errors on terminal/command line, but I'm unable to proceed. I have added a screenshot of the screen for reference as image below:
[]
I am trying to scrape the stats off the table on this webpage: http://stats.nba.com/teams/traditional/ but I am unable to find the html for the table. This is in python 2.7.10.
from bs4 import BeautifulSoup
import json
import urllib
html = urllib.urlopen('http://stats.nba.com/teams/traditional/').read()
soup = BeautifulSoup(html, "html.parser")
for table in soup.find_all('tr'):
print(table)
This is the code I have now, but nothing is being outputted.
If I try this with different elements on the page it works fine.
The table is loaded dynamically, so when you grab the html, there are no tr tags in it to be found.
The table you're looking for is NOT in that specific page/URL.
The stats you're trying to scrape come from this url:
http://stats.nba.com/stats/leaguedashteamstats?Conference=&DateFrom=&DateTo=&Division=&GameScope=&GameSegment=&LastNGames=0&LeagueID=00&Location=&MeasureType=Base&Month=0&OpponentTeamID=0&Outcome=&PORound=0&PaceAdjust=N&PerMode=PerGame&Period=0&PlayerExperience=&PlayerPosition=&PlusMinus=N&Rank=N&Season=2016-17&SeasonSegment=&SeasonType=Regular+Season&ShotClockRange=&StarterBench=&TeamID=0&VsConference=&VsDivision=
When you browse a webpage/url in a modern browser, more requests are made "behind the scene" other than the original url you use to fully render the whole page.
I know this sounds counter-intuitive, you can check out this answer for a bit more detailed explanation.
Try this code. It is giving me the HTML code. I am using requests to obtain information.
import datetime
import BeautifulSoup
import os
import sys
import pdb
import webbrowser
import urllib2
import requests
from datetime import datetime
from requests.auth import HTTPBasicAuth
from HTMLParser import HTMLParser
from urllib import urlopen
from bs4 import BeautifulSoup
url="http://stats.nba.com/teams/traditional/"
data=requests.get(url)
if (data.status_code<400):
print("AUTHENTICATED:STATUS_CODE"+" "+str(data.status_code))
sample=data.content
soup=BeautifulSoup(sample,'html.parser')
print soup
You can use selenium and PhantomJS (or chomedriver, firefox etc.) to load the page, thereby also loading all the javascript. All you need is to download selenium and the PhantomJS webdriver, then place a sleep timer after the get(url) to ensure that the page loads (actually, using a function such as WebDriverWait would be much better than sleep, but you can look more into that if you need it). Now your soup content will look exactly like that what you see when looking at the site through your browser.
from bs4 import BeautifulSoup
from selenium import webdriver
from time import sleep
url = 'http://stats.nba.com/teams/traditional/'
browser = webdriver.PhantomJS('*path to PhantomJS driver')
browser.get(url)
sleep(10)
soup = BeautifulSoup(browser.page_source, "html.parser")
for table in soup.find_all('tr'):
print(table)
I am trying to scroll down an infinite scroll page and get the links of news. The problem is when I scrolled down the page for let say 100 times, and I tried to get the links, Python launched an error that says: "StaleElementReferenceException: Message: stale element reference: element is not attached to the page document". I think its because the page is get updated and scrolled page is not available any more. here is my code for scrolling the page with Selenium Webdriver:
import urllib2
from bs4 import BeautifulSoup
from __future__ import print_function
from selenium import webdriver #open webdriver for specific browser
from selenium.webdriver.common.keys import Keys # for necessary browser action
from selenium.webdriver.common.by import By # For selecting html code
import time
driver = webdriver.Chrome('C:\\Program Files (x86)\\Google\\Chrome\\chromedriver.exe')
driver.get('http://seekingalpha.com/market-news/top-news')
for i in range(0,100):
driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
time.sleep(15)
URL = driver.find_elements_by_class_name('market_current_title')
print URL
and the code for getting the URLs
for a in URL:
links = a.get_attribute('href')
print(links)
I am wondering if there is any solution to settle this problem or it is possible to get URLs for this specific page with request library, as I couldn't do that.
I have a python script which has the functionality of sending an email to a user. I executed this script and it is working fine. In another python script I have only a button, so when I click on this button I want the other python script which sends a email to be executed.I have written the following code:
#!/usr/bin/python
import sys
import os
import Tkinter
import tkMessageBox
top=Tkinter.Tk()
def helloCallBack():
os.system('SendEmail.py')
B=Tkinter.Button(top,text="hello",command= helloCallBack)
B.pack()
top.mainloop()
I get the following error when I click on the button:
sh: 1:SendEmail.py:not found.
Could you let me know what is the reason for this error and how it can be resolved.Thanks.
I was able to figure out a way to call another python script on button click:
instead of using os.system('SendEmail.py') we need to use os.system('python SendEmail.py')
import sys
import os
from tkinter import *
window=Tk()
window.title("Running Python Script")
window.geometry('550x200')
def run():
os.system('opencv_video.py')
btn = Button(window, text="Click Me", bg="black", fg="white",command=run)
btn.grid(column=0, row=0)
window.mainloop()
If your SendEmail.py is in the same location, use os.system('SendEmail.py'). If it's in a different location, use os.system('python SendEmail.py').
#!/usr/bin/python
import sys
import sys
import os
import Tkinter
import tkMessageBox
top=Tkinter.Tk()
def helloCallBack():
os.system('python SendEmail.py')
B=Tkinter.Button(top,text="hello",command= helloCallBack)
B.pack()
top.mainloop()
use the keyword "python" to run the command
As an amateur, I am not really qualified to give advice. This is how I did it.
I want to do this kind of thing too. I have about 16 little python programs which make html, sets of checkboxes, sets of radiobuttons, text input fields, html tables etc.
In another thread here a comment was quite deprecative of using os.system calls. Not sure why, but I thought I would try another approach.
I've just started learning tkinter, so I am making each of my 'makehtml' functions run in a window.
Now I want a master window with buttons. Click a button and another window opens, say the checkboxes window, or any of the other windows for making html.
I made a module: guiHTML.py All my 'makehtml' functions are in there.
Import guiHTML in the master window.
import os, sys
# to import the files we need the paths
path = '/home/pedro/myPython/myModules/'
# append the paths
sys.path.append(path)
import tkinter as tk
from functools import partial
import guiHTML
Then, in the master window make a function like this for each button:
def openCheckboxes():
#call the checkboxes function defined in the guiHTML module
guiHTML.checkboxes()
Then, in the checkboxes button just put this:
btn3 = tk.Button(frame1, text='insert checkboxes', command=openCheckboxes)
btn3.grid(columnspan=2, column=0, row=2, sticky='w', pady=10)
Click btn3 and the checkboxes window opens.
This works for me, but I don't know if it is a good way to do this. I only began with tkinter a month ago.
If there is a better way to do this, I'd be glad to hear it from you experts!
#!/usr/bin/python
import sys
import os
import tkinter as tk
root = tk.Tk()
def helloCallBack():
os.system('call.py')
#Keep_both_files_in_the_same_Folder
b1=tk.Button(root, text="Calendar",bg="white",command=helloCallBack)
b1.pack()
root.mainloop()