Python SSL web page scraping - python-2.7

Question

I am trying to scrape web page using Python 2.7 and BeautifulSoup but I can't get past a protocol error which doesn't make much sense to me. This only happens on the specific website that I need to do this for: https://edd.telstra.com/telstra
The code I use just for basic test:
#! /usr/bin/python
from urllib import urlopen
from BeautifulSoup import BeautifulSoup
import re
# Copy all of the content from the provided web page
webpage = urlopen("https://edd.telstra.com/telstra/").read()
And I get the following error (running on Ubuntu 12.10):
Traceback (most recent call last):
File "e.py", line 8, in <module>
webpage = urlopen("https://edd.telstra.com/telstra/").read()
File "/usr/lib/python2.7/urllib.py", line 86, in urlopen
return opener.open(url)
File "/usr/lib/python2.7/urllib.py", line 207, in open
return getattr(self, name)(url)
File "/usr/lib/python2.7/urllib.py", line 436, in open_https
h.endheaders(data)
File "/usr/lib/python2.7/httplib.py", line 958, in endheaders
self._send_output(message_body)
File "/usr/lib/python2.7/httplib.py", line 818, in _send_output
self.send(msg)
File "/usr/lib/python2.7/httplib.py", line 780, in send
self.connect()
File "/usr/lib/python2.7/httplib.py", line 1165, in connect
self.sock = ssl.wrap_socket(sock, self.key_file, self.cert_file)
File "/usr/lib/python2.7/ssl.py", line 381, in wrap_socket
ciphers=ciphers)
File "/usr/lib/python2.7/ssl.py", line 143, in __init__
self.do_handshake()
File "/usr/lib/python2.7/ssl.py", line 305, in do_handshake
self._sslobj.do_handshake()
IOError: [Errno socket error] [Errno 1] _ssl.c:504: error:1408F119:SSL routines:SSL3_GET_RECORD:decryption failed or bad record mac
Could someone tell me if there is some parameter that I need to specify to get this page to download in Python? It seems that this is the problem just on this web page as the code above (plus lots of other code I tried) works fine on other HTTPS/SSL pages I tried.
Thanks for any help!

Answer 1

I can recommend using requests lib :
def get_page(login, password):
'''Docstring
'''
url = 'https://qwe.qwe'
payload = {
'user': login,
'pass': password
}
with requests.Session() as my_session:
my_session.post(url, data=payload)
data = my_session.get(url)
return data.text
More info : http://docs.python-requests.org/en/latest/user/advanced/#session-objects