How to check (not bypass) SSL certificate with urllib2.urlopen? - python-2.7

Question

I'm using Python 2.7.13. I'm trying to connect to https://www.python.org/ and verify its certificate using urllib2.urlopen. I'm getting "SSL: CERTIFICATE_VERIFY_FAILED" error in the process, and when I try to google the issue, I seem to get answers on how to bypass this security check. But I don't want my code to bypass it, I want it to use the certificate. Here's my code that fails:
import urllib2
from contextlib import closing
import ssl
ctx = ssl.create_default_context(purpose = ssl.Purpose.SERVER_AUTH, cafile = 'www.python.org.crt')
request = urllib2.Request('https://www.python.org/')
# Produces URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:661)>
with closing(urllib2.urlopen(request, context = ctx)) as response:
print response.read()
cafile = 'www.python.org.crt' is a PEM-formatted file that properly starts with -----BEGIN CERTIFICATE-----. I exported it from https://www.python.org/ using this instruction and put it into the script's working folder.

Answer 1

From the documentation: https://docs.python.org/2/library/ssl.html#ssl._https_verify_certificates
Starting with Python 2.7.9, httplib and modules which use it, such as urllib2 and xmlrpclib, default to verifying remote server certificates received when establishing client HTTPS connections. This default verification checks that the certificate is signed by a Certificate Authority in the system trust store and that the Common Name (or Subject Alternate Name) on the presented certificate matches the requested host.
So you don't need to anything special to verify the certificate. And you can see that because you have a validation failure, which shows you that validation is happening.
So why aren't you getting successful validation? The answer to this lies in your use of the cafile. The cafile parameter takes a file which overrides your systems default trusted certificate store. But the important thing to understand about trusted certificates the validation process isn't to check the host is in the trusted store but signed but a cert in the trusted store. So you don't need to pass in the actual server certificate, you would pass in the certificate from python.org's certificate vendors (digicert at the time of writing) .
But actually you don't even need to do this as you system should come with a large set of trusted certificates from many vendors. These vendors are audited to some extent ( Opinions vary on the extent - and this now getting off topic) and python's library will validate against this set as default if you leave the cafile parameter blank.
So you find that
>>> ctx = ssl.create_default_context(purpose = ssl.Purpose.SERVER_AUTH, cafile = '/tmp/python.org.cert')
>>> urllib2.urlopen("https://www.python.org/", context = ctx)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python2.7/urllib2.py", line 154, in urlopen
return opener.open(url, data, timeout)
File "/usr/lib/python2.7/urllib2.py", line 429, in open
response = self._open(req, data)
File "/usr/lib/python2.7/urllib2.py", line 447, in _open
'_open', req)
File "/usr/lib/python2.7/urllib2.py", line 407, in _call_chain
result = func(*args)
File "/usr/lib/python2.7/urllib2.py", line 1241, in https_open
context=self._context)
File "/usr/lib/python2.7/urllib2.py", line 1198, in do_open
raise URLError(err)
urllib2.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:661)>
fails, but this:
>>> ctx = ssl.create_default_context(purpose = ssl.Purpose.SERVER_AUTH,)
>>> urllib2.urlopen("https://www.python.org/", context = ctx)
<addinfourl at 140556044048504 whose fp = <socket._fileobject object at 0x7fd5c1127b50>>
works.
Finally I'd like to point you at httplib and http.client (in python 3) which are much cleaner interfaces for this sort of thing.