I am attempting to use requests package from python to access this site: https://egov.uscis.gov/casestatus/landing.do
When I ran this command:
requests.get('https://egov.uscis.gov/casestatus/landing.do')
I got the usual SSL error when your authentication verification fails..
Read through stackoverflow and adopted one of the solutions: download the certificate in (.crt) and then used openssl to convert to .pem file. I then copied the contents from this .pem file to the end of cacert.pem. However this did not work.
>>> requests.get('https://egov.uscis.gov/casestatus/landing.do')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Users\Sandra\Anaconda\lib\site-packages\requests\api.py", line 69, in get
return request('get', url, params=params, **kwargs)
File "C:\Users\Sandra\Anaconda\lib\site-packages\requests\api.py", line 50, in request
response = session.request(method=method, url=url, **kwargs)
File "C:\Users\Sandra\Anaconda\lib\site-packages\requests\sessions.py", line 465, in request
resp = self.send(prep, **send_kwargs)
File "C:\Users\Sandra\Anaconda\lib\site-packages\requests\sessions.py", line 573, in send
r = adapter.send(request, **kwargs)
File "C:\Users\Sandra\Anaconda\lib\site-packages\requests\adapters.py", line 431, in send
raise SSLError(e, request=request)
requests.exceptions.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:581)
Any pointers as to how I can overcome this without resorting to verify=False
Also Is there any difference in downloading the file via https://superuser.com/a/97203 and https://superuser.com/a/176721?
Because I have no issue with requests.get('https://www.google.com'), do other websites place restrictions on the certificate you download?
egov.usics.gov does not provide a complete chain in its SSL handshake.
You'll need to employ a workaround similar to what's suggested here until the site administrator fixes the certificate chain issue. The intermediate certificate in your case can be obtained from https://ssl-tools.net/certificates/yuox7i-symantec-class-3-secure-server-ca
There are three ways to setup CA cert:
$ pip install certifi then
>>> requests.get(url, verify=certifi.where())
>>> requests.get(url, verify='/path/to/cert_bundle_file')
>>> os.environ['REQUESTS_CA_BUNDLE'] = '/path/to/cert_bundle_file'
>>> requests.get(url)
Related
I'm am pretty much following the Google Sheets getting started (in Python) to a tee. I've gotten the program to work on my Mac laptop, but it is failing as I am trying to run it on Windows. So far, I've checked that we do not have the firewall enabled on the machine.
Below is the error that appears after clicking through the authentication prompts that pop up in the browser.
Traceback (most recent call last):
File "Authenticate.py", line 47, in <module>
main()
File "Authenticate.py", line 43, in main
tools.run_flow(flow, store)
File "C:\johnsnow\packages\test\lib\site-packages\oauth2client\_helpers.py", line 133, in positional_wrapper
return wrapped(*args, **kwargs)
File "C:\johnsnow\packages\test\lib\site-packages\oauth2client\tools.py", line 243, in run_flow
credential = flow.step2_exchange(code, http=http)
File "C:\johnsnow\packages\test\lib\site-packages\oauth2client\_helpers.py", line 133, in positional_wrapper
return wrapped(*args, **kwargs)
File "C:\johnsnow\packages\test\lib\site-packages\oauth2client\client.py", line 2054, in step2_exchange
http, self.token_uri, method='POST', body=body, headers=headers)
File "C:\johnsnow\packages\test\lib\site-packages\oauth2client\transport.py", line 282, in request
connection_type=connection_type)
File "C:\johnsnow\packages\test\lib\site-packages\httplib2\__init__.py", line 1570, in request
(response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
File "C:\johnsnow\packages\test\lib\site-packages\httplib2\__init__.py", line 1317, in _request
(response, content) = self._conn_request(conn, request_uri, method, body, headers)
File "C:\johnsnow\packages\test\lib\site-packages\httplib2\__init__.py", line 1252, in _conn_request
conn.connect()
File "C:\johnsnow\packages\test\lib\site-packages\httplib2\__init__.py", line 1044, in connect
raise SSLHandshakeError(e)
httplib2.SSLHandshakeError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:661)
I'm not exactly sure how it happened as I was working in a virtualenv, but there was a dependency issue in the version of httplib2 and oauth2. After an uninstall and then reinstall, there was an error that the two libraries were incompatible.
Doing pip freeze showed I had httplib2==0.8 but oauth2client required httplib2=>0.9.
This was resolved by doing pip install --upgrade httplib2.
I ssh to my EC2 instance. I can run these commands and they work perfectly:
aws sqs list-queues
aws s3 ls
I have a small Python script that pulls data from a database, formats it as XML, and then uploads the file to S3. This upload fails with this error:
Traceback (most recent call last):
File "./data_test/data_analytics/lexisnexis/async2.py", line 289, in <module>
insert_parallel(engine, qy, Create_Temp.profile_id, nworkers)
File "./data_test/data_analytics/lexisnexis/async2.py", line 241, in insert_parallel
s3upload(bucketname, keyname, f)
File "./data_test/data_analytics/lexisnexis/async2.py", line 89, in s3upload
bucket = conn.get_bucket(bucketname)
File "/usr/lib/python2.7/dist-packages/boto/s3/connection.py", line 506, in get_bucket
return self.head_bucket(bucket_name, headers=headers)
File "/usr/lib/python2.7/dist-packages/boto/s3/connection.py", line 525, in head_bucket
response = self.make_request('HEAD', bucket_name, headers=headers)
File "/usr/lib/python2.7/dist-packages/boto/s3/connection.py", line 668, in make_request
retry_handler=retry_handler
File "/usr/lib/python2.7/dist-packages/boto/connection.py", line 1071, in make_request
retry_handler=retry_handler)
File "/usr/lib/python2.7/dist-packages/boto/connection.py", line 1030, in _mexe
raise ex
SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590)
How can I have a script that dies, even when aws cli works?
To be clear, I'm running the Python script as the same user, from the same EC2 instance, as I run the aws cli commands.
aws --version
aws-cli/1.11.176 Python/2.7.12 Linux/4.9.43-17.38.amzn1.x86_64 botocore/1.7.34
The last line of your error messages tells you the problem:
SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590)
Your issue could be one of the following:
1) There is an error with the certificate with the server that you are connecting to.
2) The certificate chain is incomplete for the server that you are connecting to.
3) You are missing "cacert.pem". Do a Google search on "cacert.pem". This is a common problem and there is a lot of information on downloading and installing this file.
Certificate verification in Python
I'm using Python 2.7.13. I'm trying to connect to https://www.python.org/ and verify its certificate using urllib2.urlopen. I'm getting "SSL: CERTIFICATE_VERIFY_FAILED" error in the process, and when I try to google the issue, I seem to get answers on how to bypass this security check. But I don't want my code to bypass it, I want it to use the certificate. Here's my code that fails:
import urllib2
from contextlib import closing
import ssl
ctx = ssl.create_default_context(purpose = ssl.Purpose.SERVER_AUTH, cafile = 'www.python.org.crt')
request = urllib2.Request('https://www.python.org/')
# Produces URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:661)>
with closing(urllib2.urlopen(request, context = ctx)) as response:
print response.read()
cafile = 'www.python.org.crt' is a PEM-formatted file that properly starts with -----BEGIN CERTIFICATE-----. I exported it from https://www.python.org/ using this instruction and put it into the script's working folder.
From the documentation: https://docs.python.org/2/library/ssl.html#ssl._https_verify_certificates
Starting with Python 2.7.9, httplib and modules which use it, such as urllib2 and xmlrpclib, default to verifying remote server certificates received when establishing client HTTPS connections. This default verification checks that the certificate is signed by a Certificate Authority in the system trust store and that the Common Name (or Subject Alternate Name) on the presented certificate matches the requested host.
So you don't need to anything special to verify the certificate. And you can see that because you have a validation failure, which shows you that validation is happening.
So why aren't you getting successful validation? The answer to this lies in your use of the cafile. The cafile parameter takes a file which overrides your systems default trusted certificate store. But the important thing to understand about trusted certificates the validation process isn't to check the host is in the trusted store but signed but a cert in the trusted store. So you don't need to pass in the actual server certificate, you would pass in the certificate from python.org's certificate vendors (digicert at the time of writing) .
But actually you don't even need to do this as you system should come with a large set of trusted certificates from many vendors. These vendors are audited to some extent ( Opinions vary on the extent - and this now getting off topic) and python's library will validate against this set as default if you leave the cafile parameter blank.
So you find that
>>> ctx = ssl.create_default_context(purpose = ssl.Purpose.SERVER_AUTH, cafile = '/tmp/python.org.cert')
>>> urllib2.urlopen("https://www.python.org/", context = ctx)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python2.7/urllib2.py", line 154, in urlopen
return opener.open(url, data, timeout)
File "/usr/lib/python2.7/urllib2.py", line 429, in open
response = self._open(req, data)
File "/usr/lib/python2.7/urllib2.py", line 447, in _open
'_open', req)
File "/usr/lib/python2.7/urllib2.py", line 407, in _call_chain
result = func(*args)
File "/usr/lib/python2.7/urllib2.py", line 1241, in https_open
context=self._context)
File "/usr/lib/python2.7/urllib2.py", line 1198, in do_open
raise URLError(err)
urllib2.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:661)>
fails, but this:
>>> ctx = ssl.create_default_context(purpose = ssl.Purpose.SERVER_AUTH,)
>>> urllib2.urlopen("https://www.python.org/", context = ctx)
<addinfourl at 140556044048504 whose fp = <socket._fileobject object at 0x7fd5c1127b50>>
works.
Finally I'd like to point you at httplib and http.client (in python 3) which are much cleaner interfaces for this sort of thing.
Recently I've started working at a large company. I've installed python and am now trying to install packages (using pip, which is installed) but get the following error (numpy is an example, but this error occurs for every package):
> pip install numpy
Collecting numpy
Could not find a version that satisfies the requirement numpy (from versions: )
No matching distribution found for numpy
Because I have the idea that I get the error because I'm in a corporate network I tried the following (because someone else on StackOverflow did so as well):
> pip -vvv install numpy
Collecting numpy
Getting page https://pypi.python.org/simple/numpy/
Starting new HTTPS connection (1): pypi.python.org
Could not fetch URL https://pypi.python.org/simple/numpy/: connection error: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590) - skipping
1 location(s) to search for versions of numpy:
* https://pypi.python.org/simple/numpy/
Getting page https://pypi.python.org/simple/numpy/
Starting new HTTPS connection (2): pypi.python.org
Could not fetch URL https://pypi.python.org/simple/numpy/: connection error: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590) - skipping
Could not find a version that satisfies the requirement numpy (from versions: )
Cleaning up...
No matching distribution found for numpy
Exception information:
Traceback (most recent call last):
File "c:\python27\lib\site-packages\pip\basecommand.py", line 211, in main
status = self.run(options, args)
File "c:\python27\lib\site-packages\pip\commands\install.py", line 294, in run
requirement_set.prepare_files(finder)
File "c:\python27\lib\site-packages\pip\req\req_set.py", line 334, in prepare_files
functools.partial(self._prepare_file, finder))
File "c:\python27\lib\site-packages\pip\req\req_set.py", line 321, in _walk_req_to_install
more_reqs = handler(req_to_install)
File "c:\python27\lib\site-packages\pip\req\req_set.py", line 461, in _prepare_file
req_to_install.populate_link(finder, self.upgrade)
File "c:\python27\lib\site-packages\pip\req\req_install.py", line 250, in populate_link
self.link = finder.find_requirement(self, upgrade)
File "c:\python27\lib\site-packages\pip\index.py", line 571, in find_requirement
'No matching distribution found for %s' % req
DistributionNotFound: No matching distribution found for numpy
Starting new HTTPS connection (1): pypi.python.org
There was an error checking the latest version of pip
Traceback (most recent call last):
File "c:\python27\lib\site-packages\pip\utils\outdated.py", line 126, in pip_version_check
headers={"Accept": "application/json"},
File "c:\python27\lib\site-packages\pip\_vendor\requests\sessions.py", line 477, in get
return self.request('GET', url, **kwargs)
File "c:\python27\lib\site-packages\pip\download.py", line 373, in request
return super(PipSession, self).request(method, url, *args, **kwargs)
File "c:\python27\lib\site-packages\pip\_vendor\requests\sessions.py", line 465, in request
resp = self.send(prep, **send_kwargs)
File "c:\python27\lib\site-packages\pip\_vendor\requests\sessions.py", line 573, in send
r = adapter.send(request, **kwargs)
File "c:\python27\lib\site-packages\pip\_vendor\cachecontrol\adapter.py", line 46, in send
resp = super(CacheControlAdapter, self).send(request, **kw)
File "c:\python27\lib\site-packages\pip\_vendor\requests\adapters.py", line 431, in send
raise SSLError(e, request=request)
SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590)
Can anybody tell me what is going wrong and how I can fix this? My work laptop runs windows 10.
I would be very grateful if somebody can help.
The problem has been solved. The company uses a proxy, if this is temporarily turned off I can install packages without a problem.
I hope this helps others in the same situation
Why google adwords api stops on call this link:
https://adwords.google.com/api/adwords/mcm/v201502/CustomerService?wsdl
With this error - should I load some certificate before and how?
urllib2.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590)>
Using Python 2.7.10.
Full source code:
create_adwords_client_without_yaml.py
Full error code:
Traceback (most recent call last):
File "C:/Users/Crezary Wagner/PycharmProjects/learn-adwords/src/examples/create_adwords_client_without_yaml.py", line 56, in <module>
CLIENT_CUSTOMER_ID)
File "C:/Users/Crezary Wagner/PycharmProjects/learn-adwords/src/examples/create_adwords_client_without_yaml.py", line 50, in main
customer = adwords_client.GetService('CustomerService').get()
File "C:\root\Python27\lib\site-packages\googleads\adwords.py", line 256, in GetService
proxy=proxy_option, cache=self.cache, timeout=3600)
File "C:\root\Python27\lib\site-packages\suds\client.py", line 115, in __init__
self.wsdl = reader.open(url)
File "C:\root\Python27\lib\site-packages\suds\reader.py", line 150, in open
d = self.fn(url, self.options)
File "C:\root\Python27\lib\site-packages\suds\wsdl.py", line 136, in __init__
d = reader.open(url)
File "C:\root\Python27\lib\site-packages\suds\reader.py", line 74, in open
d = self.download(url)
File "C:\root\Python27\lib\site-packages\suds\reader.py", line 92, in download
fp = self.options.transport.open(Request(url))
File "C:\root\Python27\lib\site-packages\suds\transport\https.py", line 62, in open
return HttpTransport.open(self, request)
File "C:\root\Python27\lib\site-packages\suds\transport\http.py", line 67, in open
return self.u2open(u2request)
File "C:\root\Python27\lib\site-packages\suds\transport\http.py", line 132, in u2open
return url.open(u2request, timeout=tm)
File "C:\root\Python27\lib\urllib2.py", line 431, in open
response = self._open(req, data)
File "C:\root\Python27\lib\urllib2.py", line 449, in _open
'_open', req)
File "C:\root\Python27\lib\urllib2.py", line 409, in _call_chain
result = func(*args)
File "C:\root\Python27\lib\urllib2.py", line 1240, in https_open
context=self._context)
File "C:\root\Python27\lib\urllib2.py", line 1197, in do_open
raise URLError(err)
urllib2.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590)>
Python uses certificates from 'system ssl certificate store' to certify https connection, if there is not any appropriate ssl cert in the store error occurs like this.
Download ssl certificate (Open your https link in browser and click lock icon in address bar > More Information > View Certificate > Details > Export) and install it on your system as stated this link http://windows.microsoft.com/en-us/windows/import-export-certificates-private-keys#1TC=windows-7
Not sure if that's the problem here, but worth checking it.
Python 2.7.9 enabled certificate validation by default for HTTP connections.
The server you're connecting to does not have a certificate that is trusted by your client. pysphere should configure SSL appropriately for this use case.
Try making your request like:
requests.get('https://adwords.google.com/api/adwords/mcm/v201502/CustomerService?wsdl', verify=False)
Try this, it helped me:
import ssl
ssl._create_default_https_context = ssl._create_unverified_context
I encountered this issue. I had my phone setup using the same DNS block list and it wasn't immediately apparent after I'd enabled the tool and resumed work on this particular project. I suggest scrutinizing your setup and verify that there aren't any adblockers (DNS level in my case ala NextDNS/hosted PiHole) enabled. Hours upon hours spent trying out python versions, certificates, reinstalling things. Hope this helps someone!