I'm trying to request a RestAPI resource multiple times. In order to save time, I try to use urllib3.HTTPSConnectionPool instead of urllib2. However, it keeps throwing me the following error:
Traceback (most recent call last):
File "LCRestapi.py", line 135, in <module>
listedLoansFast(version, key, showAll='false')
File "LCRestapi.py", line 55, in listedLoansFast
pool.urlopen('GET',url+resource,headers={'Authorization':key})
File "/Library/Python/2.7/site-packages/urllib3/connectionpool.py", line 515, in urlopen
raise HostChangedError(self, url, retries)
urllib3.exceptions.HostChangedError: HTTPSConnectionPool(host='https://api.lendingclub.com/api/investor/v1/loans/listing?showAll=false', port=None): Tried to open a foreign host with url: https://api.lendingclub.com/api/investor/v1/loans/listing?showAll=false
I'm using python-2.7.6
Here's my code:
manager = urllib3.PoolManager(1)
url = 'https://api.lendingclub.com/api/investor/v1/loans/listing?showAll=false'
pool = urllib3.HTTPSConnectionPool(url+resource, maxsize=1, headers={'Authorization':key})
r = pool.request('GET',url+resource)
print r.data
Thanks for the help!
The problem is that you're creating a PoolManager but never using it. Instead, you're also creating an HTTPSConnectionPool (which is bound to a specific host) and using that instead of the PoolManager. The PoolManager will automatically manage HTTPSConnectionPool objects on your behalf, so you don't need to worry about it.
This should work:
# Your example called this `manager`
http = urllib3.PoolManager()
url = 'https://api.lendingclub.com/api/investor/v1/loans/listing?showAll=false'
headers = {'Authorization': key}
# Your example did url+resource, but let's assume the url variable
# contains the combined absolute url.
r = http.request('GET', url, headers=headers)
print r.data
You can specify the size for the PoolManager if you'd prefer, but you shouldn't need to unless you're trying to do something unusual with limiting resources to a pool of threads.
Related
When i try to fetch the details of gmail using google-api's (googleapiclient,oauth2client), i am getting below error:
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python2.7/site-packages/google_api_python_client-1.7.8-py2.7.egg/googleapiclient/_helpers.py", line 130, in positional_wrapper
File "/usr/local/lib/python2.7/site-packages/google_api_python_client-1.7.8-py2.7.egg/googleapiclient/discovery.py", line 224, in build
File "/usr/local/lib/python2.7/site-packages/google_api_python_client-1.7.8-py2.7.egg/googleapiclient/discovery.py", line 274, in _retrieve_discovery_doc
File "/usr/local/lib/python2.7/site-packages/httplib2-0.12.1-py2.7.egg/httplib2/__init__.py", line 2135, in request
cachekey,
File "/usr/local/lib/python2.7/site-packages/httplib2-0.12.1-py2.7.egg/httplib2/__init__.py", line 1796, in _request
conn, request_uri, method, body, headers
File "/usr/local/lib/python2.7/site-packages/httplib2-0.12.1-py2.7.egg/httplib2/__init__.py", line 1707, in _conn_request
raise ServerNotFoundError("Unable to find the server at %s" % conn.host)
httplib2.ServerNotFoundError: Unable to find the server at www.googleapis.com
but it is working fine in my pc but not from remote location.
code:
from googleapiclient.discovery import build
from oauth2client.service_account import ServiceAccountCredentials
credentials = ServiceAccountCredentials.from_json_keyfile_name(
"quickstart-1551349397232-e8bcb3368ae1.json", scopes=
['https://www.googleapis.com/auth/admin.directory.group', 'https://www.googleapis.com/auth/admin.directory.user', 'https://www.googleapis.com/auth/admin.directory.domain', 'https://www.googleapis.com/auth/gmail.readonly'])
delegated_credentials = credentials.create_delegated('jango#carbonitedepot.com')
DIRECOTORY = build('admin', 'directory_v1', credentials=delegated_credentials)
try:
results = DIRECOTORY.users().list(customer='my_customer').execute()
users = results.get('users', [])
res = []
for info in users:
print(info)
res.append(info.get("primaryEmail"))
print(res)
except Exception as e:
print(e)
Any help would be much appreciated.
Thanks in advance.
I had the same issue and I searched a lot to get it fixed, turns out I was at fault here.
I don't know from where there was podman installed instead of docker and that caused the problem.
I suggest you check your docker version and make sure the latest is running at the server, otherwise, the library works fine! Please let me know if you still face this issue, would like to dig deep!
The httplib2 is likely scanning through network interface DHCP DNS nameservers (in your registry or visible to docker) and then trying to connect through a stale DNS or possible an IPv6. Patching your socket in the main routine may solve it:
# Monkey patch to force IPv4, since FB seems to hang on IPv6
import socket
old_getaddrinfo = socket.getaddrinfo
def new_getaddrinfo(*args, **kwargs):
responses = old_getaddrinfo(*args, **kwargs)
return [response
for response in responses
if response[0] == socket.AF_INET]
socket.getaddrinfo = new_getaddrinfo
am novice in python.Extracted below code to login to website from an online post, but getting error.
Please help to fix it and an explanation will help me
import requests
with requests.Session() as c:
EMAIL = 'noob.python#gmail.com'
PASSWORD = 'Dabc#123'
URL = 'https://www.linkedin.com/'
c.get(URL)
token = c.cookies['CsrfParam']
# This is the form data that the page sends when logging in
login_data = {loginCsrfParam:token, session_key:EMAIL, session_password:PASSWORD}
# Authenticate
r = c.post(URL, data=login_data)
# Try accessing a page that requires you to be logged in
r = c.get('https://www.linkedin.com/feed/')
print r.content
Am stuck with below Error:
C:\Python27>python website.py
Traceback (most recent call last):
File "website.py", line 8, in <module>
token = c.cookies['CsrfParam']
File "C:\Python27\lib\site-packages\requests\cookies.py", line 329, in __getitem__
return self._find_no_duplicates(name)
File "C:\Python27\lib\site-packages\requests\cookies.py", line 400, in _find_no_duplicates
raise KeyError('name=%r, domain=%r, path=%r' % (name, domain, path))
KeyError: "name='CsrfParam', domain=None, path=None"
The reason you're getting the error is that you're calling a value from a list which is empty. To call the first item in the list you say list[0]. In this case the list you're calling is empty so the first value doesn't exist hence the error.
I've ran your code and there is no #id value of 'recaptcha-token' which is why the code is returning an empty list. The only place a recaptcha token is needed is for signing up so I would suggest trying to log in without creating the authenticity_token.
I want get list of all groups in my contacts. I use that code:
import gdata.gauth
import gdata.contacts.client
token = gdata.gauth.OAuth2Token(client_id = "***.apps.googleusercontent.com",
client_secret = "***",
scope = "https://www.google.com/m8/feeds/",
user_agent = "GC")
gd_client = gdata.contacts.client.ContactsClient(source = 'GCv0.1')
gd_client = token.authorize(gd_client)
gd_client.GetGroups()
But got error:
Traceback (most recent call last):
File "F:/Yandex/Sites/GoogleContacts/cli_contacts.py", line 27, in <module>
gd_client.GetGroups()
File "C:\Users\Ishayahu\27Gdata\lib\site-packages\gdata\contacts\client.py", line 218, in get_groups
return self.get_feed(uri, desired_class=desired_class, auth_token=auth_token, **kwargs)
File "C:\Users\Ishayahu\27Gdata\lib\site-packages\gdata\client.py", line 640, in get_feed
**kwargs)
File "C:\Users\Ishayahu\27Gdata\lib\site-packages\gdata\client.py", line 319, in request
RequestError)
gdata.client.RequestError: Server responded with: 400,
enter code here
I have no idea what a reason and I can't find any clue how to solve it.
UPD: It looks like I should somehow put acess_token or refresh_token into OAuth2Token, but I can't understand how
so it send headers
{'GData-Version': '3', 'Authorization': 'Bearer None', 'User-Agent': 'gdata-py/2.0.17'}
UPD2: by the way, if I test in in OAuth2Playground, it shows me a page with request access to my contacts. That script doesn't ask for it. Maybe that's the problem? How can I change it? I thought, it connect with url_redirect, but I can't uderstand, how to use it
UPD3: I was right: if I add access_token, which I get manualy from Playground, all works. But how should I get it in script?!
I am working on a simple python scraping script, I am trying to get connections from LinkedIn using their API without a redirect_uri. I worked once with some APIs, that don't require the redirect url or just https://localhost. I got the consumer_key, consumer_secret, user_secret, consumer_secret. Here's the code i am using from https://github.com/ozgur/python-linkedin:
RETURN_URL = ''
url = 'https://api.linkedin.com/v1/people/~'
# Instantiate the developer authentication class
authentication = linkedin.LinkedInDeveloperAuthentication(CONSUMER_KEY, CONSUMER_SECRET,
USER_TOKEN, USER_SECRET,
RETURN_URL, linkedin.PERMISSIONS.enums.values())
# Pass it in to the app...
application = linkedin.LinkedInApplication(authentication)
print application.get_profile() # works
print application.get_connections()
And here's the error I get:
Traceback (most recent call last):
File "getContacts.py", line 20, in <module>
print application.get_connections()
File "/home/imane/Projects/prjL/env/local/lib/python2.7/site-packages/linkedin/linkedin.py", line 219, in get_connections
raise_for_error(response)
File "/home/imane/Projects/prjL/env/local/lib/python2.7/site-packages/linkedin/utils.py", line 63, in raise_for_error
raise LinkedInError(message)
linkedin.exceptions.LinkedInError: 403 Client Error: Forbidden for url: https://api.linkedin.com/v1/people/~/connections: Unknown Error
This is my first question here, so excuse me if I didn't make it clear enough, and thank you for helping me out.
Here's what i tried with python_oauth2:
import oauth2 as oauth
import requests
url = 'https://api.linkedin.com/v1/people/~'
params = {}
token = oauth.Token(key=USER_TOKEN, secret=USER_SECRET)
consumer = oauth.Consumer(key=CONSUMER_KEY, secret=CONSUMER_SECRET)
# Set our token/key parameters
params['oauth_token'] = token.key
params['oauth_consumer_key'] = consumer.key
oauth_request = oauth.Request(method="GET", url=url, parameters=params)
oauth_request.sign_request(oauth.SignatureMethod_HMAC_SHA1(), consumer, token)
signed_url = oauth_request.to_url()
response = requests.get(signed_url)
Connections API calls are a restricted endpoint as of March, 2015. It's possible you're using sample code/documentation that was written at a time when anyone could access those endpoints. You are receiving a 403 response because your application legitimately does not have the permission required to make that request.
I'm running Django 1.5 on GAE. I have a cron job that goes over several thousands of urls and grabs their "likes" count and saves it into DB. It can take easily more than 10 min to complete it. It works when I run it locally as a normal linux cron but fails with this error on GAE:
Traceback (most recent call last):
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/runtime/wsgi.py", line 266, in Handle
result = handler(dict(self._environ), self._StartResponse)
File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/django-1.5/django/core/handlers/wsgi.py", line 255, in __call__
response = self.get_response(request)
File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/django-1.5/django/core/handlers/base.py", line 175, in get_response
signals.got_request_exception.send(sender=self.__class__, request=request)
DeadlineExceededError
My setup:
app.yaml:
- url: /tasks/*
script: myproject.wsgi.application
login: admin
cron.yaml:
- description: update_facebook_resource
url: /tasks/update_facebook_resource
schedule: every day 04:05
timezone: Europe/Berlin
views.py
def update_facebook_resource(request):
resources = Resource.objects.filter(inactive=0).order_by('id')
url_start = "https://graph.facebook.com/fql?q=select++total_count+from+link_stat+where+url%3D"
url_end = "&access_token=..."
for item in resources:
url = item.link
url_final = url_start+ "%22" + url + "%22" + url_end
data = json.load(urllib2.urlopen(url_final))
likes = data["data"][0]["total_count"]
query = Resource.objects.get(id=item.id)
query.facebook_likes = likes
query.save(update_fields=['facebook_likes'])
return http.HttpResponse('ok')
what and how should I change so GAE lets me complete it? I've read this https://developers.google.com/appengine/articles/deadlineexceedederrors but it doesn't give me what I need really.
thanks
It's not a question of just getting GAE to let you complete the function. When developing for App Engine, you do need to think in a slightly different way, precisely because of things like the request deadline. In your case, you need to break the task up into chunks, and process each of those chunks individually.
You don't say if you're using django-nonrel with the GAE datastore, or if you're using Cloud SQL and therefore the standard Django API. If the former, you can use query cursors to keep track of your progress through the Resources. After each chunk, you can use deferred tasks to trigger the next chunk, passing it the cursor so it picks up where the last one left off.