I found that some of the stock exchanges is not supported for datareader. Example, Singapore. Any workaround?
query = web.DataReader(("SGX:BLA"), 'google', start, now) return such error`
IOError: after 3 tries, Google did not return a 200 for url 'http://www.google.com/finance/historical?q=SGX%3ABLA&startdate=Jan+01%2C+2015&enddate=Apr+20%2C+2016&output=csv
It works for IDX indonesia
query = web.DataReader(("IDX:CASS"), 'google', start, now)
That URL is a 404 - pandas isn't at fault, maybe just check the URL? Perhaps they're on different exchanges with different google finance support.
Related
I'm looking for a way to fetch Media Insights metrics in Instagram Graph API (https://developers.facebook.com/docs/instagram-api/reference/media/insights) with a nested query based on the userId, even when a client switched from a Personal to a Business account.
I use this nested query to fetch all the data I need : https://graph.facebook.com/v3.2/{userId}?fields=followers_count,media{media_type,caption,timestamp,like_count,insights.metric(reach, impressions)} (this part causes the error: insights.metric(reach, impressions) - it works however for an account that has always been a Business one)
However, because some media linked to the userId were posted before the user switched to a Business account, instead of returning the data only for the media posted after, the API returns this error:
{
"error": {
"message": "Invalid parameter",
"type": "OAuthException",
"code": 100,
"error_data": {
"blame_field_specs": [
[
""
]
]
},
"error_subcode": 2108006,
"is_transient": false,
"error_user_title": "Media Posted Before Business Account Conversion",
"error_user_msg": "The media was posted before the most recent time that the user's account was converted to a business account from a personal account.",
"fbtrace_id": "Gs85pUz14JC"
}
}
Is there a way to know, thru the API, which media were created before and after the account switch from Personal to Business? Or is there a way to fetch the date on which the account was switched?
The only way I currently see to get the data I need is to use the /media edge and query insights for each media until I get an error. Then I would get approximately the date I need. However, this is not optimized at all since we are rate limited to 200 calls per user per hour.
I have the same problem.
For now, I'm Switch between queries (if first have error)
"userId"?fields=id,media.limit(100){insights.metric(reach, impressions)}
"userId"?fields=id,media.limit(100)
I show the user all insights in zero.
I don't know if they're the best alternative, like identify the time of conversion to business and get the post between this range of DateTime
I got the same problem and solved it like this:
Use the nested query just like you did, including insights.metric
If the error appears, do another call without insights.metric - to at least get all other data
For most accounts, it works and there is no additional API call. For the rest, i just cannot get the insights and i have to live with it, i guess - until Facebook/IG fixes the issue.
I got the same problem and solved it like this:
Step1: Convert your Instagram account to a Professional account.
Step2: Then According to Error Post a new post on Instagram and get their Post-ID.
Step3: Then try to get a request using that Post-ID.
{Post-ID}?fields=comments_count,like_count,timestamp,insights.metric(reach,impressions)
curl -i -X GET "https://graph.facebook.com/v12.0/{Post-ID}?fields=comments_count%2Clike_count%2Ctimestamp%2Cinsights.metric(reach%2Cimpressions)&access_token={access_token}"
For more: insights
Here is the relevant logic from a script that can handle this error while still doing a full import. It works by reducing the requested limit to 1 once the error is encountered. It will keep requesting insights until it encounters the error again, then removes insights from the fields and returns to the requested limit.
limit = 50
error_2108006 = False
metrics = 'insights.metric%28impressions%29%2C' # Must be URL encoded for replacement
url = '/PAGE_ID/media?fields=%sid,caption,media_url,media_type&limit=%s' % (metrics, limit)
# While we have more pages
while True:
# Make your API call to Instagram
posts = get_posts_from_instagram(url)
# Check for error 2108006
if posts == 2108006:
# First time getting this error, keep trying to get insights but one by one
if error_2108006 is False:
error_2108006 = True
url = url.replace('limit={}'.format(limit), 'limit=1')
continue
# Not the first time. Strip out insights and return to desired limit.
url = url.replace(metrics, '')
url = url.replace('limit=1', 'limit='.format(limit))
continue
# Do something with the data
for post in posts:
continue
# If there are more pages, fetch the next URL
if 'paging' in posts and 'next' in posts['paging']:
url = posts['paging']['next']
continue
# Done
break
I am trying to extract data from an API but in the documentation I was only given one API key, which I assume is the private key.
If this is the case how do I make a GET call in python to pull out data for say inventory based on below documentation documentation(can not post entire document) and if no URL is provided?:
Public Inventory API
1.0
[ Base url: https://partner-gateway.staging.mjplatform.com/v1] https://partners.mjfreeway.com/docs/inventory
API data related to inventory management
Schemes
catalog
GET
/catalog
Listing of Sellable Products
This request provides a detailed listing of all sellable products, also referred to throughout the system as "item masters", for the active facility. The listing can be filtered by some simple parameters.
Parameters
Name Description
category_id
integer
(query)
The ‘id’ of a single category to which you want to limit results
subcategory_id
integer
(query)
The ‘id’ of a single subcategory to which you want to limit results
strain_id
integer
(query)
The ‘id’ of a single strain to which you want to limit results
item_number
string
(query)
The item number of a particular item master, i.e. BKSA00000003
uom_type
string
(query)
The method of measurement for the item. Valid options are discrete, weight, and volume
available_online
boolean
(query)
A boolean indicator of whether the item can be sold online
This is my code so far:
import requests
# api-endpoint
URL = "https://www.mjplatform.com/catalog"
# location given here
key = "123abc"
# defining a params dict for the parameters to be sent to the API
PARAMS = {URL:key}
# sending get request and saving the response as response object
r = requests.get(url = URL, params = PARAMS)
# extracting data in json format
data = r.json()
When I run the above I get the following message:
ValueError: No JSON object could be decoded
I am not sure what I am doing to get the data if I am getting a response status 200.
Edit:
Ran print(r.text) got this message:
<!DOCTYPE html><html lang="en"><head><meta http-equiv="X-UA-Compatible" content="IE=edge"><meta charset="utf-8"/><meta name="viewport" content="width=device-width,initial-scale=1"><title>MJ Platform</title><link href="/main.b21e9284629fc8bfb7bc9b4158ad44b9.css" rel="stylesheet"></head><body><div id="defaultLoadingMessage"><div style="height:40px"></div><div class="col-md-4 col-md-offset-4"><div><h1 style="text-align:center">Loading MJ Platform</h1><div class="text-muted" style="text-align:center;width:600px;margin:auto;color:#aaa">If you continue to see this message for more than a few seconds, your browser is most likely out of date or incompatible. We support Chrome and Firefox. Other browsers may work but not provide an optimal experience. <strong>Safari and MS IE are specifically not supported.</strong></div></div></div></div><div id="app"></div><script type="text/javascript" src="/main.cccbe56cf819e9f8a6e3.js"></script></body></html>
How can browser be out of date, if I pulling information into a python anaconda window?
Was given other API information not included in documentation.
Below is my working code to get twitter followers for certain accounts (#hudsonci in this case).
My issue is the time that it is taking to pull in all of these followers. This account specifically has approx 1,000 followers ... I can only get to 300 at a time with the rate limiting restrictions. So, it is taking > an hour to get all the followers for this account. I can imagine this will become a huge pain in the ass for large accounts.
I am looking for some suggestions for how I can improve this. I feel like I am not taking full advantage of the pagination cursor, but I can't be sure.
any help is appreciated.
#!/usr/bin/env python
# encoding: utf-8
import tweepy
import time
#Twitter API credentials
consumer_key = "mine"
consumer_secret = "mine"
access_key = "mine"
access_secret = "mine"
#authorize twitter, initialize tweepy
auth = tweepy.OAuthHandler(consumer_key, consumer_secret)
auth.set_access_token(access_key, access_secret)
api = tweepy.API(auth)
def handle_errors(cursor):
while True:
try:
yield cursor.next()
except tweepy.TweepError:
time.sleep(20 * 60)
for user in handle_errors(tweepy.Cursor(api.followers,screen_name='hudsonci').items()):
print user.screen_name
As per the Twitter documentation for followers you need to use the count parameter.
Specifies the number of IDs attempt retrieval of, up to a maximum of 5,000 per distinct request.
So, adding count=5000 should help you.
You are getting 300 followers at a time because getting the followers object (as opposed to IDs only) has a page limit 20. With 15 requests per window, that comes out to be 300 followers.
Here are the docs for followers: https://developer.twitter.com/en/docs/accounts-and-users/follow-search-get-users/api-reference/get-followers-list
I would like to request tweets on a specific topic (for example: "cancer"), using Python Tweepy. But usually its time can only be specified by a specific day, for example.
startSince = '2014-10-01'
endUntil = '2014-10-02'
for tweet in tweepy.Cursor(api.search, q="cancer",
since=startSince, until=endUntil).items(999999999):
Is there a way to specify the time so I can collect "cancer" tweets between 2014-10-01 00:00:00 and 2014-10-02 12:00:00? This is for my academic research: I was able to collect cancer tweets for the last month, but the sudden burst of quantity of "breast cancer" tweets due to the cancer awareness month breaks my script and I have to collect them in different time segments, and I will not be able to retrieve the tweets for Oct 01, 2014 if I can't figure it out soon.
There is no way that I've found to specific a time using since/until.
You can hack your way around this using since_id & max_id.
If you can find a tweet made at around the times you want, you can restrict you search to those made after since_id and before max_id
import tweepy
consumer_key = 'aaa'
consumer_secret = 'bbb'
access_token = 'ccc'
access_token_secret = 'ddd'
# OAuth process, using the keys and tokens
auth = tweepy.OAuthHandler(consumer_key, consumer_secret)
auth.set_access_token(access_token, access_token_secret)
api = tweepy.API(auth)
results = api.search(q="cancer", since_id=518857118838181000, max_id=518857136202194000)
for result in results:
print result.text
For some reason this query always returned a standard string (json). For some reason starting today (Jan 24,2014) the results are GZIPped. Easy fix but I was not aware of any changes to the Graph API regarding output. Has anyone else heard of this or is experiencing this?
req = urllib2.Request(x)
req = 'https://graph.facebook.com/search?q=%22are%20eligible%20for%20disability%20compensation%20from%20the%20Department%20of%22&limit=250&type=post&until=1390665731&access_token=xxxxxxxxxxxxxxxxxxxxX2HTOgagZDZD&locale=en_US'
response = urllib2.urlopen(req)
the_page = response.read()
print the_page
sys.exit(0)
Developer NOTE: Please do not post access tokens or secret keys in Q and A.
We were having the same issue. Resolved itself 15 mins ago.