I just found out that I can get a max of 100 records for DBClusterSnapshots, luckily AWS supports pagination where you can get a list by page. I was going over the documentation for aws-sdk-go to see how my Operation implements pagination. Unfortunately there isn't a pagination method for my Operation.
This is the operation I want to paginate. It says in the doc that it supports pagination.
https://docs.aws.amazon.com/cli/latest/reference/rds/describe-db-cluster-snapshots.html
However the pagination method for my operation doesn't appear to be supported
doc: https://docs.aws.amazon.com/sdk-for-go/api/service/rds/
It only supports DBSnapshotsPages but not DBClusterSnapshotsPages
The AWS SDK for Go has the DescribeDBClusterSnapshots function:
func (c *RDS) DescribeDBClusterSnapshots(input *DescribeDBClusterSnapshotsInput) (*DescribeDBClusterSnapshotsOutput, error)
It accepts a parameter of DescribeDBClusterSnapshotsInput, which includes:
Marker *string type:"string"
An optional pagination token provided by a previous DescribeDBClusterSnapshots request. If this parameter is specified, the response includes only records beyond the marker, up to the value specified by MaxRecords.
Therefore, your code can call DescribeDBClusterSnapshots, store the marker that is returned, then make another call to DescribeDBClusterSnapshots, passing in that value for marker. This will return the next 'page' of results.
On aws sdk you can handle pagination by yourself using the response's next_page method to verify there's no pages to retrieve. In order to retrieve the next page of results, attaching some ruby example:
# object initializtion:
rds_client = Aws::RDS::Client.new
# implementation:
def self.describe_all_db_snapshots(db_instance_identifier: db_instance_identifier)
response = rds_client.describe_db_snapshots({
db_instance_identifier: db_instance_identifier,
snapshot_type: "automated",
include_shared: false,
include_public: false,
max_records: 100 })
while response.next_page? do
# use the response data here...
puts #{response}
# next pagination iterator
response = response.next_page
end
end
For more details read aws sdk documentation.
Related
I'm looking for a way to fetch Media Insights metrics in Instagram Graph API (https://developers.facebook.com/docs/instagram-api/reference/media/insights) with a nested query based on the userId, even when a client switched from a Personal to a Business account.
I use this nested query to fetch all the data I need : https://graph.facebook.com/v3.2/{userId}?fields=followers_count,media{media_type,caption,timestamp,like_count,insights.metric(reach, impressions)} (this part causes the error: insights.metric(reach, impressions) - it works however for an account that has always been a Business one)
However, because some media linked to the userId were posted before the user switched to a Business account, instead of returning the data only for the media posted after, the API returns this error:
{
"error": {
"message": "Invalid parameter",
"type": "OAuthException",
"code": 100,
"error_data": {
"blame_field_specs": [
[
""
]
]
},
"error_subcode": 2108006,
"is_transient": false,
"error_user_title": "Media Posted Before Business Account Conversion",
"error_user_msg": "The media was posted before the most recent time that the user's account was converted to a business account from a personal account.",
"fbtrace_id": "Gs85pUz14JC"
}
}
Is there a way to know, thru the API, which media were created before and after the account switch from Personal to Business? Or is there a way to fetch the date on which the account was switched?
The only way I currently see to get the data I need is to use the /media edge and query insights for each media until I get an error. Then I would get approximately the date I need. However, this is not optimized at all since we are rate limited to 200 calls per user per hour.
I have the same problem.
For now, I'm Switch between queries (if first have error)
"userId"?fields=id,media.limit(100){insights.metric(reach, impressions)}
"userId"?fields=id,media.limit(100)
I show the user all insights in zero.
I don't know if they're the best alternative, like identify the time of conversion to business and get the post between this range of DateTime
I got the same problem and solved it like this:
Use the nested query just like you did, including insights.metric
If the error appears, do another call without insights.metric - to at least get all other data
For most accounts, it works and there is no additional API call. For the rest, i just cannot get the insights and i have to live with it, i guess - until Facebook/IG fixes the issue.
I got the same problem and solved it like this:
Step1: Convert your Instagram account to a Professional account.
Step2: Then According to Error Post a new post on Instagram and get their Post-ID.
Step3: Then try to get a request using that Post-ID.
{Post-ID}?fields=comments_count,like_count,timestamp,insights.metric(reach,impressions)
curl -i -X GET "https://graph.facebook.com/v12.0/{Post-ID}?fields=comments_count%2Clike_count%2Ctimestamp%2Cinsights.metric(reach%2Cimpressions)&access_token={access_token}"
For more: insights
Here is the relevant logic from a script that can handle this error while still doing a full import. It works by reducing the requested limit to 1 once the error is encountered. It will keep requesting insights until it encounters the error again, then removes insights from the fields and returns to the requested limit.
limit = 50
error_2108006 = False
metrics = 'insights.metric%28impressions%29%2C' # Must be URL encoded for replacement
url = '/PAGE_ID/media?fields=%sid,caption,media_url,media_type&limit=%s' % (metrics, limit)
# While we have more pages
while True:
# Make your API call to Instagram
posts = get_posts_from_instagram(url)
# Check for error 2108006
if posts == 2108006:
# First time getting this error, keep trying to get insights but one by one
if error_2108006 is False:
error_2108006 = True
url = url.replace('limit={}'.format(limit), 'limit=1')
continue
# Not the first time. Strip out insights and return to desired limit.
url = url.replace(metrics, '')
url = url.replace('limit=1', 'limit='.format(limit))
continue
# Do something with the data
for post in posts:
continue
# If there are more pages, fetch the next URL
if 'paging' in posts and 'next' in posts['paging']:
url = posts['paging']['next']
continue
# Done
break
I have 2 requests
1st Request
After did my first request, I get the response where I can parse for a taskId
In my test tab, I will then parse and store it like this
let taskId = pm.response.json().body.result[0].data.task
console.log(taskId)
I can see taskId printing in my console as 938
2nd Request
I require making a GET with this dynamic URL with the taskId that I got from the first one
http://localhost:3000/fortinet/monitor/{{taskId}}
So I set the above URL , set the HTTP verb to GET
in my Pre-request Script tab, I did this
let taskId = pm.globals.get("taskId")
Result
ReferenceError: taskId is not defined
Image Result
How can I debug this further?
The most suggested way is to use :key as in
http://localhost:3000/fortinet/monitor/:taskId
See the colon before taskId. The reason being, URI values sometimes many not be environment dependent. So, based on the usecase, you can use like I said or {{taskId}}
You have to set variable, but you are doing it wrong.
try this:
pm.globals.set("taskID", pm.response.json().body.result[0].data.task)
more you can read here:
https://learning.postman.com/docs/postman/variables-and-environments/variables/
Please note, that URL which ends with resource identified like https://example.com/:pathVariable.xml or https://example.com/:pathVariable.json will not work.
You can go with https://example.com/:pathVariable with Accept: application/json header.
For passing dynamic value, first you have to set it in environment or global variable in Tests tab because tests runs after request and you will get response value after request sent, but because you get response in json you have to first parse it, so what you can write in Tests tab is as follows:
var jsonData = JSON.parse(responseBody);
postman.setEnvironmentVariable("taskId", jsonData.token); // OR
postman.setGlobalVariable("taskId", jsonData.token);
Then you can use taskId as {{taskId}} wherever you want in url parameters or in request body or form data wherever.
If you want to know in detail how to extract data from response and chain it to request then you can go to this postman's official blog post which is written by Abhinav Asthana CEO and Co Founder of Postman Company.
I am trying to extract data from an API but in the documentation I was only given one API key, which I assume is the private key.
If this is the case how do I make a GET call in python to pull out data for say inventory based on below documentation documentation(can not post entire document) and if no URL is provided?:
Public Inventory API
1.0
[ Base url: https://partner-gateway.staging.mjplatform.com/v1] https://partners.mjfreeway.com/docs/inventory
API data related to inventory management
Schemes
catalog
GET
/catalog
Listing of Sellable Products
This request provides a detailed listing of all sellable products, also referred to throughout the system as "item masters", for the active facility. The listing can be filtered by some simple parameters.
Parameters
Name Description
category_id
integer
(query)
The ‘id’ of a single category to which you want to limit results
subcategory_id
integer
(query)
The ‘id’ of a single subcategory to which you want to limit results
strain_id
integer
(query)
The ‘id’ of a single strain to which you want to limit results
item_number
string
(query)
The item number of a particular item master, i.e. BKSA00000003
uom_type
string
(query)
The method of measurement for the item. Valid options are discrete, weight, and volume
available_online
boolean
(query)
A boolean indicator of whether the item can be sold online
This is my code so far:
import requests
# api-endpoint
URL = "https://www.mjplatform.com/catalog"
# location given here
key = "123abc"
# defining a params dict for the parameters to be sent to the API
PARAMS = {URL:key}
# sending get request and saving the response as response object
r = requests.get(url = URL, params = PARAMS)
# extracting data in json format
data = r.json()
When I run the above I get the following message:
ValueError: No JSON object could be decoded
I am not sure what I am doing to get the data if I am getting a response status 200.
Edit:
Ran print(r.text) got this message:
<!DOCTYPE html><html lang="en"><head><meta http-equiv="X-UA-Compatible" content="IE=edge"><meta charset="utf-8"/><meta name="viewport" content="width=device-width,initial-scale=1"><title>MJ Platform</title><link href="/main.b21e9284629fc8bfb7bc9b4158ad44b9.css" rel="stylesheet"></head><body><div id="defaultLoadingMessage"><div style="height:40px"></div><div class="col-md-4 col-md-offset-4"><div><h1 style="text-align:center">Loading MJ Platform</h1><div class="text-muted" style="text-align:center;width:600px;margin:auto;color:#aaa">If you continue to see this message for more than a few seconds, your browser is most likely out of date or incompatible. We support Chrome and Firefox. Other browsers may work but not provide an optimal experience. <strong>Safari and MS IE are specifically not supported.</strong></div></div></div></div><div id="app"></div><script type="text/javascript" src="/main.cccbe56cf819e9f8a6e3.js"></script></body></html>
How can browser be out of date, if I pulling information into a python anaconda window?
Was given other API information not included in documentation.
I have to fetch the retweets for the tweets and create the JSON file with retweets,user id etc using the python script. Kindly help me to sort it our this issues.
Thanks in advance!!
This task require some fields of knowledge, and since you ask in a general way, I reckon you need a script to run immediately, but setting up this process requires sometime
This part to get connect to twitter API
from twython import Twython, TwythonError
APP_KEY = 'YOUR_APP_KEY'
APP_SECRET = 'YOUR_APP_SECRET'
twitter = Twython(APP_KEY, APP_SECRET)
Use Twitter API call from Twython,
you can find a list here https://twython.readthedocs.io/en/latest/api.html, the param is the same as twitter API
response = twitter.get_retweets(id, 100)
Pagnation
each call to API have limit of returns, in example for engine.get_friends_ids was limited to 5000 (https://dev.twitter.com/rest/reference/get/friends/ids), if you want to get more than 5000, you have to use the cursor in the returned result (if cur = 0 in json returned means no more results), following is example of how to handling cursor
#Set a temp to loop
cur = -1
#Stop when no more result
while cur !=0:
response = twitter.get_friends_ids(user_id=user_id, cursor=cur)
#Some code to handle the response
cur = response["next_cursor"]
API key
Key expires after some calls (https://dev.twitter.com/rest/public/rate-limits), so you need to set some code to auto change your key, or wait for some period (key reached limit return error code 429)
Response
The response from API was in JSON format, which was easy to use, you can access data by selecting base on response[key], in example
reponse["ids"] or response["next_cursor"]
this the code I'm using, is there anyway to make it run faster:
src_uri = boto.storage_uri(bucket, google_storage)
for obj in src_uri.get_bucket():
f.write('%s\n' % (obj.name))
This is an example where it pays to use the underlying Google Cloud Storage API more directly, using the Google API Client Library for Python to consume the RESTful HTTP API. With this approach, it is possible to use request batching to retrieve the names of all objects in a single HTTP request (thereby reducing the extra HTTP request overhead) as well as to use field projection with the objects.get operation (by setting &fields=name) to obtain a partial response so that you aren't sending all the other fields and data over the network (or waiting for retrieval of unnecessary data on the backend).
Code for this would look like:
def get_credentials():
# Your code goes here... checkout the oauth2client documentation:
# http://google-api-python-client.googlecode.com/hg/docs/epy/oauth2client-module.html
# Or look at some of the existing samples for how to do this
def get_cloud_storage_service(credentials):
return discovery.build('storage', 'v1', credentials=credentials)
def get_objects(cloud_storage, bucket_name, autopaginate=False):
result = []
# Actually, it turns out that request batching isn't needed in this
# example, because the objects.list() operation returns not just
# the URL for the object, but also its name, as well. If it had returned
# just the URL, then that would be a case where we'd need such batching.
projection = 'nextPageToken,items(name,selfLink)'
request = cloud_storage.objects().list(bucket=bucket_name, fields=projection)
while request is not None:
response = request.execute()
result.extend(response.items)
if autopaginate:
request = cloud_storage.objects().list_next(request, response)
else:
request = None
return result
def main():
credentials = get_credentials()
cloud_storage = get_cloud_storage_service(credentials)
bucket = # ... your bucket name ...
for obj in get_objects(cloud_storage, bucket, autopaginate=True):
print 'name=%s, selfLink=%s' % (obj.name, obj.selfLink)
You may find the Google Cloud Storage Python Example and other API Client Library Examples helpful in figuring out how to do this. There are also a number of YouTube videos on the Google Developers channel such as Accessing Google APIs: Common code walkthrough that provide walkthroughs.