We want to collect some metrics about our client public Facebook pages (~1-5K users) on a daily (or weekly) basis.
I'm talking about 3-5 typical metrics : "likes", "fan posts" etc.
I understand that according to the "Rate Limiting on the Graph API" documentation [1] it's possible to have 200 calls per 1 hour.
For now we don't have any FB public application that can help us to increase this limit. To generate application token we will create it to but I doubt it will have a lot of users.
Does anybody know will we have problems with rate limit exceptions while invoking Graph API more than 200 times per 60 min.?
I guess our expected rate is 5-10K calls per 60 min (once a day).
Phrase from the documentation [1] "Rate limiting in the FB Graph API is encountered only in rare circumstances" gives me hope that it won't be a problem.
Thank you!
[1] https://developers.facebook.com/docs/graph-api/advanced/rate-limiting
You won't have any problems initially. Facebook does not necessarily block apps immediately for being over the limits.
As per their documentation
If your app is making enough calls to be considered for rate limiting by our system, we return an X-App-Usage HTTP header
So, if you don't get any X-App-Usage header,Then your app hasn't be considered "worthy" of throttling by their automated systems yet.
So it would be best to check for this header, while making your api requests. Once you start receiving this Header, it would be best change your frequency of the API calls or give a timeout.
Related
I have a client which has a pretty popular ticket selling service, to the point that the microservice based backend is struggling to keep up, I need to come up with a solution to optimize and loadbalance the system. The infrastructure works through a series of interconnected microservices.
When a user enter the sales channels (mobile or web app), the request is directed to an AWS API Gateway which is in charge of orchestrating the communication towards the microservice in charge of obtaining the requested resources.
These resources are provided from a third party API
This third party has physical servers in each venue in charge of synchronizing the information between the POS systems and the digital sales channels.
We have a REDIS instance in charge of caching these requests that we make to the third party API, we cache each endpoint with a TTL relative to the frequency of updating the information.
Here is some background info:
We get traffic mostly from 2 major countries
On a normal day, about 100 thousands users will use the service, with an 70%/30% traffic relation in between the two countries
On important days, each country has different opening hours (Country A starts sales at 10 am UTC, but country B starts at 5 pm UTC), on these days the traffic increases some n times
We have a main MiddleWare through which all requests made by clients are processed.
We have a REDIS cache database that stores GETs with different TTLs for each endpoint.
We have a MiddleWare that decides to make the request to the cache or to the third party's API, as the case may be.
And these are the complaints I have gotten that need to be deal with:
When a country receives a high amount of requests, the country with the least traffic gets negatively affected, the clients do not respond, or respond partially because the computation layer's limit was exceeded and so the users have a bad experience
Every time the above happens, the computation layer must be manually increased from the infrastructure.
Each request has different response times, stadiums respond in +/- 40 seconds and movie theaters in 3 seconds. These requests enter a queue and are answered in order of arrival.
The error handling is not clear. The errors are mixed up and you can't tell from which country the errors are coming from and how many errors there are
The responses from the third party API are not cached correctly in the cache layer since errors are stored for the time of the TTL
I was thinking of a couple of thinks that I could suggest:
Adding in instrumentation of the requests by using AWS X-Ray
Adding in a separate table for errors in the redis cache layer (old data has to be better than no data for the end user)
Adding in AWS elastic load balancing for the main middleware
But I'm not sure how realistic would be to implement these 3 things, I'm also not sure if they would even solve the problem, I personally don't really have experience with optimizing this type of backed. I would appreciate any suggestions, recommendations, links, documentation, etc. I'm really desperate for a solution to this problem
few thoughts:
When a country receives a high amount of requests, the country with the least traffic gets negatively affected, the clients do not respond, or respond partially because the computation layer's limit was exceeded and so the users have a bad experience
A common approach in aws is to regionalize stack - assuming you are using cdk/cloud formation creating regionalized stack should be a straightforward task.
But it is a question if this will solve the problem. Your system suffers from availability issues, regionalization will isolate this problem down to regions. So we should be able to do better (see below)
Every time the above happens, the computation layer must be manually increased from the infrastructure.
AWS has an option to automatically scale up and down based on traffic patterns. This is a neat feature, given you set limits to make sure you are not overcharged.
Each request has different response times, stadiums respond in +/- 40 seconds and movie theaters in 3 seconds. These requests enter a queue and are answered in order of arrival.
It seems that the large variance is because you have to contact the servers at venues. I recommend to decouple that activity. Basically calls to venues should be done async; there are several ways you could do that - queues and customer push/pull are the approaches (please, comment if more details are needed. but this is quite standard problem - lots of data in the internet)
The error handling is not clear. The errors are mixed up and you can't tell from which country the errors are coming from and how many errors there are
That's a code fix, when you do send data to cloudwatch (do you?). You could put country as a context to all request, via filter or something. And when error is logged that context is logged as well. You probably need venue id even more than country, as you can conclude country from venue id.
The responses from the third party API are not cached correctly in the cache layer since errors are stored for the time of the TTL
Don't store errors + add a circuit breaker pattern.
I work for an Student Information System and we're using the Admin SDK directory API to create school districts Google Org Unit structures from within our software.
POST https://www.googleapis.com/admin/directory/v1/customer/customerId/orgunits
When generating these API requests we're consistently receiving dailyLimitExceeded errors even when the district's quota has not been reached.
This error can be bypassed by ignoring the error, and implementing an exponential back-off routine, but I believe this to be acting much more like the quotaExceeded error is intended to act rather than dailyLimitExceeded, in that the request succeeds afterward on the first retry of this request.
In detail, the test I just ran successfully completed 9 of these API calls and then I received this response on the 10th:
Google.Apis.Requests.RequestError
Quota limit exceeded for the day. [403]
Errors [Message[Quota limit exceeded for the day.] Location[ - ] Reason[dailyLimitExceeded] Domain[usageLimits]
From the start of the batch of API calls it took about 10 seconds to get to the point where the error occurred.
Thanks for your help!
What I would suggest is to slow down your API requests. Don't make like 10 requests in 1 second. Give it a space in between requests. You are correct to implement exponential backoff. Also, if you can, use other accounts as well to make requests.
I'm looking for ways to overcome API quotas without resorting to requesting a raise, and would like to know what is classified as .. per user.
For example:
Google People API has this quota: Read Requests per 100 seconds per user
I setup an OAuth Client ID: 123-5ampl3.apps.googleusercontent.com
And for whatever reason, my queries are going to exceed "100 seconds per user".
My question/concerns:
Can I create another client ID 123-an0th3r.apps.googleusercontent.com and have both call the same API so that I now essentially have requests per 200 seconds per user?
Or is per user not tied to the client IDs, but instead, to the project id.
Could I create another project and re-route extra API calls to there?
Or must I throttle my querying so it stays within the limit.
Thanks!
I was wondering about the performance of WSO2 throttling. Whilst checking it out, I did the following :
I created a new subscription tier (say Tier1)
I set a limit of 10 requests per hour on it
I created another subscription tier (say Tier2)
I set a limit of 20 requests per hour on it
I allowed my API to use those tiers and created a test application to use the API with Tier1
Using the API console, I started sending requests to my API
After the 10 calls(though it exceeded 10), I unsubscribed the test application and subscribed to the same API using the same application using Tier2
It took around 15 mins for the throttler to unlock. Which brings me to my first concern ...
How long does it take for a change in tier subscription to reflect on the same application????
In relation to the excess calls going through, I also set up tiers with limits on a per minute basis. Once again, I noticed excess calls going through.
What I also noticed was that this excess was in the initial number of calls and the throttling worked normally after those initial excess calls.
Which leads to my next ...
How long does it take for a newly set up tier to throttle properly? In the per-minute basis, I assume that some calls carry over to the next minute. Am I right in assuming that?
Any help in understanding this would be much appreciated. Thanks
I have to create number of ads per one loop interation. For example I have pre-set 100 campaigns on my backend and then by one click I want to create ads. During one iteration I create one AdSet, one AdCreative, one AdImage and finally one Ad. So the maximum of ads I can create is 2 ads per 5 minute (then it throwing error that limit is over for Ad creation). It's terrible. I want to see why it's so limited, but cannot find any chance too see exceeded limit. I am using PHP Facebook Marketing API SDK ver. 2.6
Hope there are some developers how know about it.
Error code:
Array (
[code] => 613
[error_subcode] => 1487742
[message] => Calls to this api have exceeded the rate limit.
[error_user_title] => There have been too many calls from this ad-account. Wait a bit and try again.
[error_user_msg] => There have been too many calls from this ad-account. Wait a bit and try again.
[error_blame_field_specs] => [type] => OAuthException )
Marketing API has it is own rate limiting logic and is excluded from all the graph api rate limitations.
You can check Marketing API Rate Limiting Marketing API Rate Limiting
Also Limit of ads creation or API Rate Limiting depends on your Marketing API Access Levels Marketing API Access Levels
If you have higher access level (i.e Standard) then your API Rate Limiting will be higher