Around 90 or 100 calls per second to
pubsub_client.projects().topics().publish(topic='projects/xxxx',body=body).execute(num_retries=0)
per second from Google App Engine App to Google Cloud Pub/Sub, results in
HttpError: <HttpError 429 when requesting https://pubsub.googleapis.com/v1/projects/xxxx:publish?alt=json returned "Request throttled due to user QPS limit being reached.">
I know there is a limit on administrative operations at 100 QPS, but certainly publishing to a topic is not an administrative operation? I know pub/sub should support millions of operations per second so I know there's something wrong.
Any help or insight would be appreciated. I need to get up to at least 300 publishes per second, trying to streamline an existing implementation using pubsub. I think this may be a bug with the implementation.
I am running this code on Google App Engine python 2.7 -- using the appengine runtime, not the flexible one as that's not approved for production code yet.
Note that publisher quota is not in terms of QPS, but in terms of throughput. The default limit is 100MB/s. See the Quotas documentation for more details. Depending on the message size you are sending, you may be running into these limits.
The "user QPS limit being reached" message on a publish usually means one of three things:
You are publishing at a throughput that is higher than the default 100MB/s quota. If that is the case, then you can apply for more quota by clicking on the "Apply for higher quota" on the Pub/Sub Quota page.
You are not authenticated against the correct Cloud project. If you are authenticated in or running your Google App Engine instances in a Cloud project that differs from the one your topic is defined in, the quota you run into may not be defined in the project you expect. More information can be found in the Google Application Defaults Credentials page.
You have manually set quota in the Quota page and that is the limit you are running into.
Related
I have 400 quotas and if I add one more I'm getting an error
'Maximum number of Resources for this API has been reached.'
What is the maximum number? 500-800?
I want to know if I can extend it for another 200-300 quotas or I need to create another API, thank you!
As per the documentation, the default quota for Resources per API is 300. Reviewing the documentation further we can see that this limit can be increased which I would suspect has already occurred on your account.
If you would like to increase this further, you can use the console again and request a service increase, a useful guide for this is here.
As for the upper limit, this is not listed and most likely wont be listed as it will be at the AWS service teams discretion to do so. Based on my experience, you can usually get 100-150% more than the default quotas just by requesting a service increase in the console. If you would like more than this you may have to create a support case and give justification for the request, but, as long as it is reasonable, it will usually be accepted.
I developed and support a client's mobile app that uses Firebase services.
Google Cloud Platform logged this event yesterday at 4:17 am:
'<my account email> has executed
google.api.serviceusage.v1.ServiceUsage.EnableService
on stackdriver.googleapis.com'
I was sleeping at the time and a review of Google Admin Console Login Audit Log does not show a login event around that same time.
Immediately, 100% errors were reporting for 'compute':
A look at the Stackdriver API overview page does not give any indication of activity:
My question, my concern, how/why did this service get activated and what is the activity driving the compute errors at 100%?
During my efforts to understand, I clicked on Compute Engine API in the API library, which enabled the API (but no VMs, Disk, etc. were created):
A short time later, Google Cloud Platform has several log entries:
google.devtools.cloudbuild.v1.CloudBuild.ListBuilds
was executed on builds
Number of returned items 1000
The 'compute' errors stopped.
When I disabled the Compute Engine API, the ListBuilds logs stopped, but the Computer Errors returned to 100%.
I have not found a definitive answer to my question.
It's clear that Stackdriver API was enabled, but I don't know why.
When enabled, 100% Compute errors were being reported (orange line on graph) without any details.
While customizing the Google Cloud Platform Dashboard for this account, I toggled/enabled the Compute Engine card/graph hoping that might reveal some clues regarding the 100% errors. That action initialized the Compute Engine API. Almost immediately the Compute errors ended but there was a surge of activity that has continued. Reviewing many resources I found information that suggest this is normal behavior.
I would still like to fully understand how Stackdriver was enabled, why it was enabled, what value it provides, if I can simply disable it and Compute API as this project will never require VM compute services.
The problem
I'm using Google Sheets API v4 from node.js (Cloud Functions for Firebase) via googleapis npm package to write to the Google Sheets.
Getting this error sporadically:
Error: Resource has been exhausted (e.g. check quota).
at Gaxios._request (/workspace/node_modules/gaxios/build/src/gaxios.js:89:23)
at process._tickCallback (internal/process/next_tick.js:68:7)
It doesn’t say which particular quota is reached, though.
At that particular moment, indeed, there were many concurrent function executions that write to the Google Sheets, and several of them failed:
GCP console
https://console.cloud.google.com/iam-admin/quotas says that the quota statuses are fine:
https://console.cloud.google.com/apis/api/sheets.googleapis.com/quotas shows that the usage at the moment of issue is within the quota limits:
The guess
I'm suspecting that I may reach the “Write requests per 100 seconds per user” quota limit… because the “traffic by response code” chart shows 2.03 reqs/sec, which gives 203 requests per 100 seconds, and all of them from one credential:
However “Per-user quota usage is not displayed”, and neither error message or GCP console give definite indication on what is the actual quota being exhausted.
Also, there is a counter-argument to this guess: total “requests per 100 seconds” in the chart above is below 100, and I presume that “requests per 100 seconds per user” cannot be higher than total.
The questions
Is my conclusion based on the charts in the GCP console correct that I’m not reaching the quota limits?
How to figure out which quota limit is referred to in the error message?
What other cause can be there behind Resource has been exhausted (e.g. check quota) error?
Update based on the comment
Here are the links to the actual code repository.
The cloud function that is triggered: the line that calls my repository class method.
The repository method itself that calls the Spreadsheet class method to update the rows.
The Spreadsheet method which prepares data for updating/appending.
Actual method that calls the googleapis.
This is not a Minimal, Reproducible Example, this is the actual code used. I don't think providing a reproducible example is worth time in this case, because the question is primarily related to interpreting the error message and figuring out which particular quota limit has been reached.
Step to reproduce: Go to this site, https://cloud.google.com/talent-solution/job-search/docs/reference/rest/v4beta1/projects.tenants.profiles/create, make a POST request with a valid tenant name, to create a new profile.
Get error: "8 RESOURCE_EXHAUSTED: Quota exceeded for quota metric 'quota metric that handles profile write requests' and limit 'quota metric that handles profile write requests per minute' of service 'jobs.googleapis.com' for consumer 'project_number:XXXXXXXXX'."
My current usage is just 100+ post/put/delete requests per month, and if I remember correctly, free tie in Google Cloud allows thousands of requests per 100 seconds.
My billing account was setup and the card is valid. I can create companies, tenants as normal.
Please help.
As John Hanley commented quota issues are usually handled by the Cloud Support team and should be addressed opening a support ticket. However, this case refers to Cloud Talent Solution Profile Search which as stated in the documentation at this moment is in 'Closed Beta' state, meaning that it doesn't allow new candidates.
In order to use this feature you will have to wait until it's released to the public or the Beta is opened for new candidates.
Yesterday my free subscription validity got over, and I registered for the GCML services. However, I am not able to access my GCS or GCML resources and am getting Access Denied errors. On trying to access the Jobs page on the console I get the following error
Quota failure for project_number:836740678279 -- The allowed Cloud ML quota for API calls in the "Management (CRUD) requests" group is exceeded, please see https://console.cloud.google.com/apis/api/ml.googleapis.com/quotas for details.
However the limit on quota is 10,000 per day, and I had only used up 105 on the said day (viewed on the graphs as pointed in the error message)
You're right in that the default daily quota for CRUD requests (Create/Read/Update/Delete) is 10,000, however there is also a 100-second quota, which is set to 50 by default. Most likely you sent too many requests within 100 seconds and exceeded it.
Note that you can increase the 100-second quota limit to 500 for your project at the
quota management page, which was linked from the error message.
I was getting the same error. Turns out I needed to setup a billing account (https://console.cloud.google.com/billing) for the project.