I'm working on istio 1.4.10, I need local rate limit replace nginx rate limit.
Istio 1.4.10 explain just grobal rate limit via redis.
but recent doc explain local rate limit with envoy.
So Can i make custom local rate limit envoyFilter ?
Related
Background 1
In my account, the limited of Compute Engine API Backend services is increased to 75.
Background 2
I only have 9 Back-end service in Load balancing
Question
When I try to create a new Load Balancer, I receive below message:
Quota 'BACKEND_SERVICES' exceeded. Limit: 9.0 globally.
Suppose I should have enough quota for creating new backend service.....
Except removing other backend service, any suggestion for fixing this issue?
Thank you in advance!
Sometimes when a quota increase is approved, the deployment of that quota increase does not happen. I have experienced this several times.
My recommendation is to request a higher quota increase and explain the details about the previous quota increase being approved but not being deployed.
As John mentioned, it’s recommended to request a higher quota for backend services, also I share with you the backend services quotas that includes all backend services (INTERNAL, INTERNAL_MANAGED, INTERNAL_SELF_MANAGED, and EXTERNAL) in your project.
We use Openshift 4.x. For an API, min pods is 5 and max is 8. Horizontal autoscaling is configured based on the avg. CPU utilization percentage. The property haproxy.router.openshift.io/pod-concurrent-connections = '10' --> restricts the number of connections to each Pod to 10. What happens if we get more requests to the pod? Does it wait in the queue or does the pods scale up horizontally?
Below is current configuration in Routes for this API:
haproxy.router.openshift.io/disable_cookies: 'true' haproxy.router.openshift.io/balance: roundrobin haproxy.router.openshift.io/pod-concurrent-connections: '10' haproxy.router.openshift.io/timeout: 50s
The HPA will spawn a new pod if the defined CPU limit is reached, not based the number of connections
So to answer your question "What happens if we get more requests to the pod?", the answer is
yes if at that time the 10 connections per pod make the pods consume more CPU that the defined CPU limit in HPA,
otherwise no and so the requests will be queued in HAProxy
The HPA may also spawn another pod before that, if the CPU goes above the defined limit. The 2 metrics (# of connections and CPU consumption) are not related in the context of k8s/OCP
Keep in mind that the CPU limit defined in the HPA is the average CPU observed for all the running pods at the time
I have an issue with a Google Cloud Platform's Cloud Endpoint.
I have a small API backed by a Cloud Function requesting some data in a Cloud SQL instance. This part is very fast.
This API is exposed via Cloud Endpoints and an ESP proxy running (as in Google Cloud Platform documentation).
When launching, the latency is reasonable (around 200ms) but after sometimes (without any intervention) , latency is rising up around 2s. Then, if I force to redeploy the Cloud Run instance latency is returning to normal.
I have another endpoint with the exact same config but with a Cloud Function backed by another Cloud SQL instance and I don't have this problem.
Do you have an idea why?
Thanks!
Antoine
Edit :
A trace with low latency:
Another with high latency:
Both are the exact same infrastructure. A restart of the Cloud Run ESP Proxy permits to reduce the latency for a while (6 hours the last time, this time it has been 24 hours without high latencies).
Update :
Updating the ESP proxy to v2 (gcr.io/endpoints-release/endpoints-runtime-serverless:2) seems to fix the issue.
Are you referring to the CheckServiceControl latency?
ESP has local cache for ServiceControl call. Cache expired in 5 minutes. The low latency maybe from the cache hit and the high latency maybe from the cache miss.
I am running a pipeline using the Apache Beam model in Google Cloud Dataflow but I am unable to scale it up from 8 workers, even though the maximum number of workers is 32.
When I try to run the same pipeline setting the number of workers to 32, it gives me the following warnings:
Autoscaling: Startup of the worker pool in zone us-central1-f reached 30 workers, but the goal was 32 workers. The service will retry. QUOTA_EXCEEDED: Quota 'DISKS_TOTAL_GB' exceeded. Limit: 4096.0
Autoscaling: Unable to reach resize target in zone us-central1-f. QUOTA_EXCEEDED: Quota 'DISKS_TOTAL_GB' exceeded. Limit: 4096.0
But still doesn't pass 8 workers. Is there any particular reasons why a pipeline won't use more than 8 workers?
The problem was quota limits. Google Dataflow uses behind the scenes VMs of Google Compute Engine and their quotas apply. The specific limitation of 8 was being caused by the In use external IP adresses quota limitation. Others quotas were also violated when I tried to scale to 32, like the Disk space. So if anyone is having the same problem I suggest going to IAM Admin > Quotas on the console while the pipeline is running to check which quotas your pipeline may violate.
Also, the logs are different if you run using a deployed template or use the Eclipse plugin to run in debug mode. The later will give much more details than the first.
Visit https://console.cloud.google.com/iam-admin/quotas
Use the filter menu that says Quota type
You can tell by the color of the Current Usage column that API limit has reached limit.
Click Edit Quotas for the API that has exceeded usage and request for new limit. This will take from few hours to day.
Dataflow will use whatever number of workers you can get. In your case, it will reach 30 workers and will use them. It will however retry constantly to reach 32, as quota could be given back by other workflows.
I had installed istio on kubernets(hosted on aws ) and exposed a service as a via istio ingress. I could achieve very less through put < 2000 requests per minute. I exposed the service as a standard ELB, I was able to achieve >600,000 request per sec. Is there any guide/steps to tune istio for high performance ?
Thanks
Joji
One thing we recommend users to do is to switch to the non-debug image for the envoy side car, e.g. replace docker.io/istio/proxy_debug with docker.io/istio/proxy in the istio and istio-initializer yaml file and redeploy istio. We are also working on reduce mixer traces. Performance is an area we are very actively working on in the next release of Istio and we welcome contribution to this!