I have configured the both local and global ratelimiting . I am able to get the global ratelimiting metrics using statsd, how can I get the local ratelimiting metrics ?
Istio local ratelimit exposes metrics mentioned in :
https://www.envoyproxy.io/docs/envoy/latest/configuration/http/http_filters/local_rate_limit_filter#statistics
Need to configure the istio ingress gateway's default mesh config to emit these metrics (By default lot of metrics are omitted to conserve resources )
Within the configmap add following entries as per your metric patterns
defaultConfig:
discoveryAddress: istiod.default.svc:15012
proxyStatsMatcher:
inclusionRegexps:
- .*http_local_rate_limit.*
inclusionPrefixes:
- upstream_rq_retry
- upstream_cx
You can check if stats are being emitted by querying :
kubectl exec -it <istio-ingress-pod> -c istio-proxy -- pilot-agent request GET stats
Related
This is the first time I have been trying to install managed Anthos into one of the clusters in GKE. I admit I do not fully understand the full process of installation and troubleshooting I have already done.
It looks like a managed service has failed to install.
When I run:
kubectl describe controlplanerevision asm-managed -n istio-system
I get this status:
Status:
Conditions:
Last Transition Time: 2022-03-15T14:16:21Z
Message: The provisioning process has not completed successfully
Reason: NotProvisioned
Status: False
Type: Reconciled
Last Transition Time: 2022-03-15T14:16:21Z
Message: Provisioning has finished
Reason: ProvisioningFinished
Status: True
Type: ProvisioningFinished
Last Transition Time: 2022-03-15T14:16:21Z
Message: Workload identity is not enabled at node level
Reason: PreconditionFailed
Status: True
Type: Stalled
Events: <none>
However, I have Workload identity enabled on a cluster level and I cannot see any options in GCP Console to set that for just a node level.
I am not sure if this is related to istiod-asm-1125-0 logging some errors. One of them is about failure to load CA secret:
Nevertheless, the service mesh does not show as added or connected in Anthos Dashboard. The cluster is registered with Anthos.
I created a new node pool with more CPU and more nodes as I was getting warning about not having enough CPU. Istio service mesh increases the need for CPU.
I migrated my deployment from old node pool to the new one.
I run istioctl analyze -A and found a few warnings about istio-injection not being enabled in a few namespaces. I fixed that.
I re run asmcli install command without CA
./asmcli install --project_id my-app --cluster_name my-cluster --cluster_location europe-west1-b --fleet_id my-app --output_dir anthos-service-mesh --enable_all
All or some of the above did the trick.
I'm looking for a way to automate config updates for Google API Gateway, i.e. change config for an existing instance of "API Gateway" in a single step.
What I've tried so far, assuming that new API config name is "my-new-config" and API Gateway name is "my-gateway":
> gcloud beta api-gateway gateways update my-gateway --api-config=my-new-config --location=us-central1
Output:
ERROR: (gcloud.beta.api-gateway.gateways.update) INVALID_ARGUMENT: update_mask does not contain any field paths
> gcloud beta api-gateway gateways update my-gateway --api-config=my-new-config --location=us-central1 --display-name random-string-for-display-name
Output:
Command executes successfully, but config change is not applied.
gcloud version: 333.0.0
OS: Debian linux
I've created 2 tickets in Google's issue tracker (one, two), but there's no activity for them after 3 weeks.
Try with aplha instead of beta and specifying de API ID flag (--api):
gcloud alpha api-gateway gateways update my-gateway --api=api-id --api-config=my-new-config --location=us-central1
You're missing the --api flag in step 2, which seems to be required. It looks like without that specified, it doesn't make the right request.
You try to update api-config of api-gateway, here api is a required flag:
From the docs, when first is specified, second is mandatory:
[--api-config=API_CONFIG : --api=API]
api-config: This flag must be specified if any of the other arguments in this group are specified.
After i've added --api, it was possible to update the gateway with the new api-config
I ran this command:
gcloud --project=xxx beta container clusters create xxx --network xxx --subnetwork xxx --cluster-secondary-range-name=xxx
Turns out I had a typo. My secondary range is actually zzz not xxx. So, I have to wait 30 minutes for my cluster creation to fail and finally see what the actual error is:
Retry budget exhausted (80 attempts): Secondary range "xxx" does not
exist in network "xxx", subnetwork "xxx".
That's bad. Until the 30 or so minutes elapses I get no error messages or logs and I can't even delete the cluster until it "finishes" and fails.
There has to be a better way! Can I get this to fail fast or at least get some kind of verbose output?
Answering the question from the title:
Can I set the “Retry budget” for gcloud beta container clusters create command?
Currently there is no possibility to change the Retry budget and it also means that $ gcloud container clusters command isn't supporting that change either.
You can follow a Feature Request for it, that was created on Issuetracker:
Issuetracker.google.com: Allow configuration of GKE "maximum retry budget" when using custom ranges
Additional resources:
Stackoverflow.com: Questions: Terraform Google container cluster adjust maximum retry budget
I am trying to create a cluster based on a single node with a machine type of n2-highcpu-2 with locationeurope-west3-a. While I'm creating the node I am getting this error message:
Insufficient regional quota to satisfy request: resource "N2_CPUS":
request requires '2.0' and is short '2.0'. the project has a quota of
'0.0' with '0.0' available. View and manage quotas at
https://console.cloud.google.com/iam-admin/quotas?usage=USED&project=gordion-project.
I am requesting an increase of quota via The quota page (See Appendix-1 below).
Later I am always getting the following mail from google and I can't increase the quota:
Hello,
We attempted to adjust the quota for your project: gordion-project but
did not find any changes that needed to be made. This could be due to
being unable to fulfill part of the request. If you would like to
reduce your quota, please reply to this message and a support
representative will get back to you. If the current values are still
insufficient, please file a new request with a reduced ask or
additional justification. To verify current quota, please navigate to
https://console.cloud.google.com/iam-admin/quotas?project=xx-xx
.
Best regards and happy computing!
Sincerely, Cloud Platform Support
What Else Can I do to create the cluster with n2 machine type?
Appendix-1
EDIT 1:
EDIT 2:
I ran the command in the CLI: gcloud container clusters create "test-cluster-1" --zone "europe-west1-b" --machine-type "n2-highcpu-2"
Got the following output:
WARNING: In June 2019, node auto-upgrade will be enabled by default for newly created clusters and node pools. To disable it, use the `--no-enable-autoupgrade` flag.
WARNING: Starting in 1.12, new clusters will have basic authentication disabled by default. Basic authentication can be enabled (or disabled) manually using the `--[no-]enable-basic-auth` flag.
WARNING: Starting in 1.12, new clusters will not have a client certificate issued. You can manually enable (or disable) the issuance of the client certificate using the `--[no-]issue-client-certificate` flag.
WARNING: Starting in 1.12, default node pools in new clusters will have their legacy Compute Engine instance metadata endpoints disabled by default. To create a cluster with legacy instance metadata endpoints disabled in the default node pool, run `clusters create` with the flag `--metadata disable-legacy-endpoints=true`.
WARNING: The Pod address range limits the maximum size of the cluster. Please refer to https://cloud.google.com/kubernetes-engine/docs/how-to/flexible-pod-cidr to learn how to optimize IP address allocation.
This will enable the autorepair feature for nodes. Please see https://cloud.google.com/kubernetes-engine/docs/node-auto-repair for more information on node autorepairs.
ERROR: (gcloud.container.clusters.create) ResponseError: code=403, message=Insufficient regional quota to satisfy request: resource "N2_CPUS": request requires '6.0' and is short '6.0'. project has a quota of '0.0' with '0.0' available. View and manage quotas at https://console.cloud.google.com/iam-admin/quotas?usage=USED&project=gordion-project.
This error means that you are encountered with the temporary resource stock-out issue at that particular zone europe-west3-a. I've tried to follow your steps on my project and I've got the same error.
The recommended workaround is to try a different zone or check that particular zone later. Have a look at the documentation Regions and Zones and try other zone.
I've quickly checked some zones like europe-west1-b, europe-west1-c, europe-west2-a, europe-west2-c, europe-west4-a, europe-west4-b, europe-west4-c and found no issues:
$ gcloud container clusters create "test-cluster-1" --zone "europe-west4-a" --machine-type "n2-highcpu-2"
WARNING: Currently VPC-native is not the default mode during cluster creation. In the future, this will become the default mode and can be disabled using `--no-enable-ip-alias` flag. Use `--[no-]enable-ip-alias` flag to suppress this warning.
WARNING: Newly created clusters and node-pools will have node auto-upgrade enabled by default. This can be disabled using the `--no-enable-autoupgrade` flag.
WARNING: Starting with version 1.18, clusters will have shielded GKE nodes by default.
WARNING: Your Pod address range (`--cluster-ipv4-cidr`) can accommodate at most 1008 node(s).
This will enable the autorepair feature for nodes. Please see https://cloud.google.com/kubernetes-engine/docs/node-auto-repair for more information on node autorepairs.
Creating cluster test-cluster-1 in europe-west4-a... Cluster is being health-checked (master is healthy)...done.
Created [https://container.googleapis.com/v1/projects/test-prj/zones/europe-west4-a/clusters/test-cluster-1].
To inspect the contents of your cluster, go to: https://console.cloud.google.com/kubernetes/workload_/gcloud/europe-west4-a/test-cluster-1?project=test-prj
kubeconfig entry generated for test-cluster-1.
NAME LOCATION MASTER_VERSION MASTER_IP MACHINE_TYPE NODE_VERSION NUM_NODES STATUS
test-cluster-1 europe-west4-a 1.14.10-gke.17 34.90.XX.XX n2-highcpu-2 1.14.10-gke.17 3 RUNNING
I was able to run my tests with my current quota.
UPDATE Test for zone europe-west1-b:
$ gcloud container clusters create "test-cluster-1" --zone "europe-west1-b" --machine-type "n2-highcpu-2"
WARNING: Currently VPC-native is not the default mode during cluster creation. In the future, this will become the default mode and can be disabled using `--no-enable-ip-alias` flag. Use `--[no-]enable-ip-alias` flag to suppress this warning.
WARNING: Newly created clusters and node-pools will have node auto-upgrade enabled by default. This can be disabled using the `--no-enable-autoupgrade` flag.
WARNING: Starting with version 1.18, clusters will have shielded GKE nodes by default.
WARNING: Your Pod address range (`--cluster-ipv4-cidr`) can accommodate at most 1008 node(s).
This will enable the autorepair feature for nodes. Please see https://cloud.google.com/kubernetes-engine/docs/node-auto-repair for more information on node autorepairs.
Creating cluster test-cluster-1 in europe-west1-b... Cluster is being health-checked (master is healthy)...done.
Created [https://container.googleapis.com/v1/projects/test-prj/zones/europe-west1-b/clusters/test-cluster-1].
To inspect the contents of your cluster, go to: https://console.cloud.google.com/kubernetes/workload_/gcloud/europe-west1-b/test-cluster-1?project=test-prj
kubeconfig entry generated for test-cluster-1.
NAME LOCATION MASTER_VERSION MASTER_IP MACHINE_TYPE NODE_VERSION NUM_NODES STATUS
test-cluster-1 europe-west1-b 1.14.10-gke.17 34.76.XX.XX n2-highcpu-2 1.14.10-gke.17 3 RUNNING
UPDATE2 Have a look at the error message:
ERROR: (gcloud.container.clusters.create) ResponseError: code=403, message=Insufficient regional quota to satisfy request: resource "N2_CPUS": request requires '6.0' and is short '6.0'. project has a quota of '0.0' with '0.0' available. View and manage quotas at https://console.cloud.google.com/iam-admin/quotas?usage=USED&project=gordion-project.
The main reason of your issue is Insufficient regional quota to satisfy request: resource "N2_CPUS". Try to request an increase in quota again and if you'll get the same reply - contact Google Cloud Support.
I have deleted the config file I used when experimenting with Kubernetes on my AWS (using this tutorial) and replaced it with another devs config file when they set up Kubernetes on a shared AWS (using this). When I run kubectl config view I see the following above the users section:
- cluster:
certificate-authority-data: REDACTED
server: <removed>
name: aws_kubernetes
contexts:
- context:
cluster: aws_kubernetes
user: aws_kubernetes
name: aws_kubernetes
current-context: aws_kubernetes
This leads me to believe that my config should be pointing to use our shared AWS but whenever I run cluster/kube-up.sh it creates a new GCE cluster so I'm thinking I'm using the wrong command to spin up the cluster on AWS.
Am I using the wrong command/missing a flag/etc? Additionally I'm thinking kube-up creates a new cluster instead of recreating a previously instantiated one.
If you are sharing the cluster, you shouldn't need to run kube-up.sh (that script only needs to be run once to initially create a cluster). Once a cluster exists, you can use the standard kubectl commands to interact with it. Try starting with kubectl get nodes to verify that your configuration file has valid credentials and you see the expected AWS nodes printed in the output.