I'm trying to create a GKE cluster using:
gcloud container clusters create mycluster --region europe-west1
but I'm having an error:
ERROR: (gcloud.container.clusters.create) ResponseError: code=403, message=Insufficient regional quota to satisfy request: resource "IN_USE_ADDRESSES": request requires '9.0' and is short '1.0'. project has a quota of '8.0' with '8.0' available. View and manage quotas at https://console.cloud.google.com/iam-admin/quotas?usage=USED&project=myproject-xxxx.
It seems that there's a problem with the limit of the resource IN_USE_ADDRESSES. This is weird because the project is new therefore I don't have any resource except the cluster I want to crate. I have a second project in which I deployed a postgres DB. Nothing more.
When I visit the quota page, it seems that I didn't exceed any limit.
Your help will be appreciated.
I had this same problem, by default the number of nodes created by the gcloud container clusters create command is 3 and it seems there is 3 static address allocated for each node.
If you really don't need all those static addresses you can specify 2 nodes which should only allocate 6 static addresses via the --num-nodes switch eg.
gcloud container clusters create <cluster-name> --num-nodes=2
IN_USE_ADDRESSES means both static and ephemeral IP addresses. To create your cluster requires a total of 9 (which includes all of your services in the region) but your quota only allows for 8 addresses total.
Solution: Request a quota increase. Follow this link to understand how to request a quota link. A quota increase is requested in the Google Cloud Console. Tip: Increase both global and regional. This will cover load balancers in the future (and other services that use Global IP addresses).
Request a quota increase from the GCP support team. Or reduce the number of nodes you are creating, gcloud container clusters create --name --num-nodes (e.g 2) --zone (choose a zone)... alternatively you can set a zone to use, i.e, gcloud config set compute/zone (your chosen zone), this way you dont have to keep on updating the zone you want to create your infra in.
you also need to choose a zone for the cluster, use gcloud container clusters create [insert cluster name here] --num-nodes 3 --zone us-central1-f(or any zone you want)...by using the region, if you specify 2 nodes, in every zone 2nodes will be provided thats why you are exceeding the quota...
Related
When i try to create a node-pool using n2 and e2 machines in google kubernetes engine it shows this error
Creation of node-pool with N1 machine type
Create node pool "pool-1" in Kubernetes Engine cluster
11 minutes ago
Insufficient regional quota to satisfy request: resource "N2_CPUS": request requires '2.0' and is short '2.0'. project has a quota of '0.0' with '0.0' available. View and manage quotas at
Create node pool "pool-5" in Kubernetes Engine cluster
9 minutes ago
Deploy error: Not all instances running in IGM after 14.847846874s. Expect 1. Current errors: [ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS]: Instance 'gxxxxxxxpool-5-5365e95a-xw5r' creation failed: The zone 'projects/xxxx/zones/us-central1-a' does not have enough resources available to fulfill the request. '(resource type:compute)'.
Creation of node-pool with E2 machine type
but my quotas are not exceeded.It is possible to create node-pool with N1 type machines.Can anyone help to resolve this issue.
Stackoverflow cannot help you solve this problem.
The problem is that Google Cloud does not have enough resources available to allocate to you.
You have several options:
Wait. Resources will eventually become available. This could be minutes or days.
Select a different zone or region. Resource usage and availability will vary from zone to zone and region to region.
Select a different instance type. Availability varies between instance types.
The GKE documentation about resource quotas says that those hard limits are only applied for clusters with 10 or fewer nodes.
Even though we have more than 10 nodes, this quota has been created and cannot be deleted
Is this a bug on GKE side or intentional and the documentation is invalid?
I had experienced a really strange error today using GKE. Our hosted gitlab-runner stopped running new jobs, and the message was:
pods "xxxx" is forbidden: exceeded quota: gke-resource-quotas, requested: pods=1, used: pods=1500, limited: pods=1500
So the quota resource is non-editable (as documentation says). The problem, however, that there was just 5 pods running, not 1500. So it can be a kubernetes bug, the way it calculated nodes count, not sure.
After upgrading control plane and nodes, the error didn't go away and I didn't know how to reset the counter of nodes.
What did work for me was to simply delete this resource quota. Was surprised that it was even allowed to /shrug.
kubectl delete resourcequota gke-resource-quotas -n gitlab-runner
After that, same resource quota was recreated, and the pods were able to run again.
The "gke-resource-quotas" protects the control plane from being accidentally overloaded by the applications deployed in the cluster that creates excessive amount of kubernetes resources. GKE automatically installs an open source kubernetes ResourceQuota object called ‘gke-resource-quotas’ in each namespace of the cluster. You can get more information about the object by using this command [kubectl get resourcequota gke-resource-quotas -o yaml -n kube-system].
Currently, GKE resource quotas include four kubernetes resources, the number of pods, services, jobs, and ingresses. Their limits are calculated based on the cluster size and other factors. GKE resource quotas are immutable, no change can be made to them either through API or kubectl. The resource name “gke-resource-quotas” is reserved, if you create a ResourceQuota with the same name, it will be overwritten.
When creating a GKE cluster in GCP, we need to specify a zone. Can all users use any zone? like this:
gcloud container clusters create gke-cluster --zone=us-central1-a
Each GCP product availability is set on a per region basis. If it's available in a region, it's available in all zones within that region. Google documents each product availability by region here. As you can see, GKE is available in all regions so you can choose whichever you want when creating your cluster.
Having said that, one exception to this is if your GCP project is under an organization that has defined an organization policy that includes a resource locations constraint. In this case, you'll only be allowed to create your cluster in the regions/zones permitted by that policy.
In addition, according with your node requirement (GPU, TPU, CPU generation, number of vCPU available,...) it's possible that some zone aren't available because these aren't available.
It's not a GKE limitation, it's a compute engine limitation.
While updating EC2 instance of Elastic Beanstalk the following error message appeared. But for my other account the following error, not observed:
Updating Auto Scaling group named: awseb-e-tiknsbmh4d-stack-AWSEBAutoScalingGroup-R3UXFI8KMCSN failed Reason: You have requested more instances (1) than your current instance limit of 0 allows for the specified instance type
Is it account base specific? And what is the best way to fix it?
Yes, there are limits which are account-specific.
You can check your EC2 quotas using the AWS Quota Service. [1]
You can view the EC2 quotas which apply for the account you are currently signed in by visiting the following URL: https://eu-central-1.console.aws.amazon.com/servicequotas/home?region=REGION#!/services/ec2/quotas by replacing REGION with your region identifier, e.g. us-east-1.
If the quota is marked as adjustable, you can request an increase via this service.
References
[1] https://docs.aws.amazon.com/servicequotas/latest/userguide/intro.html
This message is related to Amazon EC2. Even though the instance was launched by Elastic Beanstalk, the limit is related to the number of concurrent Amazon E2 instances permitted.
There are two types of limits:
Total number of concurrent instances (default: 20)
Number of concurrent instances per instance type (default varies)
You can view these limits in the Limits section of the Amazon EC2 management console. You can also request an increase in the limits.
The limits are "per account, per region".
Autoscaling helps you to automatically add or remove compute engines based on the load. The prerequisites to autoscaling in GCP are instance template and managed instance group.
This question is a part of another question's answer, which is about building an autoscaled and load-balanced backend.
I have written the below answer that contains the steps to set up autoscaling in GCP.
Autoscaling is a feature of managed instance group in GCP. This helps to handle very high traffic by scaling up the instances and at the same time it also scales down the instances when there is no traffic, which saves a lot of money.
To set up autoscaling, we need the following:
Instance template
Managed Instance group
Autoscaling policy
Health Check
Instance template is a blueprint that defines the machine-type, image, disks of the homogeneous instances that will be running in the autoscaled, managed instance group. I have written the steps for setting up an instance template here.
Managed instance group helps in keeping a group of homogeneous instances that is based on a single instance template. Assuming the instance template as sample-template. This can be set up by running the following command in gcloud:
gcloud compute instance-groups managed \
create autoscale-managed-instance-group \
--base-instance-name autoscaled-instance \
--size 3 \
--template sample-template \
--region asia-northeast1
The above command creates a managed instance group containing 3 compute engines located in three different zones in asia-northeast1 region, based on the sample-template.
base-instance-name will be the base name for all the automatically created instances. In addition to the base name, every instance name will be appended by a uniquely generated random string.
size represents the desired number of instance in the group. As of now, 3 instances will be running all the time, irrespective of the amount of traffic generated by the application. Later, it can be autoscaled by applying a policy to this group.
region (multi-zone) or single-zone: Managed instance group can be either set up in a region (multi-zone) i.e the homogeneous instances will be evenly distributed across all the zones in a given region or all the instances can be deployed in the same zone within a region. It can also be deployed as cross region one, which is currently in alpha.
Autoscaling policy determines the autoscaler behaviour. The autoscaler aggregates data from the instances and compares it with the desired capacity as specified in the policy and determines the action to be taken. There are many auto-scaling policies like:
Average CPU Utilization
HTTP load balancing serving capacity (requests / second)
Stackdriver standard and custom metrics
and many more
Now, Introducing Autoscaling to this managed instance group by running the following command in gcloud:
gcloud compute instance-groups managed \
set-autoscaling \
autoscale-managed-instance-group \
--max-num-replicas 6 \
--min-num-replicas 2 \
--target-cpu-utilization 0.60 \
--cool-down-period 120 \
--region asia-northeast1
The above command sets up an autoscaler based on CPU utilization ranging from 2 (in case of no traffic) to 6 (in case of heavy traffic).
cool-down-period flag specifies the number of seconds to wait after a instance has been started before the associated autoscaler starts to collect information from it.
An autoscaler can be associated to an maximum of 5 different policies. In case of more than one policy, Autoscaler recommends the policy that leaves with the maximum number of instances.
Interesting fact: when an instance is spun up by the autoscaler, it makes sure that the instance runs for atleast 10 minutes irrespective of the traffic. This is done because GCP bills for a minimum of ten minute running time for the compute engine. It also protects against erratic spinning up and shutting down of instances.
Best Practices: From my perspective, it is better to create a custom image with all your software installed than to use a startup script. As the time taken to launch new instances in the autoscaling group should be as minimum as possible. This will increase the speed at which you scale your web app.
This is part 2 of 3-part series about building an autoscaled and load-balanced backend.