Connection bias in Cloud Run using http/2, GCLB
GCLB , Serverless NEG, CloudRun(http/2) is used to configure the backend.
This is a chart that records access counts per container by our own log-based metrics.
Request Count
Clearly, the containers appear to be grouped. And there appears to be a large bias in the number of requests between one group and another. (Not enough requests are being allocated to the container groups that have just been launched.)
This was not seen when http1 was used in the connection between GCLB and CloudRun.
During this time, application logs are normal(few 503), so there is no application layer limiting requests. (The metrics are based on the standard request log, so the only metric is whether or not the request was made.)
How should this be remedied?
CloudRun Setting is as below
metadata:
name: app
annotations:
run.googleapis.com/client-name: gcloud
run.googleapis.com/client-version: 411.0.0
autoscaling.knative.dev/minScale: '4'
run.googleapis.com/execution-environment: gen2
autoscaling.knative.dev/maxScale: '200'
run.googleapis.com/cpu-throttling: 'false'
run.googleapis.com/startup-cpu-boost: 'true'
spec:
containerConcurrency: 100
timeoutSeconds: 300
containers:
ports:
- name: h2c
containerPort: 8080
I have tried changing the number of containerConcurrency, but this does not improve things much. There is still an imbalance, as it appears that requests are allocated to containers that have just been launched or not allocated at all.
Related
Initially, I've deployed my frontend web application and all the backend APIS in AWS ECS, each of the backend APIs has a Route53 record, and the frontend is connected to these APIs in the .env file. Now, I would like to migrate from ECS to EKS and I am trying to deploy all these application in a Minikube local cluster. I would like to keep my .env in my frontend application unchanged(using the same URLs for all the environment variables), the application should first look for the backend API inside the local cluster through service discovery, if the backend API doesn't exist in the cluster, it should connect to the the external service, which is the API deployed in the ECS. In short, first local(Minikube cluster)then external(AWS). How to implement this in Kubernetes?
http:// backendapi.learning.com --> backend API deployed in the pod --> if not presented --> backend API deployed in the ECS
.env
BACKEND_API_URL = http://backendapi.learning.com
one of the example in the code in which the frontend is calling the backend API
export const ping = async _ => {
const res = await fetch(`${process.env.BACKEND_API_URL}/ping`);
const json = await res.json();
return json;
}
Assuming that your setup is:
Basing on microservices architecture.
Applications deployed in Kubernetes cluster (frontend and backend) are Dockerized
Applications are capable to be running on top of Kubernetes.
etc.
You can configure your Kubernetes cluster (minikube instance) to relay your request to different locations by using Services.
Service
In Kubernetes terminology "Service" is an abstract way to expose an application running on a set of Pods as a network service.
Some of the types of Services are following:
ClusterIP: Exposes the Service on a cluster-internal IP. Choosing this value makes the Service only reachable from within the cluster. This is the default ServiceType.
NodePort: Exposes the Service on each Node's IP at a static port (the NodePort). A ClusterIP Service, to which the NodePort Service routes, is automatically created. You'll be able to contact the NodePort Service, from outside the cluster, by requesting <NodeIP>:<NodePort>.
LoadBalancer: Exposes the Service externally using a cloud provider's load balancer. NodePort and ClusterIP Services, to which the external load balancer routes, are automatically created.
ExternalName: Maps the Service to the contents of the externalName field (e.g. foo.bar.example.com), by returning a CNAME record with its value. No proxying of any kind is set up.
https://kubernetes.io/docs/concepts/services-networking/service/#publishing-services-service-types
You can use Headless Service with selectors and dnsConfig (in Deployment manifest) to achieve the setup referenced in your question.
Let me explain more:
Example
Let's assume that you have a backend:
nginx-one - located inside and outside
Your frontend manifest in most basic form should look following:
deployment.yaml:
apiVersion: apps/v1
kind: Deployment
metadata:
name: frontend
spec:
selector:
matchLabels:
app: frontend
replicas: 1
template:
metadata:
labels:
app: frontend
spec:
containers:
- name: ubuntu
image: ubuntu
command:
- sleep
- "infinity"
dnsConfig: # <--- IMPORTANT
searches:
- DOMAIN.NAME
Taking specific look on:
dnsConfig: # <--- IMPORTANT
searches:
- DOMAIN.NAME
Dissecting above part:
dnsConfig - the dnsConfig field is optional and it can work with any dnsPolicy settings. However, when a Pod's dnsPolicy is set to "None", the dnsConfig field has to be specified.
searches: a list of DNS search domains for hostname lookup in the Pod. This property is optional. When specified, the provided list will be merged into the base search domain names generated from the chosen DNS policy. Duplicate domain names are removed. Kubernetes allows for at most 6 search domains.
As for the Services for your backends.
service.yaml:
apiVersion: v1
kind: Service
metadata:
name: nginx-one
spec:
clusterIP: None # <-- IMPORTANT
selector:
app: nginx-one
ports:
- name: http
protocol: TCP
port: 80
targetPort: 80
Above Service will tell your frontend that one of your backends (nginx) is available through a Headless service (why it's Headless will come in hand later!). By default you could communicate with it by:
service-name (nginx-one)
service-name.namespace.svc.cluster.local (nginx-one.default.svc.cluster.local) - only locally
Connecting to your backend
Assuming that you are sending the request using curl (for simplicity) from frontend to backend you will have a specific order when it comes to the DNS resolution:
check the DNS record inside the cluster
check the DNS record specified in dnsConfig
The specifics of connecting to your backend will be following:
If the Pod with your backend is available in the cluster, the DNS resolution will point to the Pod's IP (not ClusterIP)
If the Pod backend is not available in the cluster due to various reasons, the DNS resolution will first check the internal records and then opt to use DOMAIN.NAME in the dnsConfig (outside of minikube).
If there is no Service associated with specific backend (nginx-one), the DNS resolution will use the DOMAIN.NAME in the dnsConfig searching for it outside of the cluster.
A side note!
The Headless Service with selector comes into play here as its intention is to point directly to the Pod's IP and not the ClusterIP (which exists as long as Service exists). If you used a "normal" Service you would always try to communicate with the ClusterIP even if there is no Pods available matching the selector. By using a headless one, if there is no Pod, the DNS resolution would look further down the line (external sources).
Additional resources:
Minikube.sigs.k8s.io: Docs: Start
Aws.amazon.com: Blogs: Compute: Enabling dns resolution for amazon eks cluster endpoints
EDIT:
You could also take a look on alternative options:
Alernative option 1:
Use rewrite rule plugin in CoreDNS to rewrite DNS queries for backendapi.learning.com to backendapi.default.svc.cluster.local
Alernative option 2:
Add hostAliases to the Frontend Pod
You can also use Configmaps to re-use .env files.
I just set up a NodeJS based site on Google Cloud using the Cloud Run service.
There are two DNS records: A (IPv4) and AAAA (IPv6). Whenever I access the site using Chrome, my Chrome picks the IPv6 address and NodeJS app fails hard:
TypeError [ERR_INVALID_URL]: Invalid URL: http://2001:14ba:98ae:1700:****:****:****:****/
at onParseError (internal/url.js:257:9)
at new URL (internal/url.js:333:5)
Note: I censored the address
If I force my browser to use the IPv4 address, then the site works fine.
Is there a way to make the Cloud Run service use IPv4 to the container/app? I don't mind IPv6 at the client <-> Cloud Run level.
My Cloud Run YAML looks like:
apiVersion: serving.knative.dev/v1
kind: Service
metadata:
name: xxx-dev-app-825af7f
namespace: 'xxx'
selfLink: /apis/serving.knative.dev/v1/namespaces/xxx/services/xxx-dev-app-825af7f
uid: 2d787ef2-39a7-xxx-yyy-zzz
resourceVersion: AAWfuzEBUYA
generation: 5
creationTimestamp: '2020-02-26T18:58:40.504717Z'
labels:
cloud.googleapis.com/location: europe-north1
annotations:
run.googleapis.com/client-name: gcloud
serving.knative.dev/creator: pulumi#xxx.iam.gserviceaccount.com
serving.knative.dev/lastModifier: xxx#cloudbuild.gserviceaccount.com
client.knative.dev/user-image: gcr.io/xxx/app:4860b1e137457b0e42a1896d7b95e0348d8cd7e4
run.googleapis.com/client-version: 279.0.0
spec:
traffic:
- percent: 100
latestRevision: true
template:
metadata:
name: xxx-dev-app-825af7f-00005-xoz
annotations:
run.googleapis.com/client-name: gcloud
client.knative.dev/user-image: gcr.io/xxx/app:4860b1e137457b0e42a1896d7b95e0348d8cd7e4
run.googleapis.com/client-version: 279.0.0
autoscaling.knative.dev/maxScale: '1000'
spec:
timeoutSeconds: 900
containerConcurrency: 80
containers:
- image: gcr.io/xxx/app:4860b1e137457b0e42a1896d7b95e0348d8cd7e4
ports:
- containerPort: 8080
resources:
limits:
cpu: 1000m
memory: 256Mi
requests:
cpu: 200m
memory: 64Mi
status:
conditions:
- type: Ready
status: 'True'
lastTransitionTime: '2020-02-29T18:33:33.424Z'
- type: ConfigurationsReady
status: 'True'
lastTransitionTime: '2020-02-29T18:33:28.264Z'
- type: RoutesReady
status: 'True'
lastTransitionTime: '2020-02-29T18:33:33.424Z'
observedGeneration: 5
traffic:
- revisionName: xxx-dev-app-825af7f-00005-xoz
percent: 100
latestRevision: true
latestReadyRevisionName: xxx-dev-app-825af7f-00005-xoz
latestCreatedRevisionName: xxx-dev-app-825af7f-00005-xoz
address:
url: https://xxx.run.app
url: https://xxx.run.app
AFAIK, IPv6 is only supported at Global Load balancer only. This load balancer proxied the connection and convert it to IPv4 for internal access into Google Network.Thereby, direct access to Cloud Run with IPv6 seems impossible.
However, things are in progress, especially around Load Balancing and it could solve your issue. Maybe announcements at Cloud Next in April. Stay tuned!
For the connections between Cloud Run <=> user browser: You currently cannot disable the IPv6 stack.
(As Guillaume said, upcoming support for configurable Cloud HTTPS Load Balancer would solve your problem –in fact, IPv4 is the default for GCLB, and you explicitly need to configure an IPv6 address if you want IPv6 for your GCLB).
For connections between Cloud Run Service <=> Cloud Run Service: You should be fully control what IP you connect to, on the client side.
For example, on the client side,
Force Python HTTP client to use IPv4
Force Go HTTP client to use IPv4
You can force programs to use IPv4 using their options e.g. curl --ipv4.
I'm deploying a project that is a GKE container backend, with a Cloud Endpoints API on the the front end. It goes through an Ingress and NodePort service. I've declared a readiness probe on the ESP container:
readinessProbe:
httpGet:
path: /ping
port: 8080
But I observed that the /ping path must be declared in the openapi.yaml without any security, or else it returns 404, or 401 if declared with some security definition.
Is that expected? I don't see anything about that in the Endpoints Samples repo:
https://github.com/GoogleCloudPlatform/endpoints-samples/tree/master/k8s
After more closely examining the documentation and code samples, I discovered that the ESP has a healthz argument for that:
spec:
containers:
- name: esp
image: gcr.io/endpoints-release/endpoints-runtime:1
args: [
"-p", "8080",
"-a", "127.0.0.1:8081",
"-s", "SERVICE_NAME",
"--rollout_strategy", "managed",
"-z", "healthz",
]
readinessProbe:
httpGet:
path: /healthz
port: 8080
ports:
- containerPort: 8080
(https://github.com/GoogleCloudPlatform/endpoints-samples/blob/master/k8s/esp_echo_gke_ingress.yaml)
When specifying the healthz argument for the ESP, it configures nginx to just return 200 for that path:
% if healthz:
location = /${healthz} {
return 200;
access_log off;
}
% endif
(https://github.com/cloudendpoints/esp/blob/5c9f586b14db58607d1780966cceac923809c150/start_esp/nginx-auto.conf.template#L123-L128)
So it looks like Google wants you to just do a health check on the ESP container, not your backend.
This answers my original question, although it still does lead to a second question, which is: Why not health check the backend?
I'm assuming the term "backend" is referred to the nodes of the cluster or the Pods running by GKE. For both cases, GKE is self managed and the healthy of the nodes and the pods is monitored by the Master Node so that they should be always available. More information on this regard can be found in Kubernetes Scheduler and GKE Cluster Architecture.
In this context, the healthchecks used for the GKE ingress doesn't check the backend but the services running in the cluster. To check that these services are OK, readiness and liveness probes are used as explained in this article.
If you want to health check your backend, you need to list the health check path in openapi config for ESP to forward it.
I was having the same problem, but it seems possible to have a healthcheck on the GCE sidecar AND the eventual backend (e.g. the api you have actually built).
The trick is, the healthcheck for your api has to be configured as a readinessprobe on the cloud endpoints container (not your api container). The endpoint that you check also has to have no security against it, so it is publicly accessible (although it seems like you could also configure the health check with an api key).
I followed previous suggestions and also added the "healthz" argument to the esp container. My API is running on port 80 with a unprotected status endpoint at /api/status
spec:
containers:
- name: esp
image: gcr.io/endpoints-release/endpoints-runtime:1
args: [
"--http_port=8081",
"--backend=127.0.0.1:80",
"--service=MY SERVICE",
"--rollout_strategy=managed",
"-z", "healthz",
]
ports:
- containerPort: 8081
readinessProbe:
httpGet:
path: /api/status/
port: 8081
- name: api
image: MY IMAGE
ports:
- containerPort: 80
This worked, so that check is now verifying both containers are running with one readinessProbe. Interestingly, when I checked the LoadBalancer, it had also configured one against /healthz, when it hadn't been before. That would nbe useful in diagnosing cases where the endpoints container was working, but that api is not
I already set up Google Cloud Endpoints project and can invoke http/https requests. Endpoints gives me MY_API.endpoints.MY_PROJECT.cloud.goog domain name that I can use. I'm using gRPC Cloud Endpoints with HTTP/JSON to gRPC transcoding feature.
It is deployed on Google Kubernetes Engine (deployment yaml script attached at the end).
When I'm trying to create push subscription with that URL I getting next error:
"The supplied HTTP URL is not registered in the subscription's parent
project (url="https://MY_API.endpoints.MY_PROJECT.cloud.goog/v1/path", project_id="PROJECT_ID").
My gcloud call:
gcloud pubsub subscriptions create SUB_NAME --topic=projects/MY_PROJECT/topics/MY_TOPIC --push-endpoint="https://MY_API.endpoints.MY_PROJECT.cloud.goog/v1/path"
I tried to create Cloud DNS public zone with that DNS name and set corresponding records. But I still can't verify ownership in Google Search Console.
The question is how can I set DNS TXT record for MY_API.endpoints.MY_PROJECT.cloud.goog domain to verify ownership? Or how to use Pubsub push subscription with Cloud Endpoints gRPC in other way?
I could verify ownership of domain if I have ability to change meta or headers of gRPC responses converted to HTTP. But I doubt if there is a way.
Kubernetes script I used for deployment (if it would be helpful).
apiVersion: v1
kind: Service
metadata:
name: GKE_SERVICE_NAME
spec:
ports:
# Port that accepts gRPC and JSON/HTTP2 requests over HTTP.
- port: 80
targetPort: 9000
protocol: TCP
name: http2
selector:
app: GKE_SERVICE_NAME
type: LoadBalancer
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: GKE_SERVICE_NAME
spec:
replicas: 1
template:
metadata:
labels:
app: GKE_SERVICE_NAME
spec:
containers:
- name: esp
image: gcr.io/endpoints-release/endpoints-runtime:1
args: [
"--http2_port=9000",
"--service=MY_API.endpoints.MY_PROJECT.cloud.goog",
"--rollout_strategy=managed",
"--backend=grpc://127.0.0.1:50051"
]
ports:
- containerPort: 9000
- name: MY_CONTAINER_NAME
image: gcr.io/MY_PROJECT/IMAGE_NAME:v1
ports:
- containerPort: 50051
Ultimately, your goal is to get Cloud Pub/Sub pushing to your container on GKE. There are a couple ways to do this
Domain ownership validation, as you've discovered:
You can try to do it with DNS, and there's a guide for configuring DNS for a cloud.goog domain.
You can try to do it with one of the non-DNS alternatives, which includes methods such as hosting certain kinds of HTML or Javascript snippets from the domain. This can be tricky, though, as I don't know how to make Cloud Endpoints serve static HTML or Javascript content. It serves responses in OpenAPI format, which is essentially JSON.
Have you tried putting the Cloud Pub/Sub subscription and the cloud.goog domain in the same project? It might already be considered a verified domain in that case.
Since you are already using Google Kubernetes Engine, use either Cloud Run, or Cloud Run on top of Google Kubernetes Engine. There is a difference between Cloud Run and Cloud Run on GKE, but both will run your Kubernetes containers. Push endpoints on Cloud Run don't require domain ownership validation (I'm not sure if this also covers Cloud Run on GKE). You may get other interesting benefits as well, as Cloud Run is essentially designed to address the very use case of serving a push endpoint from a container. For example, it will do autoscaling and monitoring for you.
After deploying istio, we can see two deployment based on Mixer: istio-policy and istio-telemetry. They have the same pod spec on container mixer:
containers:
- args:
- --address
- unix:///sock/mixer.socket
- --configStoreURL=k8s://
- --configDefaultNamespace=istio-system
- --trace_zipkin_url=http://zipkin:9411/api/v1/spans
image: anjia0532/istio-release.mixer:master-latest-daily
imagePullPolicy: IfNotPresent
livenessProbe:
failureThreshold: 3
httpGet:
path: /version
port: 9093
scheme: HTTP
initialDelaySeconds: 5
periodSeconds: 5
successThreshold: 1
timeoutSeconds: 1
name: mixer
ports:
- containerPort: 9093
protocol: TCP
- containerPort: 42422
protocol: TCP
resources:
requests:
cpu: 10m
I have two questions:
How can the mixer container know they should be responsible for telemetry collection or policies control? I guess they are differentiate by the different arguments of istio-proxy container?
Why not use the same deployment for both functions?
I will share my understanding on this feature.
We can see that this separation was done in version 0.6 as written in release notes.
Separate Check & Report Clusters. When configuring Envoy, it’s now
possible to use different clusters for Mixer instances that are used
for Mixer’s Check functionality from those used for Mixer’s Report
functionality. This may be useful in large deployments for better
scaling of Mixer instances.
Why is it important? The policy checks are performed before each request so this is where the latency may come from, thus we may have a need to scale the mixer-policy component.
Whereas the telemetry report happens after the request, moreover the sidecar buffers the outgoing telemetry so the call to the mixer-telemetry are not so often.
Also the latency on telemetry data is not so critical as it won't affect UX of you application.
So in short, the main reason for this is scalability issues, specially for multicluster environment.