We deployed our site in front GCLB.
LB -> Cloud run -> APP ENGINE API
Cloud run is hosting a react site and App Engine golang API.
After 12 hours we started to saw decline in the amount of clicks via google analytics but traffic was pretty much the same.
Our assumption is that "lost" traffic somehow, I can see in logs 2 main issue.
404 with address of old site components.
client disconnected before any response error.
I can understand the 404 error its cache request that looking for old site components.
But i don`t understand client disconnected error and if its related to our "lost" traffic.
Any suggestion how to analyze our "lost" traffic?
UPDATE:
I found some correlation to the client client disconnected error.
The requestUrl contains images resources for exemple
images/zoom.png?v1.0
Back end service name is empty backend_service_name: ""
not sure how it can be empty, I mapped all the resources and host
LOG
{
"insertId": "cs2fmdg2eo8nba",
"jsonPayload": {
"cacheId": "FRA-1209ea83",
"#type": "type.googleapis.com/google.cloud.loadbalancing.type.LoadBalancerLogEntry",
"statusDetails": "client_disconnected_before_any_response"
},
"httpRequest": {
"requestMethod": "GET",
"requestUrl": "https://travelpricedrops.com/images/aero.png?v1.0",
"requestSize": "78",
"userAgent": "Mozilla/5.0 (iPhone; CPU iPhone OS 14_8 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.1.2 Mobile/15E148 Safari/604.1",
"remoteIp": "109.104.52.1",
"referer": "https://travelpricedrops.com/passthru?tab=front&vert=flights&origin-iata=LEJ&destination-iata=JFK&departure-time=2021-12-26T11%3A00%3A00Z&cabin-class=economy&num-adults=1&num-youth=0&rental-duration=6&dta=48&return-time=2022-01-01T11%3A00%3A00Z&f=cf&fuid=1102&b=k&buid=1043",
"cacheLookup": true,
"latency": "0.071958s"
},
"resource": {
"type": "http_load_balancer",
"labels": {
"zone": "global",
"backend_service_name": "",
"forwarding_rule_name": "tpd-int-https-ipv4",
"target_proxy_name": "int-tpd-target-proxy-2",
"url_map_name": "int-tpd",
"project_id": "tpdrops"
}
},
"timestamp": "2021-11-09T06:13:55.121455Z",
"severity": "INFO",
"logName": "projects/tpdrops/logs/requests",
"trace": "projects/tpdrops/traces/13821ba38ae9e3191381f3f64b0a7b1a",
"receiveTimestamp": "2021-11-09T06:13:55.343086132Z",
"spanId": "a5ae86336a24bc32"
}
Config
**gcloud compute forwarding-rules describe tpd-int-https-ipv4**
IPAddress: 34.149.93.11
IPProtocol: TCP
creationTimestamp: '2021-08-30T11:49:06.047-07:00'
description: ''
fingerprint: CIAg3TcEb9Y=
id: '1815919129513727693'
kind: compute#forwardingRule
labelFingerprint: 42WmSpB8rSM=
loadBalancingScheme: EXTERNAL
name: tpd-int-https-ipv4
networkTier: PREMIUM
portRange: 443-443
selfLink: https://www.googleapis.com/compute/v1/projects/tpdrops/global/forwardingRules/tpd-int-https-ipv4
target: https://www.googleapis.com/compute/v1/projects/tpdrops/global/targetHttpsProxies/int-tpd-target-proxy-2
**gcloud compute backend-services describe tpd-prod-back**
affinityCookieTtlSec: 0
backends:
- balancingMode: UTILIZATION
capacityScaler: 0.0
group: https://www.googleapis.com/compute/v1/projects/tpdrops/regions/us-central1/networkEndpointGroups/tpd-front
cdnPolicy:
cacheKeyPolicy:
includeHost: true
includeProtocol: true
includeQueryString: true
cacheMode: CACHE_ALL_STATIC
clientTtl: 3600
defaultTtl: 3600
maxTtl: 86400
negativeCaching: false
requestCoalescing: true
serveWhileStale: 86400
signedUrlCacheMaxAgeSec: '0'
connectionDraining:
drainingTimeoutSec: 0
creationTimestamp: '2021-10-25T04:09:29.908-07:00'
description: ''
enableCDN: true
fingerprint: 5FNZk6GXJTw=
iap:
enabled: false
id: '6357784085114072710'
kind: compute#backendService
loadBalancingScheme: EXTERNAL
logConfig:
enable: true
sampleRate: 1.0
name: tpd-prod-back
port: 80
portName: http
protocol: HTTP
selfLink: https://www.googleapis.com/compute/v1/projects/tpdrops/global/backendServices/tpd-prod-back
sessionAffinity: NONE
timeoutSec: 30
**gcloud compute url-maps describe int-tpd**
creationTimestamp: '2021-08-29T06:08:35.918-07:00'
defaultService: https://www.googleapis.com/compute/v1/projects/tpdrops/global/backendServices/tpd-prod-back
fingerprint: trtG9xBMlvE=
hostRules:
- hosts:
- acpt.travelpricedrops.com
pathMatcher: path-matcher-2
- hosts:
- int.travelpricedrops.com
pathMatcher: path-matcher-1
- hosts:
- api.acpt.travelpricedrops.com
pathMatcher: path-matcher-3
- hosts:
- api.int.travelpricedrops.com
pathMatcher: path-matcher-4
- hosts:
- api.travelpricedrops.com
pathMatcher: path-matcher-5
- hosts:
- travelpricedrops.com
pathMatcher: path-matcher-6
id: '6018005644614187068'
kind: compute#urlMap
name: int-tpd
pathMatchers:
- defaultService: https://www.googleapis.com/compute/v1/projects/tpdrops/global/backendServices/tpd-acpt-back
name: path-matcher-2
- defaultService: https://www.googleapis.com/compute/v1/projects/tpdrops/global/backendServices/tpd-int-http
name: path-matcher-1
- defaultService: https://www.googleapis.com/compute/v1/projects/tpdrops/global/backendServices/tpd-api-acpt
name: path-matcher-3
- defaultService: https://www.googleapis.com/compute/v1/projects/tpdrops/global/backendServices/tpd-api-int
name: path-matcher-4
- defaultService: https://www.googleapis.com/compute/v1/projects/tpdrops/global/backendServices/tpd-api
name: path-matcher-5
- defaultService: https://www.googleapis.com/compute/v1/projects/tpdrops/global/backendServices/tpd-prod-back
name: path-matcher-6
selfLink: https://www.googleapis.com/compute/v1/projects/tpdrops/global/urlMaps/int-tpd
**gcloud compute target-http-proxies describe int-tpd-target-proxy-2**
ERROR: (gcloud.compute.target-http-proxies.describe) Could not fetch resource:
- The resource 'projects/tpdrops/global/targetHttpProxies/int-tpd-target-proxy-2' was not found
Your load balancer's configuration looks ok; you have a https-ssl-secured frontend on port 443 pointing to a http backend on port 80 which means that SSL is resolved at the load balancer and sent in plain http to your backend.
Error you're getting means (as per documentation) that the client disconnected before load balancer could reply:
client_disconnected_before_any_response - The connection to the client was broken before the load balancer sent any response.
Now to answer your questions.
Since the images are served directly by your app (I didn't see any host-path rules saying otherwise) make sure that application can serve images in time. Set your application response timeout to 10 seconds or more and this should solve the issue. Have a look at this discussion which may be quite usefull for you.
1.1 - there's also a configurable request timeout for Cloud Run services - you can check it by running gcloud run services describe SERVICE_NAME
The backend_service_name: "" string you mentioned may be empty - nothing to worry about - this is an expected behavior.
Additionally have a look at the Backend service timeout Timeouts and retries in external load balancing which may also put some light onto your case.
Lastly - have a look at How to debug failed requests with client_disconnected_before_any_response.
Related
I am working on a use case to do SNI based routing in envoy without doing TLS termination
Scenario:
Customer makes a HTTPS request for domain name dd-server_name-zion-zorfy.com
I need to extract the domain name using SNI and forward the request to the upstream with DNS name wd-server_name-zion-zorfy.com
The server_name is dynamic in nature and can have different values. Basically, I need to replace dd prefix in the host name with wd and route the request to this destination without doing the TLS termination in envoy.
I have the below envoy configuration and when I make a HTTPS request, I am getting connection timeout error
static_resources:
listeners:
- name: listener_0
address:
socket_address:
protocol: TCP
address: 0.0.0.0
port_value: 9002
listener_filters:
- name: envoy.filters.listener.tls_inspector
typed_config:
"#type": type.googleapis.com/envoy.extensions.filters.listener.tls_inspector.v3.TlsInspector
filter_chains:
- filters:
- name: envoy.filters.network.sni_dynamic_forward_proxy
typed_config:
"#type": type.googleapis.com/envoy.extensions.filters.network.sni_dynamic_forward_proxy.v3.FilterConfig
port_value: 443
dns_cache_config:
name: dynamic_forward_proxy_cache_config
dns_lookup_family: V4_ONLY
- name: envoy.filters.http.lua
typed_config:
"#type": type.googleapis.com/envoy.extensions.filters.http.lua.v3.Lua
inline_code: |
function envoy_on_request(request_handle)
host = string.match(request_handle:headers():get("host"), "dd%-(.*)")
target = "wd-" .. host
REQUESTED_SERVER_NAME = target
end
- name: envoy.tcp_proxy
typed_config:
"#type": type.googleapis.com/envoy.extensions.filters.network.tcp_proxy.v3.TcpProxy
stat_prefix: tcp
cluster: dynamic_forward_proxy_cluster
tunneling_config:
hostname: "%REQUESTED_SERVER_NAME%:443"
clusters:
- name: dynamic_forward_proxy_cluster
lb_policy: CLUSTER_PROVIDED
cluster_type:
name: envoy.clusters.dynamic_forward_proxy
typed_config:
"#type": type.googleapis.com/envoy.extensions.clusters.dynamic_forward_proxy.v3.ClusterConfig
dns_cache_config:
name: dynamic_forward_proxy_cache_config
dns_lookup_family: V4_ONLY
Can someone help me what changes I need to make in the envoy configuration to get this working. Is this even doable?
I have two services and a gateway working with eureka:
user-service running on 8081
health-service running on 8082
So far I tried this:
routes:
- id: user-service-route
uri: http://localhost:8081/user-service
predicates:
- Path=/user-service/**
filters:
- RewritePath=(?<serviceName>.*), /api/v1/{serviceName}
But it doesn't work.
try
filters:
- RewritePath=(?<serviceName>.*), /api/v1/$\{serviceName}
Prefix location for all microservices if you use load balancer:
server:
port: 8765
management:
endpoints:
web:
exposure:
include: gateway
eureka:
client:
service-url:
defaultZone: http://localhost:8761/eureka
spring:
application:
name: gateway
cloud:
gateway:
discovery:
locator:
enabled: true
lower-case-service-id: true
default-filters:
- RewritePath=/api/v1/.*?/(?<remaining>.*), /$\{remaining}
routes:
- predicates:
- Path=/api/v1/profile/**
uri: lb://profile
- predicates:
- Path=/api/v1/dictionary/**
uri: lb://dictionary
I'm trying a simple ingress in gke.
Following the example from https://cloud.google.com/kubernetes-engine/docs/how-to/load-balance-ingress
the pods are up and running, services are active. When I create ingress I'm getting
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal ADD 48m loadbalancer-controller default/my-ingress
Warning Sync 2m32s (x25 over 48m) loadbalancer-controller Error during sync: Error running backend syncing routine: googleapi: got HTTP response code 404 with body: Not Found
I can't find the source of the problem. Any suggestion of where to look?
I have checked cluster add-ons and permissions
httpLoadBalancing enabled
- https://www.googleapis.com/auth/compute
- https://www.googleapis.com/auth/devstorage.read_only
- https://www.googleapis.com/auth/logging.write
- https://www.googleapis.com/auth/monitoring
- https://www.googleapis.com/auth/servicecontrol
- https://www.googleapis.com/auth/service.management.readonly
- https://www.googleapis.com/auth/trace.append
NAME READY STATUS RESTARTS AGE
hello-kubernetes-deployment-f6cb6cf4f-kszd9 1/1 Running 0 1h
hello-kubernetes-deployment-f6cb6cf4f-lw49t 1/1 Running 0 1h
hello-kubernetes-deployment-f6cb6cf4f-qqgxs 1/1 Running 0 1h
hello-world-deployment-5cfbc486f-4c2bm 1/1 Running 0 1h
hello-world-deployment-5cfbc486f-dmcqf 1/1 Running 0 1h
hello-world-deployment-5cfbc486f-rnpcc 1/1 Running 0 1h
Name: hello-world
Namespace: default
Labels: <none>
Annotations: kubectl.kubernetes.io/last-applied-configuration:
{"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"name":"hello-world","namespace":"default"},"spec":{"ports":[{"port":6000...
Selector: department=world,greeting=hello
Type: NodePort
IP: 10.59.254.88
Port: <unset> 60000/TCP
TargetPort: 50000/TCP
NodePort: <unset> 30418/TCP
Endpoints: 10.56.2.7:50000,10.56.3.6:50000,10.56.6.4:50000
Session Affinity: None
External Traffic Policy: Cluster
Events: <none>
Name: hello-kubernetes
Namespace: default
Labels: <none>
Annotations: kubectl.kubernetes.io/last-applied-configuration:
{"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"name":"hello-kubernetes","namespace":"default"},"spec":{"ports":[{"port"...
Selector: department=kubernetes,greeting=hello
Type: NodePort
IP: 10.59.251.189
Port: <unset> 80/TCP
TargetPort: 8080/TCP
NodePort: <unset> 32464/TCP
Endpoints: 10.56.2.6:8080,10.56.6.3:8080,10.56.8.6:8080
Session Affinity: None
External Traffic Policy: Cluster
Events: <none>
Name: my-ingress
Namespace: default
Address:
Default backend: default-http-backend:80 (10.56.0.9:8080)
Rules:
Host Path Backends
---- ---- --------
*
/* hello-world:60000 (<none>)
/kube hello-kubernetes:80 (<none>)
Annotations:
kubectl.kubernetes.io/last-applied-configuration: {"apiVersion":"extensions/v1beta1","kind":"Ingress","metadata":{"annotations":{"kubernetes.io/ingress.class":"gce"},"name":"my-ingress","namespace":"default"},"spec":{"rules":[{"http":{"paths":[{"backend":{"serviceName":"hello-world","servicePort":60000},"path":"/*"},{"backend":{"serviceName":"hello-kubernetes","servicePort":80},"path":"/kube"}]}}]}}
kubernetes.io/ingress.class: gce
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal ADD 107s loadbalancer-controller default/my-ingress
Warning Sync 66s (x15 over 107s) loadbalancer-controller Error during sync: Error running backend syncing routine: googleapi: got HTTP response code 404 with body: Not Found
Pulumi Cluster Config
{
"name": "test-cluster",
"region": "europe-west4",
"addonsConfig": {
"httpLoadBalancing": {
"disabled": false
},
"kubernetesDashboard": {
"disabled": false
}
},
"ipAllocationPolicy": {},
"pools": [
{
"name": "default-pool",
"initialNodeCount": 1,
"nodeConfig": {
"oauthScopes": [
"https://www.googleapis.com/auth/compute",
"https://www.googleapis.com/auth/devstorage.read_only",
"https://www.googleapis.com/auth/service.management",
"https://www.googleapis.com/auth/servicecontrol",
"https://www.googleapis.com/auth/logging.write",
"https://www.googleapis.com/auth/monitoring",
"https://www.googleapis.com/auth/trace.append",
"https://www.googleapis.com/auth/cloud-platform"
],
"machineType": "n1-standard-1",
"labels": {
"pool": "api-zero"
}
},
"management": {
"autoUpgrade": false,
"autoRepair": true
},
"autoscaling": {
"minNodeCount": 1,
"maxNodeCount": 20
}
},
{
"name": "outbound",
"initialNodeCount": 2,
"nodeConfig": {
"machineType": "custom-1-1024",
"oauthScopes": [
"https://www.googleapis.com/auth/compute",
"https://www.googleapis.com/auth/devstorage.read_only",
"https://www.googleapis.com/auth/service.management",
"https://www.googleapis.com/auth/servicecontrol",
"https://www.googleapis.com/auth/logging.write",
"https://www.googleapis.com/auth/monitoring",
"https://www.googleapis.com/auth/trace.append",
"https://www.googleapis.com/auth/cloud-platform"
],
"labels": {
"pool": "outbound"
}
},
"management": {
"autoUpgrade": false,
"autoRepair": true
}
}
The author of this post eventually figured out, that issue persist only when cluster is bootstrapped with pulumi.
It looks like you are missing a default backend (L7 - HTTTP LoadBalancer) for your default ingress controller. From what I observed it`s not deployed when you have Istio add-on enabled in your GKE cluster (Istio has its own default ingress/egress gateways).
Please verify if it`s up and running in your cluster:
kubectl get pod -n kube-system | grep l7-default-backend
I have been able to deploy this load balancer as a HTTP load balancer, and am now trying to deploy this as a HTTPS load balancer. I am attempting this with .JUNJA and .YAML on Google Cloud.
I keep getting the error: Invalid value for field 'resource.sslCertificates[0]': 'www.example.com'. The\\ URL is malformed.
resources:
- name: web-lb-hc #Create a health check for the backend
type: compute.v1.httpsHealthCheck
properties:
port: 443
requestPath: /
- name: web-url-map #Required to map url to backend
type: compute.v1.urlMap
properties:
defaultService: $(ref.backend.selfLink)
- name: backend
type: compute.v1.backendService #Deployment of backend for VM's
properties:
port: 443
portName: https
protocol: HTTPS #Defeined HTTP port for communication with backends
backends:
- name: backend
balancingMode: UTILIZATION
capacityScaler: 1.0
group: $(ref.web-ins-group.selfLink)
maxUtilization: 0.8
connectionDraining:
drainingTimeoutSec: 300
healthChecks:
- $(ref.web-lb-hc.selfLink)
- name: web-http-proxy
type: compute.v1.targetHttpsProxy
properties:
urlMap: $(ref.web-url-map.selfLink)
sslCertificates: ["www.example.com"]
- name: web-ipaddress
type: compute.v1.globalAddress
- name: web-http-forwardingrule #Creation of forwarding rule
type: compute.v1.globalForwardingRule
properties:
target: $(ref.web-http-proxy.selfLink)
IPAddress: $(ref.web-ipaddress.address)
IPProtocol: TCP #Chosen protocol
portRange: 443-443
Any help with this would be much appriciated!
sslCertificates is an array of urls.
To determine the url for an SSL certificate follow these steps.
Get a list of SSL certificates:
gcloud compute ssl-certificates list
Get the details for the SSL certificate:
gcloud compute ssl-certificates describe NAME
In the returned data near the bottom look for selfLink. This is the value you want to use. This value will look like this:
https://www.googleapis.com/compute/v1/projects/development/global/sslCertificates/production-lb
Good Morning,
I have a problem with cf doppel. Using cf logs app --recent I got an error
cf logs app FAILED Error dialing trafficcontroller server: read tcp
10.0.0.6:45719->139.25.25.233:4443: i/o timeout. Please ask your Cloud Foundry Operator to check the platform configuration
(trafficcontroller is wss://doppler.de.cloudlab.com:4443).
the same problem with cf push.
We are using CF 239 and CF CLI 6.22.1.
The doppler config is:
- name: doppler instances: 1 vm_type: medium azs: [INDIA] stemcell: ubuntu-trusty templates:
- {name: doppler, release: cf}
- {name: metron_agent, release: cf}
- {name: syslog_drain_binder, release: cf} networks:
- name: private properties:
doppler_endpoint:
shared_secret: password
- name: loggregator_trafficcontroller instances: 1 vm_type: medium azs: [INDIA] stemcell: ubuntu-trusty templates:
- {name: loggregator_trafficcontroller, release: cf}
- {name: metron_agent, release: cf}
- {name: route_registrar, release: cf} networks:
- name: private properties:
route_registrar:
routes:
- name: doppler
registration_interval: 20s
port: 8081
uris:
- "doppler.<%= system_domain %>"
- name: loggregator
registration_interval: 20s
port: 8080
uris:
- "loggregator.<%= system_domain %>"
nc is able to establish a connection to the router.
nc -vz 139.25.25.233 4443
Connection to 139.25.25.233 4443 port [tcp/*] succeeded!
any ideas?
Update:
some more information
$ cf curl /v2/info
{
"name": "",
"build": "",
"support": "",
"version": 0,
"description": "CloudFoundry IN",
"authorization_endpoint": "https://login.de.cloudlab.com",
"token_endpoint": "https://uaa.de.cloudlab.com",
"min_cli_version": null,
"min_recommended_cli_version": null,
"api_version": "2.57.0",
"app_ssh_endpoint": "ssh.de.cloudlab.com:2222",
"app_ssh_host_key_fingerprint": null,
"app_ssh_oauth_client": "ssh-proxy",
"logging_endpoint": "wss://loggregator.de.cloudlab.com:4443",
"doppler_logging_endpoint": "wss://doppler.de.cloudlab.com:443"
}
Cloud Foundry CLI is upgraded to 6.22.2, due to that you might face issues with existing CF CLI command prompt that you have installed in your box for pushing the app. please try installing the latest CLI from the below location and verify once.
https://github.com/cloudfoundry/cli/releases
It looks like your Cloud Foundry operator changed the Doppler endpoint port from 4443 to 443.
The cf CLI caches the doppler endpoint url locally, and is still pointing at the old port.
Use cf api or cf login -a to set your endpoint again; that will refresh the locally cached data.
Finally, I found it. I was a problem with the connection to the DNS Server. After fixing that it works. DNS was not answering.