502 Server Error after configuring the Load Balancer in Google Cloud Platform - google-cloud-platform

I am using WordPress Certified by Bitnami and Automattic, and one VM Instance running in Google Cloud Compute Engine.
I configured a free SSL certificate from Let's Encrypt for my website and also configured the Certbot Auto-Renewal script.
I tried using Cloudflare and I was receiving 5xx errors sometimes, mostly 522 timeout error. I stopped using the Cloudflare service, and I tried to configure a GCP load balancer for my VM Instance.
I created an Unmanaged Instance Group and I configured the HTTP protocol for my backend service with Cloud CDN Enabled in the load balancer, and for the Frontend, I configured an HTTP and HTTPS protocol and created a Google Managed SSL Certificate for the HTTPS protocol in my load balancer.
(The SSL certificate is ACTIVE)
I used this link to configure my load balancer in Google Cloud Platform:
https://docs.bitnami.com/google-templates/how-to/configure-lb-ssl-google-templates/
The problem is that I have 2 SSL Certificates and I get 502 Server Error:
*
"Error: Server Error The server encountered a temporary error and
could not complete your request. Please try again in 30 seconds."
*
I don't know how to solve this problem.
I just want to use a very basic and common configuration for my website.
I also want to know why I received a 522 timeout error from Cloudflare and how to solve it.
I need a quick response and appreciate your answers and help in advance.

I would advise you to follow 1 to create a HTTPS Load Balancer with the backend service 2. Once you create that, you can enable CDN 3.
Regarding the errors, Make sure that your backend instance is healthy and supports HTTP/2 protocol. You can verify this by testing connectivity to the backend instance using HTTP/2.
After you verify that the VM uses the HTTP/2 protocol, make sure your firewall setup allows the health checker and load balancer to pass through.
If there are no problems with the firewall setup, ensure that the load balancer is configured to talk to the correct port on the VM. I will also suggest you to walkthrough 4 for more steps that you can take to troubleshoot this issue.

Related

How to set up a GCP Global External HTTP(S) LoadBalancer for gRPC?

I have created a google cloud load balancer: configuration.
The backend is an unmanaged instance group. For example, it consists of one VM. The gRPC service is deployed on the VM (port 443). gRPC health checks are successful. But the gRPC client cannot connect to the service. I can't find a solution to this problem.
The last thing I found in the documentation:
If you use HTTP/2, you must use TLS. HTTP/2 without encryption is not
supported.
Could this be a solution and I just need to secure the gRPC connection with SSL/TLS?
You need to enable TLS on the Load Balancer and also between the Load balancer and your backend VM

gRPC in AWS Elastic Beanstalk load balancer / network setup

I have been at this for a couple of days and just cant figure it out.
I have tried this with gRPC in node.js and java on Elastic Beanstalk. On a normal VPS its quite simple just create a proxy grpcpass and it's set. I would like to move my micro services over to AWS Elastic Beanstalk but cant get the gRPC to connect.
What I did:
Created a new Java environment on Elastic Beanstalk and deployed my service. The gRPC server is on port 9086.
I have looked around the net and the closest thing I could find to a tutorial is New – Application Load Balancer Support for End-to-End HTTP/2 and gRPC but it does not cover how to setup the load balancer for gRPC for an instance.
Using the guide I made a few changes to the Target group like so:
Created a Target Group using the instances configuration
I have tried building the target group with both http and https for port 9086,
after creating the target group I registered the instance on the target group
After that I went to the load balancer and created a listener on port 443 and forwarded it to the target group. Port 443 is also open on the security policy.
The security listener settings pointing it to the AWS certificate allocated to the url.
I have tried both http and https on the target group on port 9086 but all my gRPC client calls fail with either status 13 or 14 meaning the request is not going through. I have confirmed in the logs the gRPC server is up and running.
Does anybody know where I am going wrong here? I feel like its something simple that I am missing, just can't find any tutorials or documentation on the proper way to set this up. Is what I am trying to do even possible on AWS Elastic Beanstalk?
From what I see on your screens, your ALB targets were added but they did not pass the health check. Meaning, that they are not allowed to accept any traffic yet.
You can find a good sample of a gRPC application with an implemented health check in the attached file in this article:
https://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/deploy-a-grpc-based-application-on-an-amazon-eks-cluster-and-access-it-with-an-application-load-balancer.html#attachments-abf727c1-ff8b-43a7-923f-bce825d1b459

How to add Cloud CDN to GCP VM? Always no load balancer available

I have a running Web server on Google Cloud. It's a Debian VM serving a few sites with low-ish traffic, but I don't like Cloudflare. So, Cloud CDN it is.
I created a load balancer with static IP.
I do all the items from the guides I've found. But when it comes time to Add origin to Cloud CDN, no load balancer is available because it's "unhealthy", as seen by rolling over the yellow triangle in the LB status page: "1 backend service is unhealthy".
At this point, the only option is to choose Create a Load Balancer.
I've created several load balancers with different attributes, thinking that might be it, but no luck. They all get the "1 backend service is unhealthy" tag, and thus are unavailable.
---Edit below---
During LB creation, I don't see anywhere that causes the LB to know about the VM, except in cert issue (see below). Nowhere does it ask for any field that would point to the VM.
I created another LB just now, and here are those settings. It finishes, then it's marked unhealthy.
Type
HTTP(S) Load Balancing
Internet facing or internal only?
From Internet to my VMs
(my VM is not listed in backend services, so I create one... is this the problem?)
Create backend service
Backend type: Instanced group
Port numbers: 80,443
Enable Cloud CDN: checked
Health check: create new: https, check /
Simple host and path rule: checked
New Frontend IP and port
Protocol: HTTPS
IP: v4, static reserved and issued
Port: 443
Certificate: Create New: Create Google-managed certificate, mydomain.com and www.mydomain.com
Load balancer's unhealthy state could mean that your LB's healthcheck probe is unable to reach your backend service(Your Debian VM in this case).
If your backend service looks good now, I think there is a problem with your firewall configuration.
Check your firewall rules whether it allows healthcheck probe's IP address range or not.
Refer to the docoment below to get more detailed information.
Required firewall rule

Does Google Cloud HTTPS load balancer log back end errors?

Looking for way to debug why backend for NIFI is failing. I created a NIFI cluster (verison 1.9.0, HDF 3.1.1.4, AMBARI 2.7.3) on Google cloud. Created HTTPS load balancer terminating https front end, and back end is the instance group for SSL enabled NIFI cluster. Getting a 502 back end error in the browser when I hit the url for the load balancer. Is there a way for Google Cloud to log the error ? There must be an error returned somewhere to troubleshoot the root cause. I don't see messages in the nifi log or the vm instance /var/log/messages. Stackdriver hasn't shown me errors. I created the keystore and truststore and followed the NIFI SSL enable instructions. It might be related to the SSL configs, or possibly firewall rules are not correct. But I am looking for some more helpful information to find the error.
If I am understanding the question properly, you are looking for a way to get HTTPS load balancer logs due to back end errors and your intention is to find out the root cause.Load balancer basically return 502 error due to unhealthy backend services or for unhealthy backend VM 's.If your stackdriver logging is enabled, you might get this log using advanced filter or can search by selecting the load balancer name and look for/search 502:
Advanced filter for 502 responses due to failures to connect to backends:
resource.type="http_load_balancer"
resource.labels.url_map_name="[URL Map]"
httpRequest.status=502
jsonPayload.statusDetails="failed_to_connect_to_backend"
Advanced filter for 502 responses due to backend timeouts:
resource.type="http_load_balancer"
resource.labels.url_map_name="[URL Map]"
httpRequest.status=502
jsonPayload.statusDetails="backend_timeout"
Advanced filter for 502 responses due to prematurely closed connections:
resource.type="http_load_balancer"
resource.labels.url_map_name="[URL Map]"
httpRequest.status=502
jsonPayload.statusDetails="backend_connection_closed_before_data_sent_to_client"
The URL Map is same as the name of the load balancer for HTTP(S) for cloud console.If we create the various components of the load balancer manually, need to use the URL Map for advanced filter.
Most common root causes for "failed_to_connect_to_backend" are: 1. Firewall blocking traffic, 2. Web server software not running on backend instance, 3. Web server software misconfigured on backend instance, 4. Server resources exhausted and not accepting connections (CPU usage too high to respond, Memory usage too high, process killed ,the maximum amount of workers spawned and all are busy, Maximum established TCP connections), 5. Poorly written server implementation struggling under load or non-standard behavior.
Most common root causes for “backend_timeout” are 1. the backend instance took longer than the Backend Service timeout to respond, meaning either the application is overloaded or the Backend Service Timeout is set too low, 2. The backend instance didn't respond at all (Crashing during a request).
Most Common Root causes for” backend_connection_closed_before_data_sent_to_client” is usually caused because the keepalive configuration parameter for the web server software running on the backend instance is less than the fixed (10 minute) keepalive (HTTP idle) timeout of the GFE. There are some situations where the backend may close a connection too soon while the GFE is still sending the HTTP request.
The previous response was spot on. The nifi ssl configuration is misconfigured, causing the backend health check to fail with a bad certificate. I will open a new question to address the nifi ssl configuration.

Websockets with AWS and Elastic Beanstalk

I'm trying to get my websockets working with Amazon Web Service and Elastic Beanstalk (ELB).
I set up a proxy protocol according to:
http://docs.aws.amazon.com/ElasticLoadBalancing/latest/DeveloperGuide/enable-proxy-protocol.html
I've opened up all inbound traffic to my load balancer and other security groups on all ports, for all IPs.
I also kept the load balancer listerner as HTTP on port 80.
My websocket connection gives the following error when trying to connect on port 80:
failed: Error during WebSocket handshake: Unexpected response code: 400
And this error on port 8080:
failed: Error in connection establishment: net::ERR_CONNECTION_REFUSED
Would appreciate suggestions, I'm stuck at this point.
Thanks!
After configuring EC2 Security group your application will work with public IP.
But you still will get issue with using EB URL.
To solve this issue, you have to change EB configuration.
Go to EB environment page
Configuration
In Load Balancing section change protocol from HTTP to TCP.
Now you can click EB URL.
I was also looking for possible workaround for this issue, but it's quite easy irrespective of what platform language you are using to develop websocket program on AWS EC2, as am using Node.js nginx in my case, this should work for all supporting platforms.
Configure Security Group
In the AWS console, open the EC2 tab.
Select the relevant region and click on Security Group.
You should have an elasticbeanstalk-default security group if you
have launched an Elastic Beanstalk instance in that region for your
app.
click on Actions button at top, and select Edit inbound rules.
here in Type column select All TCP, or you can set some Custom TCP
rule as well to listen at your websocket port.
And that's it!
Note: If something is not working, check the "Events" tab in the Beanstalk application / environments and find out what went wrong.
AWS has launched new Application Load Balancer that supports web sockets. Change your ELB to Application Load Balancer and that will fix your issue.
https://aws.amazon.com/blogs/aws/new-aws-application-load-balancer/
You should create reverse proxy for Nginx server. You can include this config file in you .ebextensions folder to make Nginx support WS.
Also in the EBS's load balancer configuration change the protocol from HTTP to TCP.
Refer this blog to set up secure WebSocket.
This is a very old post but having searched around for answers on this I have found two things you need to do to get websockets on a custom port working with AWS EB.
Under the configuration of your EB environment. Go to Software and add an environment variable there. Make sure this is a case sensitive match with your code.
Again under the configuration of your EB environment. Go to Load Balancer and add the custom port as a listener.
Make sure you save the new listener under the config and on reload it should be routing websocket traffic on a custom port