We are making a REST call to a spring boot application hosted in PCF environment. There is an AWS load balancer in front of our application to handle traffic management.
We are consuming the http request in a streaming manner using apache file upload library [https://commons.apache.org/proper/commons-fileupload/]. On processing the request, we are immediately sending the response back without waiting for the whole request to arrive. The size of the http request is normally large in the range of 100 MB.
This implementation works fine without AWS load balancer in between. When AWS load balancer is present, it terminates the http call after few bytes of response has been sent.
If we defer the response sending till whole request is received in the server side, request go through without any failures.
If the size of the http request is small, then also implementation works fine.
Any idea why AWS load balancer terminates the http call, if we start sending the http response before receiving full http request.
I assume in your case you are using an application load balancer, however I'd recommend in your case is to use network load balancer, classic may work too as they provide more transparency to your requests than application locad balancer, also its recommended for API implementations
if this is still not solving your case, consider implementing HA proxy LB
https://www.loadbalancer.org/blog/transparent-load-balancing-with-haproxy-on-amazon-ec2/
Related
I have created an API that has two endpoints. I containerized that API and deployed that into the ECS Fargate container behind the Application Load Balancer.
End Points.
Get = Return the status of the API
Post = Insert data into the RDS.
api/v1/healthcheck is working
api/v1/insertRecord is not working => 502 bad Gateway
The problem I am running into is that I am able to get the HealthCheck response but I am not able to make the Post API call I am getting 502 Bad Gateway error
Target Group
My target group is directed to the healthcheck endPoint so my ecs stays up. Can someone plz tell me where am I making mistake?
The 502 (Bad Gateway) status code indicates that the server, while acting as a gateway or proxy, received an invalid response from an inbound server it accessed while attempting to fulfill the request. if the service returns an invalid or malformed response, instead of returning that nonsensical information to the client.
Possible causes: taken from
check protocol and port number during REST call
The load balancer received a TCP RST from the target when attempting
to establish a connection.
The load balancer received an unexpected response from the target,
such as "ICMP Destination unreachable (Host unreachable)", when
attempting to establish a connection.
The target response is malformed or contains HTTP headers that are
not valid.
The target closed the connection with a TCP RST or a TCP FIN while
the load balancer had an outstanding request to the target.
you can enable cloudWatch log for further debugging.
I have a set of apps deployed in Docker containers that use websockets to communicate. One is the backend and one is the frontend.
I have both VM instances inside instance groups and served up through load balancers so that I can host them at https domains.
The problem I'm having is that in Google Compute Engine, the websocket connection is being closed after 30 seconds.
When running locally, the websockets do not time out. I've searched the issue and found these possible reasons, but I can't find a solution:
Websockets might time out on their own if you don't pass "keep alive" messages to keep them active. So I pass a keep alive from the frontend to the backend, and have the backend respond to the frontend, every 10 seconds.
According to the GCE websocket support docs, some type of "upgrade" handshake needs to occur between the backend and the frontend for the websocket connection to be kept alive. But according to MDN, "if you're opening a new connection using the WebSocket API, or any library that does WebSockets, most or all of this is done for you." I am using that API, and indeed when I inspect the headers, I see those fields:
The GCE background service timeout docs say:
For external HTTP(S) load balancers and internal HTTP(S) load
balancers, if the HTTP connection is upgraded to a WebSocket, the
backend service timeout defines the maximum amount of time that a
WebSocket can be open, whether idle or not.
This seems to be in conflict with the GCE websocket support docs that say:
When the load balancer recognizes a WebSocket Upgrade request from an
HTTP(S) client followed by a successful Upgrade response from the
backend instance, the load balancer proxies bidirectional traffic for
the duration of the current connection.
Which is it? I want to keep sockets open once they're established, but requests to initialize websocket connections should still time out if they take longer than 30 seconds. And I don't want to allow other standard REST calls to block forever, either.
What should I do? Do I have to set the timeout on the backend service to forever, and deal with the fact that other non-websocket REST calls may be susceptible to never timing out? Or is there a better way?
As mentioned in this GCP doc "When the load balancer recognizes a WebSocket Upgrade request from an HTTP(S) client followed by a successful Upgrade response from the backend instance, the load balancer proxies bidirectional traffic for the duration of the current connection. If the backend instance does not return a successful Upgrade response, the load balancer closes the connection."
In addition, the websocket timeout is a combination of the LB timeout and the bakckend time out. I understand that you already have modified the backend timeout. so you can also adjust the load balancer timeout according to your needs, pleas keep in mind that the default value is 30 seconds.
We have a similar, strange issue from GCP but this is without even using a load balancer. We have a WSS server, we can connect to it fine, it randomly stops the feed, no disconnect message, just stops sending WSS feeds to the client. It could be after 2-3 minutes it could be after 15-20 minutes but usually never makes it longer than that before dropping the connection.
We take the exact same code, the exact same build,(its all containerized) and we drop it on AWS and the problem with WSS magically disappears.
There is no question this issue is GCP related.
We are using Google Load Balancer with Tomcat Server. We have kept a specific timeout on load balancer from cloud console setting portal. Whenever any request takes more than the timeout time, GLB returns 502 i.e expected.
Here the problem is -
Whenever the request takes more than the given time, on tomcat side we are getting the same request again exactly after the timeout e.g when we have the timeout as 30 sec we got the same request on tomcat exactly after 30 sec.
On browser the response time for 502 is exactly twice of the timeout time. (It might be because of network turn around time but why always exact twice)
I assume you are referring to HTTP(S) load balancing. In this scenario, a reverse proxy is sitting in front of your application, handling requests and forwarding them to your backends. This proxy (a GFE) will retry as documented:
HTTP(S) load balancing retries failed GET requests in certain circumstances, such as when the response timeout is exhausted. It does not retry failed POST requests. Retries are limited to two attempts. Retried requests only generate one log entry for the final response. Refer to Logging for more information.
I'm developing a stocks app and have to keep users browser updated with pricing changes
I don't need to access past data, browser just have to get current data whenever it changes
is it possible to filter a dynamodb stream and expose an endpoint (behind api gateway) that could be used with a javascript EventSource?
I realize this is not using Server Sent Events but AWS just announced Serverless WebSockets for API Gateway. Pricing is based on minutes connected and number of messages sent.
Product Launch Article: https://aws.amazon.com/about-aws/whats-new/2018/12/amazon-api-gateway-launches-support-for-websocket-apis/
Documentation: https://docs.aws.amazon.com/apigateway/latest/developerguide/apigateway-websocket-api.html
Pricing: https://aws.amazon.com/api-gateway/pricing/
API Gateway is a store-and-forward service. It collects the response from whatever the back-end may happen to be (Lambda, an HTTP server, etc.) and then returns it en block to the browser -- it doesn't stream the response, so it would not be suited for use as an Eventsource.
AWS doesn't currently have a managed service offering that is obviously suited to this use case... you'd need a server (or more than one) on EC2, consuming the data stream and relaying it back to the connected browsers.
Assuming that running EC2 servers is an acceptable option, you then need HTTPS and load balancing. Application Load Balancer supports web sockets, so it also might also support an eventsource. A Classic ELB in TCP (not HTTP) mode should support an eventsource without a problem, though it might not correctly signal to the back-end when the browser connection is lost. Both of those balancers can also offload HTTPS for you. Network Load Balancer would definitely work for balancing an eventsource, but your instances would need to provide the HTTPS, since NLB doesn't offload it for you.
A somewhat unorthodox alternative might actually be AWS IoT, which has built-in websocket support... Not the same as eventsource, of course, but a streaming connection nonetheless... in such an environment, I suppose each browser user could be an addressable "thing."
I just noticed that gcp http(s) load balancer now supports websockets. I went to try it out and am having some problems. I have a gloabl https load balancer setup with Cloud CDN and a simple, no url-map, backend (Node.js). When I go to make websocket connection, I get a successful upgrade response but when I go to send frames to the server, they are never received. The server can send frames back to the client just fine. It is almost like the load balancer doesn't know that the connection has been upgraded and therefore doesn't allow any data sent from the client.
When I look in the logs for the https load balancer, I see the 101 Switching Protocols response and then statusDetails is "client_disconnected_after_partial_response" almost like it was a normal http request.
Any help would be appreciated.
After some investigation it seems that when a GCP load balancer is going through the GCP CDN, client messages never make it to the backend.
I worked around this by duplicating the backend configuration I wanted in a new backend called "websocket-prototype," and then just changing the "use cdn" setting for the websocket prototype. Finally, I mapped the path to my websocket server to the configuration without the CDN.