I'm new to AWS and Node so I really appreciate everyone's suggestion on this one. Recently in my system, something strange happened. Sometimes (~20 times/ month), a lambda request got timeout. It's just a simple request in middleware for authentication (I'm using node.JS serverless). The issues is I cannot trace this request from API gateway. Sorry I'm suck at explaining things so I'm gonna put it like this:
My system's flow: CloudFront -> API Gateway -> Lambda -> RDS
Normally if I want to trace a request:
I get the requestID from Lambda's Log
And search it in the API Gateway's Log:
But for these Timeout request:
I cannot find anything at all in API Gateway's Log:
Like it appeared out of nowhere... Please help and thanks for the suggestion.
You may already be aware but API Gateway has a hard max timeout limit of 29 seconds and stops waiting for the integration to respond after that time.
The Request ID that you are pulling from the lambda logs is the lambda request ID. This is a bit of speculation but I'm guessing since the Lambda took longer than 29 seconds and never returned to API Gateway since API Gateway timed out and stopped waiting, the API Gateway never received a response from the Lambda function to know what the lambda request ID is.
There still should be some API Gateway logs for this invocation but it won't contain any of the normal integration response log lines because it never received a response from the lambda integration.
Perhaps try searching for 504 in the API Gateway logs since this is the response code when the gateway times out.
Or as AADProgramming recommends, the X Ray service was designed to help users track requests across many services so enabling that would be helpful here.
Related
I have created a micro service architecture which flows as follows:
Api call -> Api gateway -> Eventbridge -> SNS -> Lambda
The reason for this is to use SNS instead of SQS to decouple applications for true serverless compute without the need for lambda to continuously poll sqs, pub sub over push poll.
The trouble is that although the execution is fine and the lambdas run as expected the return received by the user or app is the eventbridge response. I can’t find any docs on how eventbridge handles responses for http requests through API gateway.
Does anyone have any ideas or docs to push me in the right direction.
Thanks!
In your setup it's not possible to have the Lambda response proxied back to the api request initiator, as your client is very much decoupled of the actual request processing.
Almost identical issue was experienced here
You need to rethink the process as a whole:
what operation you want to complete via the API request?
does the processing of the request really need to be asynchronous (= does it take long time to complete?)
can you handle the request with a Lambda function, delegate to sns from there and finally generate desired response back to the client?
So as it turns out the answer is yes and no for anyone coming across this in the future.
With the current setup another database is required and the responses can be inserted into it with a transaction ID. This transaction ID can be generated by the client during the request so a subsequent call to find the response in the table can be made.
Alternatively Websocket or GraphQL api’s or would allow for asynchronous invocation if really depends on your use case and accepted complexity.
Thanks for everyone’s inputs!
I am getting below response when I hit AWS API gateway.
502: Bad Gateway
{
"message": "Internal server error"
}
I know that API gateway request payload limit is 10MB and I haven't exceeded that, but response size might be beyond 10 MB. Can response size affect?
Could find nothing in CloudWatch, not even getting logs related to this invocation.
Tried to follow this, but could find nothing.
First, Try to test your lambda function by invoking it in the api gateway Console or in the test part in the lambda console.
If the lambda function works, then maybe its a problem related to the parameters, content ( Integration request / response) or the permissions of the API Gateway + lambda.
Check also the time out of the lambda /!\
There's a lot of questions out there pertaining to "lambda" but I haven't touched that in my life, so it probably isn't relevant to this question.
All I've done so far is import a swagger json spec into AWS API Gateway and try my best to configure it to connect to my REST api.
My issue here is that when I try calling the API gateway endpoint through the test console, everything works as expected.
But then why I try querying the same endpoint from outside AWS like any end user, I get this nondescript 500 response.
The nifty thing is that since I own the backend service, I can see the logging, which clearly indicates that a successful 200 response was returned (which should be the json response shown in the first screenshot).
So now I know that it's AWS's fault that I'm getting a 500. But I'm honestly not sure what the issue could be.
Here are all relevant configuration screens for this particular endpoint.
Method Request
Integration Request
Integration Response
Method Response
If there's any screen/setting I didn't provide that would be helpful, just let me know.
TLDR, api gateway endpoint works in online console ui, but not in real life
I follows #MarkB's advice and linked API Gateway to an IAM role with permissions to write to Cloudwatch and then here's the kicker: you have to redeploy the stage to jumpstart Cloudwatch (this is actually very poorly documented, aws).
And when I found out that I had to redeploy the gateway, that's when I also conveniently discovered that for any of the configuration changes I was making to be applied, I actually had to redeploy the entire thing.
So after redeploying, everything started to work.
I have built an EC2 reverse proxy (Nginx) that communicates with an external API endpoint over the internet. I have a Route53 DNS with an A record linking to my EC2. There are a few endpoints (Nginx locations) and depending on which url you hit, you are redirected to a specific proxy location, and forwarded to the right endpoint on the external API. It all works great.
Now i want to create some type of job that will test this process periodically to ensure that it's running and notify me if it's not. AWS has so many tools and i think i need to use Lambda and API Gateway.
I'd like to hit my url(Route53 DNS) go thru the EC2 and receive a response from the endpoint server. My site does this, postman can, but i can't figure out how to accomplish this in an automated way and alert me based on the response values.
how can i test my full pathway (www.example.com/option -> nginxEC2 path('/option') -> www.endpoint.com/option) and be notified based on the results.
EDIT: i need to be able to send a body with this. if i send it without body the server returns 404, if i can send with a body/payload, i'll get a response.
EDIT: basically looking for a way to hit my DNS, which thru A record, routes to my reverse proxy, to an endpoint. i just need to do an HTTP request to the Domain, and get and answer back and know the status code.
Mark B's solution is the closest as the free site he sent me has an option to pay for this service. gonna leave it open a few more days.
You definitely don't need API Gateway for this. That wouldn't help you test this at all. API Gateway would just give you an entirely new API that you would need to test.
You could use Lambda for this as you mentioned. You would write a Lambda function that hits the URLs you want to test, checks the results, and sends you a message over SES or SNS or some other means when it fails. The Lambda function could be configured to automatically run on a schedule.
However, AWS already has a service that does exactly what you are looking for: Route53 Health Checks.
What you are describing is called an HTTP health check or HTTP uptime monitor. There are tons of services that provide this feature, some of them free.
It looks like the word that you're looking for is trace -- you want to trace requests along your application. AWS offer for that is X-Ray. As you see in their official documentation, you need to use their SDK to instrument your application, which talks to a deamon in your EC2 instance. You can then integrate with CloudWatch and SNS to be notified upon errors (e.g. 4xx codes): https://aws.amazon.com/blogs/devops/using-amazon-cloudwatch-and-amazon-sns-to-notify-when-aws-x-ray-detects-elevated-levels-of-latency-errors-and-faults-in-your-application/
Hope it helps!
I want to submit a GET request using Authorization and Content-Type(application/json) as headers. Authorization would contain "Bearer " + token.
I am using Lambda function as to trigger when user hits this API endpoint. As this is get request, I am sending body = undefined in parameters.
But since 4-5 hours I am getting "internal server error" as of status code =502.
I noticed that these requests are due to bad gateway, still they are captured in cloudwatch logs.
The thing is, I want to know how to setup a GET request in AWS API gateway as I have not much experience in this technology.