I am using AWS Api Gateway and I'd like to trace full request and response. Some of my integrations are lambdas and some other http endpoints.
I enabled stage > Logs > "Log full requests/responses data" and I can see logs on CloudWatch.
This seemed to be perfect until I discovered bodies were being TRUNCATED... There is a limitation of 1024 bytes in ApiGateway sending logs to cloudwatch.
Is there any solution to this?
I am considering using a lambda as a proxy (with http-proxy) as my last option...
As you stated, there is no current known solution to this problem and AWS is aware of the problem.
API Gateway currently limits log events to 1024 bytes. Log events larger than 1024 bytes, such as request and response bodies, will be truncated by API Gateway before submission to CloudWatch Logs.
You can see additional known issues at the AWS documentation page for API Gateway here.
https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-known-issues.html
According to the AWS documentation, full logging should not be turned on in AWS API Gateway, in production, because of PII (source).
This is purely for development troubleshooting purposes. So the log limitation 1024 bytes continue to exists, because of this.
Related
I'm new to AWS and Node so I really appreciate everyone's suggestion on this one. Recently in my system, something strange happened. Sometimes (~20 times/ month), a lambda request got timeout. It's just a simple request in middleware for authentication (I'm using node.JS serverless). The issues is I cannot trace this request from API gateway. Sorry I'm suck at explaining things so I'm gonna put it like this:
My system's flow: CloudFront -> API Gateway -> Lambda -> RDS
Normally if I want to trace a request:
I get the requestID from Lambda's Log
And search it in the API Gateway's Log:
But for these Timeout request:
I cannot find anything at all in API Gateway's Log:
Like it appeared out of nowhere... Please help and thanks for the suggestion.
You may already be aware but API Gateway has a hard max timeout limit of 29 seconds and stops waiting for the integration to respond after that time.
The Request ID that you are pulling from the lambda logs is the lambda request ID. This is a bit of speculation but I'm guessing since the Lambda took longer than 29 seconds and never returned to API Gateway since API Gateway timed out and stopped waiting, the API Gateway never received a response from the Lambda function to know what the lambda request ID is.
There still should be some API Gateway logs for this invocation but it won't contain any of the normal integration response log lines because it never received a response from the lambda integration.
Perhaps try searching for 504 in the API Gateway logs since this is the response code when the gateway times out.
Or as AADProgramming recommends, the X Ray service was designed to help users track requests across many services so enabling that would be helpful here.
I am preparing for AWS exam and I found some documentation about AWS CloudTrail and AWS X-RAY where it creates confusion on their usage requirement.
I have came across following question where requirement was to trace and analyse the user request as it travels through Amazon API Gateway APIs to underlying services.
As per my understanding, we can use CludTrail to trace and analyse the user request. But the correct answer was AWS XRAY.
The documents which have referred mentions that, we can use AWS CloudTrail logs for tracing,Security Analysis, Resource Change Tracking and Compliance/Auditing. On the other hand, we can use AWS X-RAY to analyse and debug applications running on distributed micro service architecture.
XRAY and CloudTrail usage both have the term Analyse and trace. So it is quite confusing to which service should we choose under such requirement to trace and analyse the user request
X-Ray is more detailed in the information it provides for the request's flow and state. It scans the request all the way through its lifetime from when it is received in the api gateway to whatever services are called and executed after that. So I imagine that is why it is the preferred option.
I have created an API using AWS api gateway. Under stages the "Log full requests/responses data" checked and "Enable CloudWatch Logs" is also checked.
When i look at the logs in cloudwatch i see some of the logs are "TRUNCATED". In-fact all of the logs are truncating request and response body. Is there any way to view the entire request/response.
Since there will be multiple integration points it make sense to see the entire logs.
Looks like it's one of the known issues in AWS API Gateway.
API Gateway currently limits log events to 1024 bytes. Log events
larger than 1024 bytes, such as request and response bodies, will be
truncated by API Gateway before submission to CloudWatch Logs.
API Gateway limits log events to 1024 bytes and cannot be increased. The log events larger than 1024 bytes, such as request and response bodies, will be truncated by API Gateway before submission to CloudWatch Logs.
A workaround can be using Lambda proxy integration with API Gateway.
With Lambda proxy integration, API Gateway passes the request as is to the integrated Lambda function with only exception that the order of the request parameters is not preserved.
This request data includes the request headers, query string parameters, URL path variables, payload, body and request context. As Lambda does not truncate log entry, all headers and query string parameters are logged in Lambda's CloudWatch log and can be seen.
Downside of this approach is that lambda would add to the cost.
Read about API Gateway CloudWatch logs here : https://cloudnamaste.com/api-gateway-cloudwatch-logs/
If I create a public endpoint using AWS API Gateway, the entire world could access it. This would be a problem because the end point would trigger an AWS Lambda function. If we assume that I can't query a data source to determine the frequency that the incoming IP address queried the resource in the past, what would be the best practice for protecting this end point from abuse? Do I have any other security options?
I realize I could use a reCaptcha but this would still invoke the AWS Lambda function and would incur costs if done a million times over a short window of time.
A very simple way of protecting your API gateway
Use AWS Cloudfront with TTL 0 and pass custom headers from AWS Cloudfront to API gateway
Use AWS WAF with AWS Cloudfront
AWS API Gateway also handles some basic level of DDOS attacks.
Kindly also view these blogs for securing AWS API Gateway
https://aws.amazon.com/blogs/compute/protecting-your-api-using-amazon-api-gateway-and-aws-waf-part-i/
https://aws.amazon.com/blogs/compute/protecting-your-api-using-amazon-api-gateway-and-aws-waf-part-2/
You are probably looking for throttling limit configuration or usage plan definition:
To prevent your API from being overwhelmed by too many requests,
Amazon API Gateway throttles requests to your API using the token
bucket algorithm, where a token counts for a request. Specifically,
API Gateway sets a limit on a steady-state rate and a burst of request
submissions against all APIs in your account. In the token bucket
algorithm, the burst is the maximum bucket size.
When request submissions exceed the steady-state request rate and
burst limits, API Gateway fails the limit-exceeding requests and
returns 429 Too Many Requests error responses to the client. Upon
catching such exceptions, the client can resubmit the failed requests
in a rate-limiting fashion, while complying with the API Gateway
throttling limits.
As an API developer, you can set the limits for individual API stages
or methods to improve overall performance across all APIs in your
account. Alternatively, you can enable usage plans to restrict client
request submissions to within specified request rates and quotas. This
restricts the overall request submissions so that they don't go
significantly past the account-level throttling limits.
References:
https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-request-throttling.html
https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-create-usage-plans-with-console.html#api-gateway-usage-plan-create
I have this problem
In summary,
I send the correct request to amazon API though, It doesn't work somehow (return 400Error).
I did everything in my application side.
Now, I would like to know the Amazon Server log to research into the problem.
How can I check the amazon API server log, or is it possible?
The answer to your previous question is not entirely correct in my opinion. It's just a generic answer about HTTP 400 codes, and not very helpful in your situation. Did you look here to see the official list of error response codes from SNS and their meanings?
If you want logs of your AWS API calls you need to enable CloudTrail.