Amazon introduced support for Request Tracing in 2016 for all of its Application Load Balancers (ALBs).
How are Access Logging, Request Tracing and CloudTrail logging, different from each other?
When to use what?
How is one better than the other and under what circumstances?
CloudTrail logs API calls within your account. Load balancer log files log HTTP client requests to your load balancer (e.g. web traffic). Very different types of information.
Load Balancer access logs record information about who, where and what HTTP requests are being made from clients. This information is useful for traffic analysis.
Request Tracing adds a custom header X-Amzn-Trace-Id into the request to your origin (typically a web server). This is used to trace (track) requests thru your origin servers.
Related
We have a server configured under a ALB associated with a WAF
As the underlying service receives requests for your web sites, it forwards those requests to AWS WAF for inspection against your rules. Once a request meets a condition defined in your rules, AWS WAF instructs the underlying service to either block or allow the request based on the action you define.
Above is mentioned in AWS FAQ and according to that I have a understanding that requests first comes to ALB and then forward to WAF.
My questions are in my environment I see some bad requests comes and returns 400 on ALB. they are not counted in WAF so is that mean bad requests are process in ALB? Bad requests are not forwarded to WAF? If I want to forward all the ALB request to WAF is it possible?
I've been working on improving performance of my APIs hosted on various AWS services this week, and I've found that a significant portion of the time on some of the non-Lambda calls (namely ECS) is taken by DNS Lookup, TCP Handshake, and SSL Handshake. Below are the response times broken down by Postman.
The endpoint pointing to an ECS service does not cache the handshakes
Headers returned by ECS endpoint
The endpoint pointing to a Lambda does
Headers returned by Lambda endpoint
More details:
Both services are hosted in the same zone, the API Gateway for both is also hosted in the same zone, the only difference is that for the Lambda, the route goes from Route53 -> API Gateway -> Lambda integration, where the ECS goes Route53 -> API Gateway -> Application load balancer (private VPC link, HTTP) -> ECS service, the service being an NGINX reverse proxy listening for HTTP and routing those requests to a Django container hosted in the same service.
Question:
I'm looking to enable or set up caching for the ECS service in order to cut down response times to below or close to 100ms. I haven't found any details on this caching behavior or by extension how to set it up. How can I go about setting it up, and perhaps read more about this behavior? Thanks
Does the Lambda request come second? The cache will reuse the same handshake for each sub/domain within the session, so the first should always have a penalty. Caching the DNS is generally at machine level, but TLS/SSL has to be handshook on every new session.
In both situations, it is API Gateway which terminates your public SSL. API Gateway makes a separate internal request to Lambda or ECS. Depending how you have setup your routing, you could also remove the API Gateway from the ECS route and have the NGINX terminate SSL and serve the certificate and remove one of the (slower) hops?
1.) BotControl
2.) AmazonIpReputationList
3.) AnonymousIp
4.) CommonRuleSet
5.) KnownBadInputs
6.) LinuxRuleSet
7.) SQL injection
these are all rule sets I have set in place but is there a way for me to tests against them to make sure they are working as intended?
basically you would like to test your rules, here is my 2 cents
Counting the web requests that match the rules in a web ACL
Configure all the rules in a web ACL to count web requests
Set the default action for the web ACL to allow requests
Viewing a sample of the web requests that API Gateway CloudFront or an Application Load Balancer has forwarded to AWS WAF Classic
In the AWS WAF Classic console, you can view a sample of the requests
that API
Gateway CloudFront or an Application Load Balancer has forwarded to AWS WAF
Classic for inspection. For each sampled request, you can view detailed data
about the request, such as the originating IP address and the headers included
in the request. You also can view which rule the request matched, and whether
the rule is configured to allow or block requests.
Logging and monitoring web ACL traffic
You can enable logging to get detailed information about traffic that
is analyzed by your web ACL. Logged information includes the time that
AWS WAF received a web request from your AWS resource, detailed
information about the request, and details about the rules that the
request matched. You can send your logs to an Amazon CloudWatch Logs
log group, an Amazon Simple Storage Service (Amazon S3) bucket, or an
Amazon Kinesis Data Firehose.
please refer this doc for more information.
In AWS X-Ray if request starts with API Gateway - it populates trace with request info, such as URL or method. But if request initiates with ALB - only trace ID is supplied.
Is there a way supply information similar to API Gateway?
From the documentation:
Elastic Load Balancing application load balancers add a trace ID to
incoming HTTP requests in a header named X-Amzn-Trace-Id.
Load balancers do not send data to X-Ray, and do not appear as a node
on your service map.
It looks like if you wanted more information about the request you could possibly tie that back to a record in the Load Balancer's access log.
We have recently started using AWS ELB. We are struggling to capture the following headers at ELB.
Referer
User-Agent
Any way to get it done?
AFAIK you cannot configure the access logs format for the LB (classic or other wise).
For classic ELB look at the logging documentation to see what fields are logged.
It does log the user agent but the referer is not logged.
Depending on what kind of application architecture you have behind your ELB you can configure that application for access logs and you would have more control over the access logs.
Example: Say you have an Spring boot based EC2 behind your ELB, you can configure the spring boot embedded tomcat to log whatever headers you need in the access logs.