Does Cloudfront require special settings to trigger a log?
I have the following flow:
Devices -> Cloudfront -> API Gateway -> Lambda Function
which works, but Cloudwatch doesn't seem to create logs for the lambda function (or API Gateway).
However, the following flow creates logs:
Web/Curl -> API Gateway -> Lambda Function
In comments, above, we seem to have arrived at a conclusion that unanticipated client-side caching (or caching somewhere between the client and the AWS infrastructure) may be a more appropriate explanation for the observed behavior, since there is no known mechanism by which an independent CloudFront distribution could access a Lambda function via API Gateway and cause those requests not to be logged by Lambda.
So, I'll answer this with a way to confirm or reject this hypothesis.
CloudFront injects a header into both requests and responses, X-Amz-Cf-Id, containing opaque tokens that uniquely identify the request and the response. Documentation refers to these as "encrypted," but for our purposes, they're simply opaque values with a very high probability of uniqueness.
In spite of having the same name, the request header and the response header are actually two uncorrelated values (they don't match each other on the same request/response).
The origin-side X-Amz-Cf-Id is sent to the origin server in the request is only really useful to AWS engineers, for troubleshooting.
But the viewer-side X-Amz-Cf-Id returned by CloudFront in the response is useful to us, because not only is it unique to each response (even responses from the CloudFront cache have different values each time you fetch the same object) but it also appears in the CloudFront access logs as x-edge-request-id (although the documentation does not appear to unambiguously state this).
Thus, if the client side sees duplicate X-Amz-Cf-Id values across multiple responses, there is something either internal to the client or between the client and CloudFront (in the client's network or ISP) that is causing cached responses to be seen by the client.
Correlating the X-Amz-Cf-Id from the client across multiple responses may be useful (since they should never be the same) and with the CloudFront logs may also be useful, since this confirms the timestamp of the request where CloudFront actually generated this particular response.
tl;dr: observing the same X-Amz-Cf-Id in more than one response means caching is occurring outside the boundaries of AWS.
Note that even though CloudFront allows min/max/default TTLs to impact how long CloudFront will cache the object, these settings don't impact any downstream or client caching behavior. The origin should return correct Cache-Control response headers (e.g. private, no-cache, no-store) to ensure correct caching behavior throughout the chain. If the origin behavior can't be changed, then Lambda#Edge origin response or viewer response triggers can be used to inject appropriate response headers -- see this example on Server Fault.
Note also that CloudFront caches 4xx/5xx error responses for 5 minutes by default. See Amazon CloudFront Latency for an explanation and steps to disable this behavior, if desired. This feature is designed to give the origin server a break, and not bombard it with requests that are presumed to continue to fail, anyway. This behavior may cause various problems in testing as well as production, so there are cases where it should be disabled.
Related
AWS CloudFront and Azure CDN can dynamically compress files under certain circumstances. But do they also support dynamic compression for HTTP range requests?
I couldn't find any hints in the documentations only on the Google Cloud Storage docs.
Azure:
Range requests may be compressed into different sizes. Azure Front Door requires the content-length values to be the same for any GET HTTP request. If clients send byte range requests with the accept-encoding header that leads to the Origin responding with different content lengths, then Azure Front Door will return a 503 error. You can either disable compression on Origin/Azure Front Door or create a Rules Set rule to remove accept-encoding from the request for byte range requests.
See: https://learn.microsoft.com/en-us/azure/frontdoor/standard-premium/how-to-compression
AWS:
HTTP status code of the response
CloudFront compresses objects only when the HTTP status code of the response is 200, 403, or 404.
--> Range-Request has status code 206
See:
https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/ServingCompressedFiles.html
https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/206
• Yes, Azure CDN also supports dynamic compression for HTTP range requests wherein it is known as ‘object chunking’. You can describe object chunking as dividing the file to be retrieved from the origin server/resource into smaller chunks of 8 MB. When a large file is requested, the CDN retrieves smaller pieces of the file from the origin. After the CDN POP server receives a full or byte-range file request, the CDN edge server requests the file from the origin in chunks of 8 MB.
• After the chunk arrives at the CDN edge, it's cached and immediately served to the user. The CDN then prefetches the next chunk in parallel. This prefetch ensures that the content stays one chunk ahead of the user, which reduces latency. This process continues until the entire file is downloaded (if requested), all byte ranges are available (if requested), or the client terminates the connection.
Also, this capability of object chunking relies on the ability of the origin server to support byte-range requests; if the origin server doesn't support byte-range requests, requests to download data greater than 8mb size will fail.
Please find the below link for more details regarding the above: -
https://learn.microsoft.com/en-us/azure/cdn/cdn-large-file-optimization#object-chunking
Also, find the below link for more clarification on the types of compression and the nature of compression for Azure CDN profiles that are supported: -
https://learn.microsoft.com/en-us/azure/cdn/cdn-improve-performance#azure-cdn-standard-from-microsoft-profiles
Some tests have shown when dynamic compression is enabled in AWS CloudFront the range support is disabled. So Range and If-Range headers are removed from all request.
I have an application deployed on AWS Elastic Beanstalk, I added some simple licensing to stop abuse of the api, the user has to pass a licensekey as a field
i.e
search.myapi.com/?license=4ca53b04&query=fred
If this is not valid then the request is rejected.
However until the monthly updates the above query will always return the same data, therefore I now point search.myapi.com to an AWS CloudFront distribution, then only if query is not cached does it go to actual server as
direct.myapi.com/?license=4ca53b04&query=fred
However the problem is that if two users make the same query they wont be deemed the same by Cloudfront because the license parameter is different. So the Cloudfront caching is only working at a per user level which is of no use.
What I want to do is have CloudFront ignore the license parameters for caching but not the other parameters. I dont mind too much if that means user could access CloudFront with invalid license as long as they cant make successful query to server (since CloudFront calls are cheap but server calls are expensive, both in terms of cpu and monetary cost)
Perhaps what I need is something in front of CloudFront that does the license check and then strips out the license parameter but I don't know what that would be ?
Two possible come to mind.
The first solution feels like a hack, but would prevent unlicensed users from successfully fetching uncached query responses. If the response is cached, it would leak out, but at no cost in terms of origin server resources.
If the content is not sensitive, and you're only trying to avoid petty theft/annoyance, this might be viable.
For query parameters, CloudFront allows you to forward all, cache on whitelist.
So, whitelist query (and any other necessary fields) but not license.
Results for a given query:
valid license, cache miss: request goes to origin, origin returns response, response stored in cache
valid license, cache hit: response served from cache
invalid license, cache hit: response served from cache
invalid license, cache miss: response goes to origin, origin returns error, error stored in cache.
Oops. The last condition is problematic, because authorized users will receive the cached error if the make the same query.
But we can fix this, as long as the origin returns an HTTP error for an invalid request, such as 403 Forbidden.
As I explained in Amazon CloudFront Latency, CloudFront caches responses with HTTP errors using different timers (not min/default/max-ttl), with a default of t minutes. This value can be set to 0 (or other values) for each of several individual HTTP status codes, like 403. So, for the error code your origin returns, set the Error Caching Minimum TTL to 0 seconds.
At this point, the problematic condition of caching error responses and playing them back to authorized clients has been solved.
The second option seems like a better idea, overall, but would require more sophistication and probably cost slightly more.
CloudFront has a feature that connects it with AWS Lambda, called Lambda#Edge. This allows you to analyze and manipulate requests and responses using simple Javascript scripts that are run at specific trigger points in the CloudFront signal flow.
Viewer Request runs for each request, before the cache is checked. It can allow the request to continue into CloudFront, or it can stop processing and generate a reaponse directly back to the viewer. Generated responses here are not stored in the cache.
Origin Request runs after the cache is checked, only for cache misses, before the request goes to the origin. If this trigger generates a response, the response is stored in the cache and the origin is not contacted.
Origin Response runs after the origin response arrives, only for cache misses, and before the response goes onto the cache. If this trigger modifies the response, the modified response stored in the cache.
Viewer Response runs immediately before the response is returned to the viewer, for both cache misses and cache hits. If this trigger modifies the response, the modified response is not cached.
From this, you can see how this might be useful.
A Viewer Request trigger could check each request for a valid license key, and reject those without. For this, it would need access to a way to validate the license keys.
If your client base is very small or rarely changes, the list of keys could be embedded in the trigger code itself.
Otherwise, it needs to validate the key, which could be done by sending a request to the origin server from within the trigger code (the runtime environment allows your code to make outbound requests and receive responses via the Internet) or by doing a lookup in a hosted database such as DynamoDB.
Lambda#Edge triggers run in Lambda containers, and depending on traffic load, observations suggest that it is very likely that subsequent requests reaching the same edge location will be handled by the same container. Each container only handles one request at a time, but the container becomes available for the next request as soon as control is returned to CloudFront. As a consequence of this, you can cache the results in memory in a global data structure inside each container, significantly reducing the number of times you need to ascertain whether a license key is valid. The function either allows CloudFront to continue processing as normal, or actively rejects the invalid key by generating its own response. A single trigger will cost you a little under $1 per million requests that it handles.
This solution prevents missing or unauthorized license keys from actually checking the cache or making query requests to the origin. As before, you would want to customize the query string whitelist in the CloudFront cache behavior settings to eliminate license from the whitelist, and change the error caching minimum TTL to ensure that errors are not cached, even though these errors should never occur.
I'm new to AWS Lambda and Lambda Edge and I'm trying to understand the purpose. In Lambda Edge I see in the promo page it kind of implies Edge is a middleware in that you can modify the request.
I see in the nodejs examples that to continue processing of a request you can use callback(null, request); but whenever I use that I get a 502 response.
For example can I add/modify a header and then continue a request to a Cloudformation or API Gateway backend or must the lambda return an object of some sort?
Here is an example (logs show header is added and all is well, just that curl returns 502):
exports.handler = (event, context, callback) => {
console.log(context);
console.log(event);
const request = event;
request.headers.bar = 'foo';
console.log(event);
callback(null, request);
};
One of the issues that sometimes stumps people when trying to use Lambda#Edge is that testing your script successfully in the Lambda console only tests whether your code can run without throwing exceptions.
What it doesn't test for is whether your test event looks like a test event that CloudFront would generate, or whether your returned values would actually be interpreted as valid by CloudFront.
The event you are passed is a complex object containing one record, and inside there is cf (CloudFront) which contains request. If you are a response trigger, it also contains response.
See Lambda#Edge Event Structure. There are test event templates in the Lambda console for various types of CloudFront interactions.
So you need to get the original event's request from the right part of the structure:
const request = event; // incorrect
const request = event.Records[0].cf.request; // correct
Headers are confusing at first, but this is actually an example of very sensible engineering design. HTTP headers are not case sensitive in HTTP/1.x and are always lowercase in http/2, but JavaScript object keys are always case sensitive... so a sensible representation of headers takes all of these factors into account, as well as the fact that some headers can appear more than once and the ordering can be relevant.
In headers, the object key is always lowercase, and each value is an array of objects that contains a key (which must match the outer key, except for lettercase) and a value (the header).
request.headers.bar = 'foo'; // incorrect
request.headers['bar'] = [ { key: 'Bar', value: 'foo' } ]; // correct
Additionally, certain headers are blacklisted -- for reasons of security or simple sanity, you can't add or manipulate them.
See Headers in the Lambda#Edge section of the CloudFront Developer Guide.
Also, remember that CloudFront is a cache, and caches have cache keys -- the unique value that identifies a specific request, so that other, identical requests can be determined to actually be identical and served with the same response. The cache key in CloudFront consists of only what CloudFront is configured to send to the origin -- which does not include headers that you haven't whitelisted for forwarding to the origin in the Cache Behavior settings. Trying to set or modify a header inappropriately will result in a 502 error. In the example above, you would need to whitelist the Bar header for forwarding to the origin, in the cache distribution settings.
You might find it initially easier to learn by trying to modify a response, rather than a request, because they are somewhat more forgiving.
Note that in request triggers, you have essentially four possible outcomes:
leave the request unmodified and return control to CloudFront by using return callback(null, request); without changing anything
modify the request by modifying the request object and then `return callback(null, request);
stop further CloudFront processing, and generate a response directly by building a valid response object and calling return callback(null, response);
throw a hard exception by setting the first callback() response to something other than null
In a viewer request trigger, generating a response returns the response to the viewer without checking the cache and without caching the response.
An origin request trigger only fires after the cache has already been checked, and the object is not there. If you generate a response in an origin request trigger, the response is stored in the cache and returned to the requester. The request is never sent to the origin if you generate a response in this trigger. If you modify the request, it is sent to the origin and the response is cached unless configured not to be cached.
An origin response trigger modifies or replaces the response from the origin, and the modified response is stored on the cache.
A viewer response trigger modifies or replaces the response that was either fetched from the cache or from the origin. The modified response is not cached.
Response triggers are also able to inspect the original request, in cases where this might be desirable.
I have used AWS API Gateway with the endpoint as a lambda function. I have enabled the cache functionality provided by API Gateway, to reduce both response time & the number of calls forwarded to my lambda function.
The lambda function queries another data store to return data. If data is not found, an asynchronous call is made to update the data store and "data not found" is returned to the caller. Now the API Gateway is even caching this result, which we do not want to happen. This results in the cache always returning "data not found" for its lifetime (1 hr TTL), even though data is updated asynchronously in the data store.
I'm aware of the request header (Cache-Control: max-age=0) which can invalidate cache and get response directly from Lambda, as mentioned in this documentation page.
But this will not be useful because the caller is unaware whether data is present in data store or not and hence cannot selectively send such request header.
So, my 2 questions are:
Does the API Gateway caches other HTTP response as well, like 404 (apart from 200)?
Is it possible to tell the API Gateway not to cache specific responses?
As you observed, API Gateway caches the result of your endpoint regardless of the status code.
No, not at this time. I can bounce the idea to see if this is something we can support in the future.
Even if API Gateway conditionally did not cache the 404, the caller would need to call the endpoint again anyway, so why not return the result synchronously? This pattern is how most cached APIs that I'm aware of behave and would allow your API to work with what API Gateway offers today.
The scenario is: I'm implementing a RESTful web-service that will act as a cache to entities stored on remote a C system. One of the web-service's requirements is that, when the remote C system is offline, it would answer GET requests with the last cached data, but flagging it as "stale".
The way I was planning to flag the data as stale was returning a HTTP status code other than 200 (OK). I considered using 503 (service unavailable), but I believe that it would make some C#/Java HTTP clients throw exceptions, and that would indirectly force the users to use exceptions for control flow.
Can you suggest a more appropriate status code? Or should I just return 200 and add a staleness flag to the response body? Another option would be defining a separate resource that informs the connectivity state, and let the clients handle that separately.
Simply set the Last-Modified header appropriately, and let the client decide if it's stale. Stale data will have the Last-Modified date farther back than "normal". For fresh data, keep the Last-Modified header current.
I would return 200 OK and an appropriate application-specific response. No other HTTP status code seems appropriate, because the decision if and how to use the response is being passed to the client. I would also advise against using standard HTTP cache control headers for this purpose. I would use them only to control third-party (intermediary and client) caches. Using these headers to communicate application-specific information uneccesarily ties application logic to cache control. While it might not be immediately obvious, there are real long-term benefits in the ability to independently evolve application logic and caching strategy.
If you are serving stale responses RFC-2616 says:
If a stored response is not "fresh enough" by the most
restrictive freshness requirement of both the client and the
origin server, in carefully considered circumstances the cache
MAY still return the response with the appropriate Warning
header (see section 13.1.5 and 14.46), unless such a response
is prohibited (e.g., by a "no-store" cache-directive, or by a
"no-cache" cache-request-directive; see section 14.9).
In other words, serving 200 OK is perfectly fine.
In Mark Nottingham's caching article he says
Under certain circumstances — for example, when it’s disconnected from
a network — a cache can serve stale responses without checking with
the origin server.
In your case, your web service is behaving like an intermediary cache.
A representation is stale when either it's Expires or Max-age header has passed. Therefore if you returned a representation with
Cache-control: Max-age=0
Then you are effectively saying that the representation you are returning is already stale. Assuming that when you retrieve representations from the "System C" that the data can be considered fresh for some non-zero amount of time, your web service can return representations with something like,
Cache-control: Max-age=3600
The client can check cache control header for max-age == 0 to determine if the representation was stale when it was first retrieved or not.