I have an application that uses the google bucket to store media, as there are many media, it was necessary to use the CDN service to reduce latency when loading them. However, when I use the CDN, the media is public and accessible to any unauthenticated person. Is there any way to cache the media and at the same time keep it private through an authentication token?
I tried in many ways following the documentation, keeping the Cache Type capturing the information from the Cache-Control header and the authorization token, but after caching the media it is accessible without the authentication token.
Can anybody help me?
Is it possible to include the auth header as part of the cache key? This would require the origin server to have auth functionality.
This would enable the object to be put into cache when the key is present. Also, only a request with the auth header can retrieve the object from cache.
If a user without the key, or an incorrect key, attempts to get the file, it will be cache miss. The origin server should then authenticate the header and thus no allow the object to be delivered.
To add details to the link shared by John Hanley, using Signed URLs will cache the content and at the same time, keep the access private to your stored media. You can follow this documentation for a more detailed procedure.
Related
I have a resource called Sites.
I am planning to have an endpoint as follows:
/tenant/:tenantId/sites/:siteId
The endpoint is to return a site’s tree which will vary based on the userId extracted from the JWT token.
Since it will vary based on the user requesting it, should the endpoint have userId in the URI- may be as a query parameter?
How should caching work in this case?
The sites tree returned by this endpoint will also change based on updates in another resource (i.e users/groups)
Should the cache be discarded for all users whenever there is a change in the sites resource itself or when there is a change in groups?
I am using API gateway so will need to purge cache through client cache control header when any of the resources are updated.
Since, the data will vary on the user requesting it, the endpoint should have the userId in the URI - it could be simply a path parameter similar to the tenantId and siteId
caching can be done on the basis of If-modified-since header to indicate if the data has changed or not.
The If-Modified-Since HTTP header indicates the time for which a browser first downloaded a resource from the server. This helps determine whether or not the resource has changed since the last time it was accessed.
From a security point of view if a user only can access his own sites the user id should not be on the path (or query param), because if you do that, any user can modify the URL in its browser and try to access the other user sites. To avoid that the URL should not have any userId (you can replace it with something like /me) and the service that will handle the request should extract the id information from the token
I don't know if you are using an in-memory cache of distributed cache and if sites/users/groups are different services (deployed on different servers) or if they live in the same application, anyway, when any of the resources that cache depends on are modified, you should invalidate the cache for that users
Currently, Akamai has many upload mechanisms like FTP, RFTP, NetStorage Api and Aspera.
I would like to know whether Akamai NetStorage has a presigned URL feature-- A presigned URL is a URL that you can provide to your users to grant temporary access to a specific Akamai’s directory, Using the URL, a user can either READ the object or WRITE an Object (or update an existing object)
Thanks,
As far as I know NetStorage doesn't have that functionality out of the box, however you could implement that using Edge logic. Basically you create an Akamai Edge Configuration that uses NetStorage as origin and enforces Edge URL authentication.
You can generate the tokens using the App of your choice (i.e. AWS or Azure), and then have the token validated on the Akamai Edge. Feel free to contact Akamai Technical support if you need more guidance.
It depends on your architecture. Netstorage is Read/Writeable - subject to correct authentication headers accompanying your "presigned" URL. Such access to NetStorage is NOT scaleable. So if you just want to give a URL to many users, then you simply need a property configured in Akamai to do what you want.
I am making an app which includes a messaging feature. Through the messaging feature, users can send photos to others users. These photos should be completely private.
At first, I thought of S3's signedURL feature. But then I realized that I cannot make caching work which is done by my CDN provider and my client side because caching is done based on URLs.
So I moved on to CloudFront's signed cookie. It seemed promising at first, but I found another problem. Users who got signed cookies can access to any content in the allowed scope. But I should not allow to show photos that were sent in other chat rooms. Users who have signed cookies should not be able to access to photo urls that were not shared in their rooms. So I cannot use signed cookies.
I moved on to CloudFlare and found a post that they were allowed to use special cache keys instead of url based caching. (https://blog.bigbinary.com/2019/01/29/how-to-cache-all-files-using-cloudflare-worker-along-with-hmac-authentication.html) I do not know how much the Enterprise Plan is, but Business Plan which is one level below is $200/month.
The business plan allows CloudFlare users to use token authentication. (https://blog.cloudflare.com/token-authentication-for-cached-private-content-and-apis/) (https://support.cloudflare.com/hc/en-us/articles/115001376488-How-to-setup-Token-Authentication-) I might be able to utilize this token authentication by making my images including tokens like this:
<Image source={{
uri: 'https:image_url.jpeg',
method: 'GET',
headers: {
Authorization: token
},
}}
style={{width: width, height: height}}
/>
Another thing I could do is getting signed URLs from CloudFront, not from a S3 level. In that way, I can make my CDN(CloudFront, in this case) to properly cache my S3 images and then make unique URLs per photo. But I still have to deal with client side caching as URLs clients see are always different. I have to save URLs in Localstorage as this(https://stackoverflow.com/a/37817503) answer suggested. Or I can use a React Native caching library. However, I will deploy this app on the web as well as mobile environment, so I am not sure if it will be a viable option for me to use such caching libraries.
To sum up, signed URLs cause two-level problems. It does not work with CDN caching. It does not work with client caching. I should use CloudFront's signed URLs and deal with client side caching(which is not ideal) Or I should use CloudFlare's token method. Bandwidth is free for CloudFlare, though Business Plan costs $200. So will it be worth it if I assume my app scales well?
What discourages me from using CloudFlare is it is not well documented. I have to deal with workers in CloudFlare, but the only document I found about how to use signed URL in the CDN level is this (https://developers.cloudflare.com/workers/about/tips/signing-requests/#verifying-signed-requests) And the only one I found about how to access to S3 private bucket from CloudFlare is this (https://help.backblaze.com/hc/en-us/articles/360010017893-How-to-allow-Cloudflare-to-fetch-content-from-a-Backblaze-B2-private-bucket)
Is CloudFlare with token verification method the right way to go for me? Is there any other method I can try out?
If I set up my app to generate pre-signed URLs for access to S3 media (so that I can set the files to be private, unless accessed via a logged in user) then would I be right in saying that, if someone has access to the URL (within the expiry time) they can see the file, despite it being "private"?
So if someone was to send the URL to someone else, then it's not really private any more.
I guess there's no other way but this just seems odd to me.
Yes, you are correct that a signed URL can be "shared" because it is valid until it expires (or until the credentials that signed it expire or are otherwise invalidated, whichever comes first).
One common solution is for your application to generate signed URLs as the page is being rendered, using very short expiration times.
Another is for the link to the secured content to actually be a link back to the application, which verifies the user's authority to access the object, and then returns an HTTP redirect to a freshly-generated signed URL with a short expiration time (e.g. 5 seconds).
HTTP/1.1 302 Found
Location: https://example-bucket.s3.amazonaws.com/...?X-Amz-...
Signed URLs cannot be tampered with using currently feasible compute capabilities, so it is impractical to the point of impossibility for a signed URL to be modified by a malicious user.
Note also that a signed URL (for either S3 or CloudFront) only needs to be not-yet-expired when the download starts. The time required for the download to actually finish can be arbitrarily long, and the download will not be interrupted.
There is no ready-made service for the following option, but using a combination of CloudFront Lambda#Edge triggers and DynamoDB, it is possible to create a genuinely single-use URL, which consists of a randomly generated "token" stored in the Dynamo table and associated with the target object. When the URL is accessed, you use a DynamoDB conditional update in the Lambda trigger to update the (e.g.) "view_count" value from 0 to 1. If the token isn't in the table or the view count isn't 0, the conditional update fails, so access is denied; otherwise CloudFront allows the request to proceed -- exactly once. CloudFront accesses the S3 content using an Origin Access Identity, which all happens behind the scenes, so nothing related to the actual authentication of the request between CloudFront and S3 is accessible to the user. (For cryptographic-quality random token generation, you can also use KMS's GenerateRandom API action.)
There are a number of alternative approaches, including other uses of Lambda#Edge triggers to do things like inspect a request for an application-provided cookie and then querying the application server to authenticate the user.
CloudFront also supports signed cookies that it parses and interprets, itself, but these provide wildcard-based access to all your assets matching a specific URL and path (e.g. /images/*) and there is nothing to prevent a user from sharing their cookies, so these are probably not useful for your use case.
CloudFront signed URLs do support the option of allowing access only if the signed URL is used from a specific source (client) IP address, but this has potential problems in there is no assurance that a 1:1 correlation exists between users and IP addresses. Many users can be behind the same address (particularly in corporate network environments) or a single user's address can change at any moment.
The complexity of the possible implementations varies wildly, and what you need depends in part on how secure you need for your content to be. In many cases, more extreme solutions accomplish little more than discouraging honest users, because the user can still download the resource and share it via other means.
That would still be a separate user requesting content. For a separate user, the certificate would not longer be valid.
Source: https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/private-content-signed-urls.html
The main security concern in direct js browser uploads to S3 is that users will store their S3 credentials on the client side.
To mitigate this risk, the S3 documentation recommends using a short lived keys generated by an intermediate server:
A file is selected for upload by the user in their web browser.
The user’s browser makes a request to your server, which produces a temporary signature with which to sign the upload request.
The temporary signed request is returned to the browser in JSON format.
The browser then uploads the file directly to Amazon S3 using the signed request supplied by your server.
The problem with this flow is that I don't see how it helps in the case of public uploads.
Suppose my upload page is publicly available. That means the server API endpoint that generates the short lived key needs to be public as well. A malicious user could then just find the address of the api endpoint and hit it everytime they want to upload something. The server has no way of knowing if the request came from a real user on the upload page or from any other place.
Yeah, I could check the domain on the request coming in to the api, and validate it, but domain can be easily spoofed (when the request is not coming from a browser client).
Is this whole thing even a concern ? The main risk is someone abusing my S3 account and uploading stuff to it. Are there other concerns that I need to know about ? Can this be mitigated somehow?
Suppose my upload page is publicly available. That means the server
API endpoint that generates the short lived key needs to be public as
well. A malicious user could then just find the address of the api
endpoint and hit it everytime they want to upload something. The
server has no way of knowing if the request came from a real user on
the upload page or from any other place.
If that concerns you, you would require your users to login to your website somehow, and serve the API endpoint behind the same server-side authentication service that handles your login process. Then only authenticated users would be able to upload files.
You might also want to look into S3 pre-signed URLs.