I have some images stored in Amazon S3 bucket. I am pointing to them in my web pages using <img src>.
I don't want the images to be viewable when a user is not logged in. Can I make it private from my backend when a user logs out?
Use Presigned Object URLs to get the images from a private S3 bucket. These presigned URLs are valid only for a specified duration. Here is a basic example of how to generate these presigned URLs using boto3 in Python:
import boto3
AWS_ACCESS_KEY_ID = <access_key_id>
AWS_SECRET_KEY = <secret_key>
AWS_REGION = <region_name>
client = boto3.client(
's3',
aws_access_key_id = AWS_ACCESS_KEY_ID,
aws_secret_access_key = AWS_SECRET_KEY,
region_name = AWS_REGION
)
PRESIGNED_DOWNLOAD_URL = client.generate_presigned_url(
ClientMethod = 'get_object',
Params = {
'Bucket': AWS_BUCKET_NAME,
'Key': FILE_NAME,
},
ExpiresIn = 3600,
)
print(PRESIGNED_DOWNLOAD_URL)
Here, you just have to provide AWS_ACCESS_KEY_ID, AWS_SECRET_KEY, AWS_REGION of the bucket, AWS_BUCKET_NAME and the FILE_NAME you want to download. ExpiresIn=3600 is given in seconds. So, this URL will be valid for 60 minutes.
Rest of the part has to be handled by your application. That is when a user logs in, generate presigned URLs and get images from the URL in your template. Otherwise, don't render the images in your template.
You should:
Keep the objects in Amazon S3 private (not publicly accessible)
Have your application generate Pre-Signed URLs
A Pre-Signed URL grants time-limited access to a private object in S3. Once the time period expires, it is no longer accessible.
Your application is responsible for authenticating users and determining whether they are permitted to access an object. If so, the application should generate the Pre-Signed URL. The URL can then be included in HTML the same as a normal URL (eg in an <img> tag).
Related
I am struggling to read .m3u8 file in Javascript and modify its segments to get signed by Cloudfront for streaming hls content.
const s3ObjectKey = `${folderName}/${folderName}.m3u8`;
const url = `${process.env.CLOUDFRONT_DOMAIN}/${s3ObjectKey}`;
const privateKey = fs.readFileSync(
new URL("../private_key.pem", import.meta.url),
{
encoding: "utf8",
}
);
const keyPairId = process.env.CLOUDFRONT_KEY_PAIR_ID;
const dateLessThan = new Date(new Date().getTime() + 60 * 60000);
const m3u8Url = cloudfrontSignedUrl({
url,
keyPairId,
dateLessThan,
privateKey,
});
After I get the signed m3u8Url I need to modify its segments to be signed.
Any help will be appreciated.
Answer form AWS Blog post
import boto3
import os
KEY_PREFIX = os.environ.get('KEY_PREFIX')
S3_BUCKET = os.environ.get('S3_BUCKET')
SEGMENT_FILE_EXT = os.environ.get('SEGMENT_FILE_EXT', '.ts')
required_vars = [KEY_PREFIX, S3_BUCKET]
if not all(required_vars):
raise KeyError(f'Missing required environment variable/s. Required vars {required_vars}.')
s3 = boto3.client('s3')
def lambda_handler(event, context):
try:
s3_key = event['pathParameters']['proxy']
obj = s3.get_object(Bucket=S3_BUCKET, Key=s3_key)
body = obj['Body'].read().decode('utf-8')
qp = event['queryStringParameters']
params = ['?']
# reconstruct query param uri
[(params.append(p.replace(KEY_PREFIX, '') + '=' + qp[p] + "&")) for p in qp if KEY_PREFIX in p]
sign_params = ''.join(params).rstrip("&")
# append query params to each segment
resp_body = body.replace(SEGMENT_FILE_EXT, ''.join([SEGMENT_FILE_EXT, sign_params]))
return {
'statusCode': 200,
'body': resp_body
}
except Exception as e:
print(e)
return {'statusCode': 500, 'body': ''}
Let’s go over the key areas in the python code:
KEY_PREFIX environment variable is the prefix pattern that Lambda uses to identify the CloudFront query params. In my example it is -PREFIX.
S3_BUCKET environment variable the S3 bucket name to which Lambda will make the request to get the main manifest.
Note: For illustration purposes I used an environment variable that
is set, for instance, when the Lambda function is created. You can
change this part to have a lookup logic, which is especially helpful
if you have different S3 buckets that are used to store media
content.
SEGMENT_FILE_EXT environment variable file extension of your media file. It default’s to .ts, you can override the value by setting this environment variable in Lambda function configuration.
s3_key variable is the path to the file in your S3 bucket which is represented as a URL that is used by the client to make the request. In my example the URL is https://myapp.com/movies/movie_1/index.m3u8 and the s3_key is movies/movie_1/index.m3u8, which matches exactly with the s3 bucket folder structure as shown earlier in Figure 5. By following this convention, when creating a S3 folder structure for the URL path, s3_key will dynamically get the correct path to the file in your S3 bucket. To learn more about Lambda proxy integration, see Set up a proxy integration with a proxy resource.
sign_params variable is the reconstructed CloudFront signed URL query parameters.
resp_body variable, is the final modified main manifest that is returned to the client. The replace function appends the CloudFront signed URL query parameters to each segment in the manifest file. The final result is assigned to the resp_body variable.
This Lambda function is getting the manifest file in its original form from your S3 bucket and modifying it so that when the playback player makes a request to get the next segment, the request already includes the CloudFront signed URL query parameters. This allows you to restrict access to the video content based on user permissions for each video. Figure 6 illustrates manifest before and after the modification.
i'm using cloudflare cd to serve my website
all my static files and downloads on gcs and i'm using GCS pytho library for generating signed url v4 i case users want to download some files from my website
the problem is when i use this function generate_download_signed_url_v4 from google
it give me back the signed url with link statring with https://storage.googleapis.com/my_bucket/........
i want to change this link with my own sub domain ex. download.doamin.com
i figured out that i have to use bucket_bound_hostname
but when i use it and try to download with given url i get this message
<Error>
<Code>NoSuchKey</Code>
<Message>The specified key does not exist.</Message>
</Error>
and this is the fuction i use
def generate_download_signed_url_v4(bucket_name, blob_name):
storage_client = storage.Client()
bucket = storage_client.bucket(bucket_name)
blob = bucket.blob(blob_name)
url = blob.generate_signed_url(
version="v4",
# This URL is valid for ...
expiration=datetime.timedelta(minutes=10),
# Allow GET requests using this URL.
method="GET",
bucket_bound_hostname="mysub.domain.com",
scheme='https'
)
return url
PS. i've added Cname in dns setting for c.storage.googleapis.com
I am currently creating a basic app with React-Native (frontend) and Flask/MongoDB (backend). I am planning on using AWS S3 as cheap cloud storage for all the images and videos that are going to be uploaded and viewed. My current idea (and this could be totally off), is when a user uploads content, it will go through my Flask API and then to the S3 storage. When a user wants to view content, I am not sure what the plan of attack is here. Should I use my Flask API as a proxy, or is there a way to simply send a link to the content directly on S3 (which would avoid the extra traffic through my API)?
I am quite new to using AWS and if there is already a post discussing this topic, please let me know, and I'd be more than happy to take down this duplicate. I just can't seem to find anything.
Should I use my Flask API as a proxy, or is there a way to simply send a link to the content directly on S3 (which would avoid the extra traffic through my API)?
If the content is public, you just provide an URL which points directly to the file on the S3 bucket.
If the content is private, you generate presigned url on your backend for the file for which you want to give access. This URL should be valid for a short amount of time (for example: 15/30 minutes). You can regenerate it, if it becomes unavailable.
Moreover, you can generate a presigned URL which can be used for uploads directly from the front-end to the S3 bucket. This might be an option if you don't want the upload traffic to go through the backend or you want faster uploads.
There is an API boto3, try to use it.
It is not so difficult, I have done something similar, will post code here.
I have done like #Ervin said.
frontend asks backend to generate credentials
backend sends to frontend the credentials
Frontend upload file to S3
Frontend warns backend it has done.
Backend validate if everything is ok.
Backend will create a link to download, you have a lot of security options.
example of item 6) To generate a presigned url to download content.
bucket = app.config.get('BOTO3_BUCKET', None)
client = boto_flask.clients.get('s3')
params = {}
params['Bucket'] = bucket
params['Key'] = attachment_model.s3_filename
params['ResponseContentDisposition'] = 'attachment; filename={0}'.format(attachment_model.filename)
if attachment_model.mimetype is not None:
params['ResponseContentType'] = attachment_model.mimetype
url = client.generate_presigned_url('get_object', ExpiresIn=3600, Params=params)
example of item 2) Backend will create presigned credentials to post your file on S3, send s3_credentials to frontend
acl_permission = 'private' if private_attachment else 'public-read'
condition = [{'acl': acl_permission},
["starts-with", "$key", '{0}/'.format(folder_name)],
{'Content-Type': mimetype }]
bucket = app.config.get('BOTO3_BUCKET', None)
fields = {"acl": acl_permission, 'Bucket': bucket, 'Content-Type': mimetype}
client = boto_flask.clients.get('s3')
s3_credentials = client.generate_presigned_post(bucket, s3_filename, Fields=fields, Conditions=condition, ExpiresIn=3600)
example of item 5) Here are an example how backend can check if file on S3 are ok.
bucket = app.config.get('BOTO3_BUCKET', None)
client = boto_flask.clients.get('s3')
response = client.head_object(Bucket=bucket, Key=s3_filename)
if response is None:
return None, None
md5 = response.get('ETag').replace('"', '')
size = response.get('ContentLength')
Here are an example how frontend will ask for credentials, upload file to S3 and inform backend it is done.
I tried to remove a lot of particular code.
//frontend asking backend to create credentials, frontend will send some file metadata
AttachmentService.createPostUrl(payload).then((responseCredentials) => {
let form = new FormData();
Object.keys(responseCredentials.s3.fields).forEach(key => {
form.append(key, responseCredentials.s3.fields[key]);
});
form.append("file", file);
let payload = {
data: form,
url: responseCredentials.s3.url
}
//Frontend will send file to S3
axios.post(payload.url, payload.data).then((res) => {
return Promise.resolve(true);
}).then((result) => {
//when it is done, frontend informs backend
AttachmentService.uploadSuccess(...).then((refreshCase) => {
//Success
});
});
});
Got a bucket called www.foo.site. In that site there's a landing page, an about page and some pages in a few bar/* folders. Each bar/* has an index.html page: bar/a/index.html, bar/b/index.html etc.
The landing page is running fine (meaning www.foo.site will load when I browse to it) but the /about/index.html page and /bar/index.html pages aren't getting served when I click on my about links etc. If I curl the URLs I get 404. I've tried setting the origin path and origin domain name separately:
First try:
domain name: www.foo.site.s3.amazonaws.com
origin path: (blank)
Second try:
domain name: s3-us-west-1.amazonaws.com
origin path: www.foo.site
Default document is index.html for both.
Neither one worked. All of the S3 pages mentioned above are directly browsable. Meaning https://s3-us-west-1.amazonaws.com/www.foo.site/bar/index.html loads the expected html.
This must be some Cloudfront setting I'm missing. Possibly something in my DNS records? Is it possible to serve html files in S3 "folders" via Cloudfront?
Here are a couple of resources that are helpful when serving index.html from S3 implicitly via https://domain/folder/ rather than having to explicitly use https://domain/folder/index.html:
Serving index pages from a non-root location via CloudFront
Serving index pages from a non-root location via CloudFront (now unavailable)
Implementing Default Indexes in CloudFront Origins Using Lambda#Edge
The key thing when configuring your CloudFront distribution is:
do not configure a default root object for your CloudFront distribution
If you configure index.html as the default root object then https://domain/ will correctly serve https://domain/index.html but no subfolder reference such as https://domain/folder/ will work.
It's also important to not use the dropdown in Cloudfront when connecting the CF distribution to the S3 bucket. You need to use the URL for the S3 static site instead.
CloudFront serves S3 files with keys ending by /
After investigation, it appears that one can create this type of files in S3 programatically. Therefore, I wrote a small lambda that is triggered when a file is created on S3, with a suffix index.html or index.htm
What it does is copying the object dir/subdir/index.html into the object dir/subdir/
import json
import boto3
s3_client = boto3.client("s3")
def lambda_handler(event, context):
for f in event['Records']:
bucket_name = f['s3']['bucket']['name']
key_name = f['s3']['object']['key']
source_object = {'Bucket': bucket_name, 'Key': key_name}
file_key_name = False
if key_name[-10:].lower() == "index.html" and key_name.lower() != "index.html":
file_key_name = key_name[0:-10]
elif key_name[-9:].lower() == "index.htm" and key_name.lower() != "index.htm":
file_key_name = key_name[0:-9]
if file_key_name:
s3_client.copy_object(CopySource=source_object, Bucket=bucket_name, Key=file_key_name)
Im using boto's S3 Key.generate_url to get a expired signed token. This works as expected, except when used with cross-origin requests (CORS). Since my bucket is not in the default region it will get a 307 redirect and this is making the OPTIONS request redirect and fail. If I change the hostname of the generated url to my bucket's location ie xxbucket-namexx.s3-us-west-1.amazonaws.com the CORS request works fine.
Any ideas of how to get this to always generate the canonical hostname for these generated URLs?
Here is how I'm generating the URL.
conn = boto.connect_s3()
bucket = conn.get_bucket('bucket-name')
key = Key(bucket)
key.key = 'image-name'
signed_url = key.generate_url(
60*5, method='PUT', policy='public-read',
force_http=True,
headers={'Content-Type': self.request.get('contentType'),
'Content-Length': self.request.get('contentLength')},
)