When I use the method generate_presigned_url -- the resulting url contains my bucket name and path to the object and I am quite paranoid of exposing them!
Is there a way to generate a temp url that will not have the path and bucket name ?
The short answer to my question is No!
Discussion with boto: https://github.com/boto/boto3/issues/1591
Apparently the work around I followed is that move the images to a new randomly named folder and generate the url! After URL expiry, delete the randomly named folder. Downside -- Time in copying the images. Space is not an issue if the url expiry is in seconds.
Related
I've been looking through the presigned URL documentation, and have not encountered much information about how often I can generate these presigned URLs. Judging by the length of the identifier, I'm thinking it's probably collision-safe to generate a new one for every URL request.
I think a more conventional method is to run a cron job to generate a new one and store it in a record DB with the file key, and this would be perfectly doable, but I was wondering if I could skip this step and just generate it on the fly.
There is no way to have a presigned url without expiry time and the maximum expiry time you can set is 1 week. These URLs are designed to be temporary to allow users to access your S3 bucket, either for READ the object or WRITE an Object (or update an existing object).
So as an answer, it is not a bad practice to generate a new presigned URL per request since its nature to be temporary.
I am trying to replace an object(video) in google cloud bucket after doing certain operations over it giving it the same name. Giving it same name because it's already shared to multiple users. While doing an operation over it and while replacing it, some chunks of video becomes temporarily unavailable for people who are playing that video at that time and they face issue for a few seconds because of this.
So I have a question that whether its possible to replace the object in-place without affecting the existing version loaded in some places. Also to add I have CDN above this bucket too. Can object versioning on this bucket help me here? I want to keep the name same so that I dont have to send this link again to everyone
I had a similar situation. When I called support, they had me name the new file EXACTLY the same as the original file. Delete the original file from your bucket. Upload the new file that has the exact same name, and the URL will be the same as the original URL.
I have some functionality that uploads Documents to an S3 Bucket.
The key names are programmatically generated via some proprietary logic for the layout/naming convention needed.
The results of my S3 upload command is the actual url itself. So, it's in the format of
REGION/BUCKET/KEY
I was planning on storing that full url into my DB so that users can access their uploads.
Given that REGION and BUCKET probably wouldn't change, does it make sense to just store the KEY - and then dynamically generate the full url when the client needs it?
Just want to know what the desired pattern here is and what others do. Thanks!
Storing the full URL is a bad idea. As you said in the question, the region and bucket are already known, so storing the full URL is a waste of disk space. Also, if in the future say, you want to migrate your assets to a different bucket may be in a different region, having full URLs stored in the DB just make things harder.
My S3 bucket hosts a static website. I do not have cloudfront set up.
I recently updated the files in my S3 bucket. While the files got updated, I confirmed manually in the bucket. It still serves an older version of the files. Is there some sort of caching or versioning that happens on Static websites hosted on S3?
I haven't been able to find any solution on SO so far. Note: Cloudfront is NOT enabled.
Is there some sort of caching or versioning that happens on Static websites hosted on S3?
Amazon S3 buckets provide read-after-write consistency for PUTS of new objects and eventual consistency for overwrite PUTS and DELETES
what does this mean ?
If you create a new object in s3, you will be able to immediately access your object - however in case you do an update of an existing object, you will 'eventually' get the newest version of you object from s3, so s3 might still deliver you the previous version of the object.
I believe that starting some time ago, read-after-write consistency is also available for update in the US Standard region.
how much do you need to wait ? well it depends, Amazon does not provide much information about this.
what you can do ? no much. If you want to make sure you do not have any issue with your S3 bucket delivering the files, upload a new file in your bucket, you will be able to access it immediately
Solution is here:
But you need to use CloundFront. like #Frederic Henri said, you cannot do much in S3 bucket itself, but with CloudFront, you can invalidate it.
CloudFront will have cached that file on an edge location for 24 hours which is the default TTL (time to live), and will continue to return that file for 24 hours. Then after the 24 hours are over, and a request is made for that file, CloudFront will check the origin and see if the file has been updated in the S3 bucket. If is has been updated, CloudFront will then serve the new updated version of the object. If it has not been updated, then CloudFront will continue to serve the original version of the object.
However where you update the file in the origin and wish for it to be served immediately via your website, then what needs to be done is a CloudFront invalidation. An invalidation wipes the file(s) from the CloudFront cache, so when a request is made to CloudFront, it will see that there are no files on the cache, will then check the origin and will serve the new updated file in the origin. Running an invalidation is recommended each time files are updated in the origin.
To run an invalidation:
click on the following link for CloudFront console
-- https://console.aws.amazon.com/cloudfront/home?region=eu-west-1#
open the distribution in question
click on the 'Invalidations' tab to the right of all the tabs
click on 'Create Invalidation'
on the popup, it will ask for the path. You can enter /* to invalidate every object from the cache, or enter the exact path tot he file, such as /images/picture.jpg
finally click on 'Invalidate'
this typically will be completed within 2/3 minutes
then once the invalidation is complete, when you request the object again through CloudFront, CloudFront will check the origin and return the updated file.
It sounds like Akshay tried uploading with a new filename and it worked.
I just tried the same (I was having the same problem), and it resolved the file not being available for me.
Do a push of index.html
index.html not updated
mv index.html index-new.html
Do a push of new-index.htlml
After this, index-html was immediately available.
That's kind of shite - I can't share one link to my website if I want to be sure that the recipient will see the latest version? I need to keep changing the filename and re-sharing the new link.
I have set up a public bucket in S3 and copied multiple objects into it. In this case they are jpeg photos.
I want to share all these objects with anonymous public users (friends), but I want to send them one static website address for the bucket and for the objects to show up as a list (or at least show all the images) when they click on that one address link.
Is this possible to display the objects this way using S3 to public users who don't have an S3 account?
The alternative I know of is to send them a unique link to each of the objects in the bucket (which would take forever!).
Any advice would be helpful.
S3 doesn't have anything built-in to do a "directory index" like nginx and Apache can do. It can be done with AWS Lambda, though.
I built a rudimentary image index with lambda, you might be able to adapt it to solve your problem.
yes.
you can host an static webpage inside a s3 bucket: http://docs.aws.amazon.com/AmazonS3/latest/dev/WebsiteHosting.html
just generate a static html page with links to all the photos, upload it in the bucket, set the bucket to serve as a static webpage and give the link to it.
Or, for the extra lazy :) https://github.com/rgrp/s3-bucket-listing
Thanks for your answers, they helped me to find a really simple solution. On a different forum I found someone has written some script and put it in a link that you just upload straight into your bucket and that puts all the objects into a simple list...... genius!
This is the link:
http://regexp.s3.amazonaws.com/list.html
So for the less techy people (like me) you literally upload that link above into your bucket. Even if you haven't downloaded it onto your PC, just copy and paste it into the upload file path.
When I uploaded it, the file appeared in the S3 bucket as list.html
Make sure the file is readable and you've set the ACL appropriately. And make sure your bucket has a policy that allows anyone to access it.
Your bucket objects(content) are then shown at the url link below.
http://<your bucket name>.s3.amazonaws.com/list.html
Where <your bucket name> is written above, replace that part with just the name of your bucket.
And you should be able to click on that link and see the list of objects in your bucket. Once you get your head around it, it is actually very simple.