According to Object Key and Metadata - Amazon Simple Storage Service, Amazon S3 buckets have a flat structure, meaning that an object created in a folder folder1/object1.txt would have the key folder1/object1.txt. However there is a discrepancy between the docs and what the AWS console shows.
When you click on the checkbox next to the object1.txt the properties panel slides in from the right and there is a key property under the overview section that reads key object1.txt. This according to the documentation is incorrect. Additionally if you click on the object link the new overview screen shows a different panel in which they key is folder1/object1.txt.
My Question is: What is the reason for this discrepancy and which panel is displaying the correct information? Is the key represented in the first panel something entirely different than the s3 object key?
The documentation is correct.
However, since humans enjoy the concept of folders and directories, Amazon S3 provides something called a Common Prefix, which is similar to the concept of a path.
When listing the contents of a bucket, paths (effectively keys without the final "object name") are provided are a list of CommonPrefixes. The AWS Management Console uses this to allow users to step through folder hierarchies.
However, the Key of all objects include their full path.
Here's something interesting... if a user clicks "New Folder" in the Amazon S3 management console, then a zero-length file is created with the name of the folder. This causes the folder to appear as a common prefix, even if no files exist "inside" the folder.
Correct object key would obviously be folder1/object1.txt.
As for the reason for the discrepancy I'd sign it off to a poor UI decision.
Related
I would like to create a folder structure displayed like below.
How should I specify the key?
*Top level folder would be today's date
/yyyymmdd/*.jason
e.g.
/2021-05-21/example.json
Folders do not actually exist in Amazon S3. You can create an object with any path and the folder magically 'appears'. Then, if the object is deleted, the folder will disappear. Amazon S3 calls them CommonPrefixes.
The Amazon S3 management console has a "Create Folder" button. This actually creates a zero-length object with the same name as the folder (e.g. 2021-05-21/). This 'forces' the folder to appear because there is an object inside it. However, the zero-length object is hidden in the console.
So, if you wish to 'create a folder', simply create a zero-length object with the name of the folder.
Or, better yet, do not create the folder. Just pretend that it exists and things will work perfectly fine.
I am trying to list all items with specific prefix in S3 bucket. Here is directory structure that I have:
Item1/
Item2/
Item3/
Item4/
image_1.jpg
Item5/
image_1.jpg
image_2.jpg
When I set prefex to be Item1/Item2, I get as a result following keys:
Item1/Item2/
Item1/Item2/Item3/Item4/image_1.jpg
Item1/Item2/Item3/Item5/image_1.jpg
Item1/Item2/Item3/Item5/image_2.jpg
What I would like to get is:
Item1/Item2/
Item1/Item2/Item3
Item1/Item2/Item3/Item4
Item1/Item2/Item3/Item5
Item1/Item2/Item3/Item4/image_1.jpg
Item1/Item2/Item3/Item5/image_1.jpg
Item1/Item2/Item3/Item5/image_2.jpg
Is there anyway to achieve this in golang?
Folders do not actually exist in Amazon S3. It is a flat object storage system.
For example, using the AWS Command-Line Interface (CLI) I could copy a command to an Amazon S3 bucket:
aws s3 cp foo.txt s3://my-bucket/folder1/folder2/foo.txt
This work just fine, even though folder1 and folder2 do not exist. This is because objects are stored with a Key (filename) that includes the full path of the object. So, the above object actually has a Key (filename) of:
folder1/folder2/foo.txt
However, to make things easier for humans, the Amazon S3 management console makes it appear as though there are folders. In S3, these are called Common Prefixes rather than folders.
So, when you make an API call to list the contents of the bucket while specifying a Prefix, it simply says "List all objects whose Key starts with this string".
Your listing doesn't show any folders because they don't actually exist.
Now, just to contradict myself, it actually is possible to create a folder (eg by clicking Create folder in the management console). This actually creates a zero-length object with the same name as the folder. The folder will then appear in listings because it is actually listing the zero-length object rather than the folder.
This is probably why Item1/Item2/ appears in your listing, but Item1/Item2/Item3 does not. Somebody, at some stage, must have "created a folder" called Item1/Item2/, which actually created a zero-length object with that Key.
There seems to be some inconsistency in Amazon S3's behavior.
If in bucket a "Bucket1", I create folder "Folder1" and upload a file say "sample.txt" into it. Next I delete this file. At the bucket level I can see "Folder1" on S3 Console.
Now in the same bucket if I upload a file "Folder2/sample.txt" and just delete sample.txt file, then Folder2 also disappears from console?
Why this inconsistency? AFAIK we do not have any API to create/delete folder at SDK level.
Am I missing something here or is this an actual issue?
Thanks in advance for any help.
A "Folder" in S3 is simply a 0-byte object with a / character at the end of the key name.
So, using the AWS CLI or SDKs, you can "create a folder" by "putting" an object that matches those criteria.
The AWS Management Console also does something extra: it simulates folders, even of they were not explicitly created. So, if you uploaded your object as "Folder2/sample.txt", it extrapolates and simulates "Folder2/" at the parent folder level. You can do this yourself with the CLI/SDKs using the delimiter parameter.
When you delete that object, since "Folder2" did not actually exist as a 0-byte object ending with / (see first paragraph), then "Folder2/" disappears from the management console.
I've been setting up aws lambda functions for S3 events. I want to set up a new structure for my bucket, but it's not possible--so I set up a new bucket the way I want and will migrate old things and send new things there. I wanted to have some of the structure the same under a given base folder name old-bucket/images and new-bucket/images. I set up CloudFront to serve from old-bucket/images now, but I wanted to add new-bucket/images as well. I thought the behavior tab would set it such that it would check the new-bucket/images first then old-bucket/images. Alas, that didn't work. If the object wasn't found in the first, that was the end of the line.
Am I misunderstanding how behaviors work? Has anyone attempted anything like this?
That is expected behavior. An origin tells Amazon CloudFront where to obtain the data to serve to users, based upon a prefix, suffix, etc.
For example, you could serve old-bucket/* from one Amazon S3 bucket, while serving new-bucket/* from a different bucket.
However, there is no capability to 'fall-back' to a different origin if a file is not found.
You could check for the existence of files before serving the link, and then provide a different link depending upon where the files are stored. Otherwise, you'll need to put all of your files in the location that matches the link you are serving.
I have set up a public bucket in S3 and copied multiple objects into it. In this case they are jpeg photos.
I want to share all these objects with anonymous public users (friends), but I want to send them one static website address for the bucket and for the objects to show up as a list (or at least show all the images) when they click on that one address link.
Is this possible to display the objects this way using S3 to public users who don't have an S3 account?
The alternative I know of is to send them a unique link to each of the objects in the bucket (which would take forever!).
Any advice would be helpful.
S3 doesn't have anything built-in to do a "directory index" like nginx and Apache can do. It can be done with AWS Lambda, though.
I built a rudimentary image index with lambda, you might be able to adapt it to solve your problem.
yes.
you can host an static webpage inside a s3 bucket: http://docs.aws.amazon.com/AmazonS3/latest/dev/WebsiteHosting.html
just generate a static html page with links to all the photos, upload it in the bucket, set the bucket to serve as a static webpage and give the link to it.
Or, for the extra lazy :) https://github.com/rgrp/s3-bucket-listing
Thanks for your answers, they helped me to find a really simple solution. On a different forum I found someone has written some script and put it in a link that you just upload straight into your bucket and that puts all the objects into a simple list...... genius!
This is the link:
http://regexp.s3.amazonaws.com/list.html
So for the less techy people (like me) you literally upload that link above into your bucket. Even if you haven't downloaded it onto your PC, just copy and paste it into the upload file path.
When I uploaded it, the file appeared in the S3 bucket as list.html
Make sure the file is readable and you've set the ACL appropriately. And make sure your bucket has a policy that allows anyone to access it.
Your bucket objects(content) are then shown at the url link below.
http://<your bucket name>.s3.amazonaws.com/list.html
Where <your bucket name> is written above, replace that part with just the name of your bucket.
And you should be able to click on that link and see the list of objects in your bucket. Once you get your head around it, it is actually very simple.