When I'm uploading a folder to S3 bucket (drag and drop), I see that it is not considered as an object - I can't get it with its key (GetObjectAsync()), and ListObjectsV2Async() doesn't return it (I'm using the .net sdk).
When I'm creating a folder under a bucket I can get it and it appears in list of bucket objects.
What is the reason for that?
Amazon S3 does not have the concept of a 'Directory' or a 'Folder'.
Instead, the full path of an object is stored in its Key (filename).
For example, an object can be stored in Amazon S3 with a Key of: invoices/2020-09/inv22.txt
This object can be created even if the invoices and 2020-09 directories do not exist. When viewed through the Amazon S3 console, it will appear as though those directories were automatically created, but if the object is deleted, those directories will disappear (because they never existed).
If a user clicks the "Create Folder" button in the Amazon S3 management console, a zero-length object is created with the same name as the folder. This 'forces' the folder to appear even if there are no objects 'inside' the folder. However, it is not actually a folder.
When using the ListObjects command while specifying a Delimiter of /, a list of CommonPrefixes is returned. This is equivalent to what you would normally consider a sub-directory.
Related
In linux to create nested folders, irrespective of the intermediate folders exist or not can be done using the below command.
mkdir -p /home/user/some_non_existing_folder1/some_non_existing_folder2/somefolder
Similar to this i want to create a nested folder structure in S3 and place my files there later
how can i do this using aws cli
Folders do not actually exist in Amazon S3.
For example, if you have an empty bucket you could upload a file to invoices/january.txt and the invoices folder will magically 'appear' without needing to be specifically created.
Then, if you were to delete the invoices/january.txt object, then the invoices folder will magically 'disappear' (because it never actually existed).
This works because the Key (filename) of an Amazon S3 object contains the full path of the object. The above object is not called january.txt -- rather, it is called invoices/january.txt. The Amazon S3 console will make it appear as if the folder exists, but it doesn't.
If you click the Create folder button in the S3 management console, then a zero-length object is created with the name of the folder. This causes the folder to 'appear' because it contains a file, but the file will not appear (well, it does appear, but humans see it as a folder). If this zero-length object is deleted, the folder will 'disappear' (because it never actually existed).
Therefore, if you wish to create a directory hierarchy before uploading files, you could upload zero-length objects with the same names as the folders you want to create. You can use the aws s3 cp command to upload such a file.
Or, just upload the files to where you want them to appear, and the folders will magically appear automatically.
# create bucket
aws s3 mb s3://main_folder
# created nested folder
aws s3api put-object --bucket main_folder --key nested1/nested2/nested3/somefoldertosync
# sync my local folder to s3
aws s3 sync /home/ubuntu/somefoldertosync s3://main_folder/nested1/nested2/nested3/somefoldertosync
currently i am using the above way to carry on with my work
I have been provided with the access and secret key for an Amazon S3 container. No more details were provided other than to drop some files into some specific folder.
I downloaded Amazon CLI and also the Amazon SDK. So far, seems to be no way for me to check the bucket name or list the folders where I'm supposed to drop my files. Every single command seems to require the knowledge of a bucket name.
Trying to list with aws s3 ls gives me the error:
An error occurred (AccessDenied) when calling the ListBuckets operation: Access Denied
Is there a way to list the content of my current location (I'm guessing the credentials I was given are linked directly to a bucket?). I'd like to see at least the folders where I'm supposed to drop my files, but the SDK client for the console app I'm building seems to always require a bucket name.
Was I provided incomplete info or limited rights?
Do you know the bucket name or not? If you don't and you don't have permission to ListAllMyBuckets and GetBucketLocation on * and ListBucket on the bucket in question, then you can't get the bucket name. That's how it is supposed to work. If you know the bucket, then you can run aws s3 s3://bucket-name/ to get objects in the bucket.
Note, that S3 buckets don't have the concept of "folder". It's User interface "sugar" to make it look like folders and files. Internally, it's just the key and the object
Looks like it was just not possible without enhanced rights or with the actual bucketname. I was able to procure both later on from the client and able to complete the task. Thanks for the comments.
I was trying to use the putObject() function of AWS S3 to put a object into a folder in a bucket, I am not able to specify the folder name in the function, I was able to put the object into the bucket but not into the folder. Is there any way in which I can specify the Folder name
There are no separate folder names.
The object key is path + filename, so to upload cat.jpg into images/funny/ you upload the file as images/funny/cat.jpg.
Do not use a leading /.
I have been trying to upload a static website to s3 with the following cli command:
aws s3 sync . s3://my-website-bucket --acl public-read
It successfully uploads every file in the root directory but fails on the nested directories with the following:
An error occurred (InvalidRequest) when calling the ListObjects operation: Missing required header for this request: x-amz-content-sha256
I have found references to this issue on GitHub but no clear instruction of how to solve it.
s3 sync command recursively copies the local folders to folder like s3 objects.
Even though S3 doesn't really support folders, the sync command creates the s3 objects with a format which will have the folder names in their keys.
As reported on the following amazon support thread "forums.aws.amazon.com/thread.jspa?threadID=235135" the issue should be solved by setting the region correctly.
S3 has no concept of directories.
S3 is an object store where each object is identified by a key.
The key might be a string like "dir1/dir2/dir3/test.txt"
AWS graphical user interfaces on top of S3 interpret the "/" characters as a directory separator and present the file list "as is" it was in a directory structure.
However, internally, there is no concept of directory, S3 has a flat namespace.
See http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingMetadata.html for more details.
This is the reason directories are not synced as there is no directories on S3.
Also the feature request is open in https://github.com/aws/aws-cli/issues/912 but has not been added yet.
Let's say i have 2 aws accounts: Account1 and AccountZ
I installed and configured s3cmd to have access to Account1.
I created a bucket in AccountZ and made it publicly read/write
I performed an s3cmd put of a text.txt from Account1 to s3://AccountZ/test.txt
Then, after it uploaded, I tried to copy paste AccountZ/test.txt to a different bucket, and it says that there was an error ("The following objects were not copied due to errors from: <AccountZ folder>"). So, I tried to change the permissions to the file, and it says I dont have permissions to do that.
If "upload" a file using the S3 console into AccountZ target directory, that resulting file IS copy/paste-able. So there seems to be an issue with the uploaded file due to the PUT
If i change the permissions config of s3cmd to be the key/secret of AccountZ, then uploaded file's permissions work just fine and the copy/paste command is successful.
How do I upload/PUT a file to S3 so that I can then copy/paste the resulting file in the S3 console?
When an object is uploaded to S3, the owner of the object is the account that created it. In this case, the owner of the object is Account1, even though the bucket exists in AccountZ. The default permissions on the object only allow it to be modified by the owner of the object (Account1). The only thing that AccountZ will be able to do with the object is delete it.
When you create a bucket policy, that policy will automatically apply to any objects in the bucket that are 'owned' by the same account that owns the bucket. Since AccountZ owns the bucket and Account1 owns the object, the bucket policy of public read/write isn't going to apply here.
Try specifying an ACL (eg 'public-read-write') when the object is uploaded. If you need to modify an object that has already been uploaded, try the PutObjectAcl call from the S3 API using Account1's credentials. (http://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectPUTacl.html)
In a similar strategy to what #ScottWolf proposed, I had to do the following to solve my problem:
The solution was that I had to go add a bucket policy in the source data bucket (Account1) that gave permissions to the target. then i had to re-configure my s3 api to use AccountZ's credentials and then just do a copy from Account1 to AccountZ