I'm using the web interface of Amazon's S3, and when I right-click a folder X and choose Delete, X seems to be deleted. Then when I refresh the interface, X may either disappear or remain; if I keep clicking refresh, the folder is either missing or present. Is this a bug, or am I doing something wrong? The folder is still present, as far as I can tell; one of my EMR jobs complains that the output folder X still exists.
I had the same problem in the AWS web interface after AWS Command Line (CLI)-deleting a "recursive" folder in a bucket. Some objects randomly reappeared (not files, but in fact "folders") in the web interface. Even though i tried to delete these folders in the web interface, they were still there (The interface said the operation was successful...)
Solution that worked for me in the AWS web interface: Right clicked the folder -> CUT, and PASTE into another folder. Worked great, and then deleted the new folder. Gone!
Tried the same as Kristoffer's answer, but CUT/PASTE to another folder made the new folder to not be deletable.
Further hacking: create a new temporary bucket. CUT/PASTE the folder to this bucket and delete the bucket.
S3 does not actually use folders. Instead the path separators in object paths are treated like folders. If you want to remove a folder, all the contents of the folder will have to get deleted.
If there is any delay in deleting all of the contents, the folder may continue to exist.
As of March 2017 the AWS Console UI has changed and you can no longer enter a 'versioning mode' described in my old post.
It seems now folder with versioned files can be deleted freely without restriction.
If this is not the case please drop a comment so I can correct this post.
Previous Version of AWS Console
If you are using the AWS Management Console and you have versioning turned ON, you must be in 'versioning mode' to delete the folder.
You enter 'versioning mode' by going to the top of the page and next to where it says 'Versions:' select the Show button. Then you can proceed to right-click and delete your folder.
Hope this helps someone.
I encountered this issue when I was unable to delete an empty folder from an S3 bucket that had Versioning enabled.
I was able to delete the empty folder by using the "empty bucket configuration" from the S3 Buckets listing:
Select the bucket you'd like to empty, and click the Delete button:
AWS warns you that the bucket isn't empty, and offers a link to use the empty bucket configuration. Click the link:
Proceed through this screen by typing permanently delete to delete all the objects in this bucket:
You should then be able to verify that your S3 bucket is truly empty.
Tried various alternatives to delete from Web interface to delete a folder with sub folders in it without luck.
I had an installation of S3 browser and then tried from S3 Browser interface, worked.
I think I'm seeing similar behavior. My bucket has versioning turned on; even with an empty folder/directory within the bucket, attempting to "delete" the folder/directory within the bucket via the AWS web UI console does not result in it actually being removed. I presume the "deleted" versions of the files within that path still exist (but are not visible in the web console), therefore the bucket isn't truly empty, and isn't truly getting deleted. You may need to check via the CLI tools if existing deleted versions of files in that folder/directory exist (but are not visible in the web console) and delete the files permanently, then attempt to remove the folder/directory in your bucket.
I have the same problem that I cant delete a s3 bucket, with the message "An error occurred (AccessDenied) when calling the DeleteBucket operation: Access Denied"
After a while, I delete the bucket policy in tab "permission" button "bucket policy" and It worked like a charm, with:
aws s3 rb s3://elasticbeanstalk-us-west-..../ --force
I hope this help! Is another option
Pablo
had an "elastic-bean-stalk" bucket and had to delete "bucket policy" before it would delete.
pitney
I had the same problem and didn't have access to the amazon console but I could delete it with this Java code
AmazonS3Client amazonS3Client = new AmazonS3Client(basicAWSCredentials);
ObjectListing objectListing = amazonS3Client.listObjects("bucketName", "prefix");
DeleteObjectsRequest deleteObjectsRequest = new DeleteObjectsRequest("bucketName");
List<DeleteObjectsRequest.KeyVersion> keysToDelete = new ArrayList<>();
objectListing.getObjectSummaries().forEach(s3ObjectSummary -> {
keysToDelete.add(new DeleteObjectsRequest.KeyVersion(s3ObjectSummary.getKey()));
});
deleteObjectsRequest.setKeys(keysToDelete);
amazonS3Client.deleteObjects(deleteObjectsRequest);
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-java-sdk</artifactId>
<version>1.7.4</version>
</dependency>
Try delete with another account, like administrator account. For me it works only with this method.
If you're having trouble fully deleting an empty folder in an S3 bucket that has versioning turned on (i.e. removing all trace of the folder, including in 'Show versions' mode), you can usually get around it by deleting the folder's delete marker(s) using the API/CLI:
$ aws s3api list-object-versions --bucket YOUR-BUCKET --prefix PATH-TO-YOUR-FOLDER
{
"DeleteMarkers": [
{
"Owner": {
"DisplayName": "YOUR-ACCOUNT-NAME",
"ID": "YOUR-ACCOUNT-CANONICAL-ID"
},
"Key": "PATH-TO-YOUR-FOLDER/",
"VersionId": "UNIQUE-VERSION-ID",
"IsLatest": true,
"LastModified": "2022-12-09T07:18:57+00:00"
}
]
}
$ aws s3api delete-objects --bucket YOUR-BUCKET --delete 'Objects=[{Key=PATH-TO-YOUR-FOLDER/,VersionId=UNIQUE-VERSION-ID}]'
{
"Deleted": [
{
"Key": "PATH-TO-YOUR-FOLDER/",
"VersionId": "UNIQUE-VERSION-ID",
"DeleteMarker": true,
"DeleteMarkerVersionId": "UNIQUE-VERSION-ID"
}
]
}
Try use the new S3 console. The delete feature works for folders.
Related
So, I have the following structure on S3:
mainbucket
DataFeeds/
Statement/
We had incidents where the DataFeeds/ folder was being deleted! So, I tested with following:
aws s3api put-object --bucket mainbucket --key DataFeeds/.donotdelete
But, if I execute this (deleting blah.txt even if it does not exists) the DataFeeds/ folder gets deleted too:
aws s3 rm s3://mainbucket/DataFeeds/blah.txt
So, how do we prevent a folder from being deleted on S3?
Versions used:
aws-cli/2.2.46 Python/3.9.7 Darwin/20.6.0 source/x86_64 prompt/off
Folders do not exist in Amazon S3. It is a 'flat' storage service where the Key (filename) of an object includes the full path, including directories.
Amazon S3 will 'simulate' folders for you. For example, if you upload a file to invoices/january.txt, then the invoices directory will 'magically' appear. If you then delete that object, the directory will then 'disappear'. This is because it never actually existed.
If you use the Create folder button in the S3 management console, it will create a zero-length object with the same name as the directory. This will 'force' the directory to appear in the bucket listing. Deleting the zero-length object will cause the directory to disappear if there are no objects with that same Prefix (path).
The best advice for using S3 is to pretend that folders exist. You can place an object in any path and the (pretend) directories will magically appear. Do not worry about directories disappearing, since they never actually existed!
If you really need empty directories to stay around, use that Create folder button to create the zero-length object. It will stay around until you delete the zero-length object.
After deleting a few folders in our S3 bucket, I am not able to see any of my folders through the web console. We had around 10 folders and ended up deleting 6 of them. The remaining four show up when I do an 'ls' on that S3 bucket through the CLI but the bucket shows up empty on the web console.
When I turn on 'Versions' I see everything (including the 6 folders that were deleted). Am I overlooking something extremely simple?
Folders do not actually exist in Amazon S3.
For example, you could create an object like this:
aws s3 cp foo.txt s3://my-bucket/folder1/folder2/foo.txt
This would instantly 'create' folder1 and folder2. Or, to be more accurate, the folders would 'appear' but they don't actually exist because the full filename (Key) of the object is folder1/folder2/foo.txt.
If you were then to delete that object, the folders would 'disappear' because they never actually existed.
Sometimes, if a system wants to forcefully make a folder 'appear', it can create a zero-length object with the same name as the folder. This makes the folder 'appear', but it is really the empty file that is appearing.
Bottom line: Don't worry about creating and deleting folders. They will appear when necessary and disappear when not being used. Do not try to map normal filesystem behaviour to Amazon S3.
I just want to update my index.html file in an S3 bucket. I was going to delete the file and then add the new one just using the website. I can't see any way to do this. Am I thinking wrong? Am I supposed to do all this stuff in code? I just want to hack it together for now so can I do it on the website?
In the console, you can select your file, click "More" and delete is one of the options. There's the upload button to the left as well. I also suggest getting familiar with the AWS CLI so that you also can sync directories and such.
You can go into S3 select a single file and delete it. Afterwards you can upload a new file.
Old console: Select the file in S3 and click Actions and then click delete.
New console: Select the file in S3 and click More and than click delete.
You could use a client like Cyberduck as well.
There seems to be some inconsistency in Amazon S3's behavior.
If in bucket a "Bucket1", I create folder "Folder1" and upload a file say "sample.txt" into it. Next I delete this file. At the bucket level I can see "Folder1" on S3 Console.
Now in the same bucket if I upload a file "Folder2/sample.txt" and just delete sample.txt file, then Folder2 also disappears from console?
Why this inconsistency? AFAIK we do not have any API to create/delete folder at SDK level.
Am I missing something here or is this an actual issue?
Thanks in advance for any help.
A "Folder" in S3 is simply a 0-byte object with a / character at the end of the key name.
So, using the AWS CLI or SDKs, you can "create a folder" by "putting" an object that matches those criteria.
The AWS Management Console also does something extra: it simulates folders, even of they were not explicitly created. So, if you uploaded your object as "Folder2/sample.txt", it extrapolates and simulates "Folder2/" at the parent folder level. You can do this yourself with the CLI/SDKs using the delimiter parameter.
When you delete that object, since "Folder2" did not actually exist as a 0-byte object ending with / (see first paragraph), then "Folder2/" disappears from the management console.
I want to download a directory from my s3.
When I need a file, the s3 management console (aws web console) allows me to download it, but when a directory, I have to use aws-cli, like:
$ aws s3 cp s3://mybucket/mydirectory/ . --recursive
My question is: Is there any way to get the s3 uri (s3://mybucket/mydirectory/) from s3 management console?
It's URL is available, but it is slightly different from s3 URI required by aws-cli. I could not find any menu to get the uri.
Thank you in advance!
No, it is not displayed in the console. However, it is simply:
s3://<bucket-name>/<key>
Directories are actually part of the key. For example, foo.jpg stored in an images directory will actually have a key (filename) of images/foo.jpg.
(self-answer)
Because it seems there was no such way, I have created one:
pip install aws-s3-url2uri
And command aws_s3_url2uri will be available after installation.
This command internally converts the web console URLs to S3 URIs, so works with URLs and URIs and local paths:
aws_s3_url2uri ls "https://console.aws.amazon.com/s3/home?region=<regionname>#&bucket=mybucket&prefix=mydir/mydir2/"
calls
aws s3 ls s3://mybucket/mydir/mydir2/
internally.
To convert an S3 URL displayed in the console such as https://s3.us-east-2.amazonaws.com/my-bucket-name/filename to an S3 URI, remove the https://s3.us-east-2.amazonaws.com/ portion and replace it with s3://, like so:
s3://my-bucket-name/filename
It looks like this feature is now available in the AWS Web Console.
It is accessible in two ways:
Selecting the checkbox next to the file and clicking on "Copy S3 URI" button.
Clicking on the file, and clicking on the "Copy S3 URI" button on the top right.
You can get the value from the console by selecting the file in the console. Choose Copy path on the Overview tab to copy the S3:// link to the object.
It is possible to get the S3-URI for a proper key/file in the console, by selecting the key and clicking on the Copy path button, this will place the s3-URI for the file on the clipboard.
However, directories are not keys as such but just key prefixes, so this will not work for them.
You may fail to get s3uri if you are created a new bucket.
You can get s3uri after creating a new folder in your bucket >> select check box to the newly created folder >> then copy the s3uri that appears at the top.