Is there any way to get s3 uri from aws web console? - amazon-web-services

I want to download a directory from my s3.
When I need a file, the s3 management console (aws web console) allows me to download it, but when a directory, I have to use aws-cli, like:
$ aws s3 cp s3://mybucket/mydirectory/ . --recursive
My question is: Is there any way to get the s3 uri (s3://mybucket/mydirectory/) from s3 management console?
It's URL is available, but it is slightly different from s3 URI required by aws-cli. I could not find any menu to get the uri.
Thank you in advance!


No, it is not displayed in the console. However, it is simply:
s3://<bucket-name>/<key>
Directories are actually part of the key. For example, foo.jpg stored in an images directory will actually have a key (filename) of images/foo.jpg.


(self-answer)
Because it seems there was no such way, I have created one:
pip install aws-s3-url2uri
And command aws_s3_url2uri will be available after installation.
This command internally converts the web console URLs to S3 URIs, so works with URLs and URIs and local paths:
aws_s3_url2uri ls "https://console.aws.amazon.com/s3/home?region=<regionname>#&bucket=mybucket&prefix=mydir/mydir2/"
calls
aws s3 ls s3://mybucket/mydir/mydir2/
internally.


To convert an S3 URL displayed in the console such as https://s3.us-east-2.amazonaws.com/my-bucket-name/filename to an S3 URI, remove the https://s3.us-east-2.amazonaws.com/ portion and replace it with s3://, like so:
s3://my-bucket-name/filename


It looks like this feature is now available in the AWS Web Console.
It is accessible in two ways:
Selecting the checkbox next to the file and clicking on "Copy S3 URI" button.
Clicking on the file, and clicking on the "Copy S3 URI" button on the top right.


You can get the value from the console by selecting the file in the console. Choose Copy path on the Overview tab to copy the S3:// link to the object.


It is possible to get the S3-URI for a proper key/file in the console, by selecting the key and clicking on the Copy path button, this will place the s3-URI for the file on the clipboard.
However, directories are not keys as such but just key prefixes, so this will not work for them.


You may fail to get s3uri if you are created a new bucket.
You can get s3uri after creating a new folder in your bucket >> select check box to the newly created folder >> then copy the s3uri that appears at the top.

Related

How to move S3 data to one partition level up?

Let's say I have a table in the path
s3:bucketname/tablename/month/day/deviceid
I want to move all files to s3:bucketname/tablename/month/day/. In other words, I want to ignore the last partition.
Note that there are already data in the path s3:bucketname/tablename/month/day/
Can I use aws s3 to achieve this?
I have more than 100K files so I cannot do this manually.

I believe you can use the AWS console on the web. Open the source folder (deviceid), select all its contents by the checkbox to the left on Name.
Then select Actions → Copy.
Open the destination folder (day), click Actions → Paste. Confirm the operation.
Once the operation has finished successfully, you can delete the source folder with all its contents.

You can do this in 2 ways.
If you are not familiar with the aws cli, then go to the console, select all the files, cut and paste on the desired directory. However, this is very time consuming and inefficient task.
I recommend the aws cli for S3. You can find some sample commands here.

How to create a new folder in S3 using AWS PowerShell

I have a bucket already created but I want to create new folders inside this bucket, not upload data or anything else, just create new folders. How can I do this ?
Thanks

AWS S3 doesn't really have a first class concept of a "folder" or "directory". S3 objects have prefixes, which are segmented by slashes, so there is certainly the appearance of folders, but it is not possible to have a truly empty directory structure.
However, their AWS Console user experience does present content as such, and provides a button to "Create Folder". When using that button, the UI provides the below message:
When you create a folder, S3 console creates an object with the above
name appended by suffix "/" and that object is displayed as a folder
in the S3 console.
You could try using PowerShell's Put Object API/cmdlet to create empty objects named per that instruction. For example, you could create a folder named "my-new-folder" by creating an object named "my-new-folder/".

S3 is object storage; it's not a regular file system. Generally speaking, there is no need to attempt to create folders. Simply upload objects, for example teams/east/falcons/logo.png.
If you really, really want to give the impression that there are folders then you can create zero-sized objects whose names ends in / (or whatever your preferred folder delimiter is). The simplest way to do this is with the AWS S3 console but any SDK will let you do it too (simply issue a PutObject with no body).

I was seaching for this myself and found this.
use -content where content = key then -file or -folder are not needed
$s3Path = "/folder/" + 'subfolder/'
Write-S3Object -BucketName $s3Bucket -Key $s3Path -Content $s3Path

Delete object from AWS S3 bucket (not in code)

I just want to update my index.html file in an S3 bucket. I was going to delete the file and then add the new one just using the website. I can't see any way to do this. Am I thinking wrong? Am I supposed to do all this stuff in code? I just want to hack it together for now so can I do it on the website?

In the console, you can select your file, click "More" and delete is one of the options. There's the upload button to the left as well. I also suggest getting familiar with the AWS CLI so that you also can sync directories and such.

You can go into S3 select a single file and delete it. Afterwards you can upload a new file.
Old console: Select the file in S3 and click Actions and then click delete.
New console: Select the file in S3 and click More and than click delete.
You could use a client like Cyberduck as well.

Inconsistent behavior in Amazon S3 for folder/key upload, delete through console

There seems to be some inconsistency in Amazon S3's behavior.
If in bucket a "Bucket1", I create folder "Folder1" and upload a file say "sample.txt" into it. Next I delete this file. At the bucket level I can see "Folder1" on S3 Console.
Now in the same bucket if I upload a file "Folder2/sample.txt" and just delete sample.txt file, then Folder2 also disappears from console?
Why this inconsistency? AFAIK we do not have any API to create/delete folder at SDK level.
Am I missing something here or is this an actual issue?
Thanks in advance for any help.

A "Folder" in S3 is simply a 0-byte object with a / character at the end of the key name.
So, using the AWS CLI or SDKs, you can "create a folder" by "putting" an object that matches those criteria.
The AWS Management Console also does something extra: it simulates folders, even of they were not explicitly created. So, if you uploaded your object as "Folder2/sample.txt", it extrapolates and simulates "Folder2/" at the parent folder level. You can do this yourself with the CLI/SDKs using the delimiter parameter.
When you delete that object, since "Folder2" did not actually exist as a 0-byte object ending with / (see first paragraph), then "Folder2/" disappears from the management console.

Can't delete directory from Amazon S3

I'm using the web interface of Amazon's S3, and when I right-click a folder X and choose Delete, X seems to be deleted. Then when I refresh the interface, X may either disappear or remain; if I keep clicking refresh, the folder is either missing or present. Is this a bug, or am I doing something wrong? The folder is still present, as far as I can tell; one of my EMR jobs complains that the output folder X still exists.

I had the same problem in the AWS web interface after AWS Command Line (CLI)-deleting a "recursive" folder in a bucket. Some objects randomly reappeared (not files, but in fact "folders") in the web interface. Even though i tried to delete these folders in the web interface, they were still there (The interface said the operation was successful...)
Solution that worked for me in the AWS web interface: Right clicked the folder -> CUT, and PASTE into another folder. Worked great, and then deleted the new folder. Gone!

Tried the same as Kristoffer's answer, but CUT/PASTE to another folder made the new folder to not be deletable.
Further hacking: create a new temporary bucket. CUT/PASTE the folder to this bucket and delete the bucket.

S3 does not actually use folders. Instead the path separators in object paths are treated like folders. If you want to remove a folder, all the contents of the folder will have to get deleted.
If there is any delay in deleting all of the contents, the folder may continue to exist.

As of March 2017 the AWS Console UI has changed and you can no longer enter a 'versioning mode' described in my old post.
It seems now folder with versioned files can be deleted freely without restriction.
If this is not the case please drop a comment so I can correct this post.
Previous Version of AWS Console
If you are using the AWS Management Console and you have versioning turned ON, you must be in 'versioning mode' to delete the folder.
You enter 'versioning mode' by going to the top of the page and next to where it says 'Versions:' select the Show button. Then you can proceed to right-click and delete your folder.
Hope this helps someone.

I encountered this issue when I was unable to delete an empty folder from an S3 bucket that had Versioning enabled.
I was able to delete the empty folder by using the "empty bucket configuration" from the S3 Buckets listing:
Select the bucket you'd like to empty, and click the Delete button:
AWS warns you that the bucket isn't empty, and offers a link to use the empty bucket configuration. Click the link:
Proceed through this screen by typing permanently delete to delete all the objects in this bucket:
You should then be able to verify that your S3 bucket is truly empty.

Tried various alternatives to delete from Web interface to delete a folder with sub folders in it without luck.
I had an installation of S3 browser and then tried from S3 Browser interface, worked.

I think I'm seeing similar behavior. My bucket has versioning turned on; even with an empty folder/directory within the bucket, attempting to "delete" the folder/directory within the bucket via the AWS web UI console does not result in it actually being removed. I presume the "deleted" versions of the files within that path still exist (but are not visible in the web console), therefore the bucket isn't truly empty, and isn't truly getting deleted. You may need to check via the CLI tools if existing deleted versions of files in that folder/directory exist (but are not visible in the web console) and delete the files permanently, then attempt to remove the folder/directory in your bucket.

I have the same problem that I cant delete a s3 bucket, with the message "An error occurred (AccessDenied) when calling the DeleteBucket operation: Access Denied"
After a while, I delete the bucket policy in tab "permission" button "bucket policy" and It worked like a charm, with:
aws s3 rb s3://elasticbeanstalk-us-west-..../ --force
I hope this help! Is another option
Pablo

had an "elastic-bean-stalk" bucket and had to delete "bucket policy" before it would delete.
pitney

I had the same problem and didn't have access to the amazon console but I could delete it with this Java code
AmazonS3Client amazonS3Client = new AmazonS3Client(basicAWSCredentials);
ObjectListing objectListing = amazonS3Client.listObjects("bucketName", "prefix");
DeleteObjectsRequest deleteObjectsRequest = new DeleteObjectsRequest("bucketName");
List<DeleteObjectsRequest.KeyVersion> keysToDelete = new ArrayList<>();
objectListing.getObjectSummaries().forEach(s3ObjectSummary -> {
keysToDelete.add(new DeleteObjectsRequest.KeyVersion(s3ObjectSummary.getKey()));
});
deleteObjectsRequest.setKeys(keysToDelete);
amazonS3Client.deleteObjects(deleteObjectsRequest);
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-java-sdk</artifactId>
<version>1.7.4</version>
</dependency>

Try delete with another account, like administrator account. For me it works only with this method.

If you're having trouble fully deleting an empty folder in an S3 bucket that has versioning turned on (i.e. removing all trace of the folder, including in 'Show versions' mode), you can usually get around it by deleting the folder's delete marker(s) using the API/CLI:
$ aws s3api list-object-versions --bucket YOUR-BUCKET --prefix PATH-TO-YOUR-FOLDER
{
"DeleteMarkers": [
{
"Owner": {
"DisplayName": "YOUR-ACCOUNT-NAME",
"ID": "YOUR-ACCOUNT-CANONICAL-ID"
},
"Key": "PATH-TO-YOUR-FOLDER/",
"VersionId": "UNIQUE-VERSION-ID",
"IsLatest": true,
"LastModified": "2022-12-09T07:18:57+00:00"
}
]
}
$ aws s3api delete-objects --bucket YOUR-BUCKET --delete 'Objects=[{Key=PATH-TO-YOUR-FOLDER/,VersionId=UNIQUE-VERSION-ID}]'
{
"Deleted": [
{
"Key": "PATH-TO-YOUR-FOLDER/",
"VersionId": "UNIQUE-VERSION-ID",
"DeleteMarker": true,
"DeleteMarkerVersionId": "UNIQUE-VERSION-ID"
}
]
}

Try use the new S3 console. The delete feature works for folders.