Change Content-Disposition of existing S3 object - amazon-web-services

In S3 REST API I am adding metadata to an existing object by using the PUT (Copy) command and copying a key to the same location with 'x-amz-metadata-directive' = 'REPLACE'
What I want to do is change the download file name by setting:
Content-Disposition: attachment; filename=foo.bar;
This sets the metadata correctly but when I download the file it still uses the key name instead of 'foo.bar'
I use a software tool S3 Browser to view the metadata and it looks correct (apart from 'Content-Disposition' being all lower case as that's was S3 ask me to sign)
Then using S3 Browser I just pressed, then save without changing anything and now it works???
What am I missing how come setting a metadata 'Content-Disposition: attachment; filename=foo.bar;' from my web app does not work but does work from S3 Browser?

Edited for clarity:
Content-Disposition must be set explicitly and not included as x-amz-meta-Content-Disposition. All metadata header names must start with "x-amz-meta-" and be all lowercase.
Thanks to #Eyal for clarifying.
Original:
>SOLVED:
>
>The Doco at http://docs.amazonwebservices.com/AmazonS3/latest/dev/index.html?RESTAuthentication.html
>
>seems to be wrong it says:
>
>Notice how the 'x-amz-' headers are sorted, white-space trimmed, converted tolowercase, and multiple headers with the same name have been joined using a comma toseparate values.
>
>Note how only the Content-Type and Content-MD5HTTPentity headers appear in the StringToSign .The otherContent-* entity headers do not.
However Content-Disposition must be set specifically and not included as : x-amz-meta-Content-Disposition
>
>It now works fine.

here: this uses the cli to set the content-disposition header on all files in a path inside a bucket (and also sets them as public):
aws s3 ls s3://mybucket/brand_img/ios/|awk {'print $4'} > objects.txt
while read line; do aws s3api copy-object --bucket mybucket \
--copy-source /mybucket/brand_img/ios/$line --key brand_img/ios/$line \
--metadata-directive REPLACE --metadata Content-Disposition=$line --acl public-read; done < objects.txt

Related

How to upload a file to S3 without a name

I am reading data from S3 bucket using Athena and the data from following file is correct.
# aws s3 ls --human s3://some_bucket/email_backup/email1/
2020-08-17 07:00:12 0 Bytes
2020-08-17 07:01:29 5.0 GiB email_logs_old1.csv.gz
When I change the path to _updated as shown below, I get an error.
# aws s3 ls --human s3://some_bucket/email_backup_updated/email1/
2020-08-22 12:01:36 5.0 GiB email_logs_old1.csv.gz
2020-08-22 11:41:18 5.0 GiB  
This is because of the extra file without name in the same location. I have no idea how I managed to upload a file without a name. I will like to know how to repeat it (so that I can avoid it)
All S3 files have a name (in fact the full path is actually the object key, which is the metadata to define your object name).
If you see a blank named file in the path of s3://some_bucket/email_backup_updated/email1/ you have likely created a file named s3://some_bucket/email_backup_updated/email1/.
As I mentioned earlier S3 objects use key, for this reason the file hierarchy does not exist. You simply are filtering by prefix instead.
You should be able to validate this by performing the following without the trailing slash aws s3 ls --human s3://some_bucket/email_backup_updated/email1.
If you add an extra non-breaking space at the end of the destination path, the file will be copied to S3 but with a blank name. for e.g.
aws s3 cp t.txt s3://some_bucket_123/email_backup_updated/email1/ 
(Note the non-breaking space after email1/ )
\xa0 is actually non-breaking space in Latin1, also chr(160). The non breaking space itself is the name of the file!
Using the same logic, I can remove the "space" file by adding the non-breaking space at the end.
aws s3 rm s3://some_bucket_123/email_backup_updated/email1/ 
I can also login to console and remove it from User Interface.

can not add file in aws s3 bucket using postman

I am trying to add a file in s3-bucket in my AWS account using postman. see below screenshot.
I pass Host in the header as a divyesh.vkinds.com.s3.amazonaws.com where divyesh.vkinds.com is my bucket name. and in Body I am giving file as index.html as file type like image below.
but it is giving me The provided 'x-amz-content-sha256' header does not match what was computed.
error. I searched for it but can't find anything.
Please check content-header. Add Content-Type as text/plain and date in this format XX-XX-XXXX
I have also faced the same problem. The issue was that, postman does not calculate the SHA. It defaults to a SHA of empty string e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
So in the postman headers, add an explicit key x-amz-content-sha256. Caluclate the value of SHA256 for your file using a sha command and provide as the value. Below command works on linux flavors
shasum -a 256 index.html
Couple of other observations in the question.
You can change the Body as binary and choose the file you want to upload.
Provide the complete path including the file name in the upload URL. E.g. if you provide the URL as <your bucket name>.s3.<region>.amazonaws.com/test/index.html, the file will be copied to test directory in the bucket with name as index.html
I encountered this situation recently, and the issue was that I was copying an active log file which changed between when my side calculated the hash and when the file was actually uploaded. My solution was to copy the file to a temporary location, then upload that stable file.

Sync objects to S3 with expires and cache-control headers using the CLI?

I'm trying to sync some objects to S3 and set the Expires and Cache-Control headers, but I'm at my wit's end here. Nothing seems to work. Here's my latest attempt:
aws s3 sync . s3://my-bucket \
--expires "2020-06-16T13:27:40Z" \
--cache-control "max-age=315360000, public, s-maxage=31536000, max-age=31536000, immutable" \
--exclude "*" \
--metadata-directive REPLACE \
--include "bundles"
The result: no Expires header, no Cache-Control header. I've looked around in the console (only one metadata, Content-Type), I've used get-object to look at it, and I looked at the response with curl. I'm not really sure about metadata-directive - it is not mentioned under --expires in the docs, but the docs for the directive option indicates it must be set for the other ones to work. What crazy incantation must I conjure to have these headers set on my objects?
This eventually turned out to be a PEBCAC. This does indeed work when run in isolation. I ran it as part of a multi-step process that would first sync some files, except the ones I wanted the headers on, and then sync the files with headers. Problem was, I fat-fingered the exclude pattern in the first sync, so basically all the files where already synced, and so the header-setting sync did nothing. Ah, ain't being a developer lovely?

Correct gzip encoding on S3

While transferring my files using "aws s3 sync", transferred files does not have right Content-type and Content-encoding. I am able to solve the types by tweaking /etc/mime.types however no idea how to set right encoding for ".gz" extension so zipped files are served as text apart from:
changing types on s3 afterwards (seems like double-work to me)
aws-cli using exclude / include with correct types
(this results in multiple commands)
Any idea how to solve this? Thanks...
Here is how I solved it,
aws s3 sync /tmp/foo/ s3://bucket/ --recursive
--exclude "*" --include "*.gz" --content-type "text/plain; charset=UTF-8"
By default aws s3 sync command assumes the best matching content types. If you want to change the default behavior you need to handle them separately.
Reference:
https://docs.aws.amazon.com/cli/latest/reference/s3/sync.html
Hope it helps.

Cannot delete Amazon S3 key that contains bad character

I just began to use S3 recently. I accidentally made a key that contains a bad character, and now I can't list the contents of that folder, nor delete that bad key. (I've since added checks to make sure I don't do this again).
I was using an old "S3" python module from 2008 originally. Now I've switched to boto-2.0, and I still cannot delete it. I did quite a bit of research online, and it seems the problem is I have an invalid XML character, so it seems a problem at the lowest level, and no API has helped so far.
I finally contacted Amazon, and they said to use "s3-curl.pl" from http://aws.amazon.com/code/128. I downloaded it, and here's my key:
<Key>info/[01</Key>
I think I was doing a quick bash for loop over some files at the time, and I have "lscolors" set up, and so this happened.
I tried
./s3curl.pl --id <myID> --key <myKEY> -- -X DELETE https://mybucket.s3.amazonaws.com/info/[01
(and also tried putting the URL in single/double quotes, and also tried to escape the '[').
Without quotes on the URL, it hangs. With quotes, I get "curl: (3) [globbing] error: bad range specification after pos 50". I edited the s3-curl.pl to do curl --globoff and still get this error.
I would appreciate any help.
This solved the issue, just delete the main folder:
aws s3 rm "s3://BUCKET_NAME/folder/folder" --recursive
You can use the s3cmd tool from here. You first need to run
s3cmd fixbucket <bucket name that contains bad file>.
You can then delete the file using
s3cmd del <bucket>/<file>
In my case there were newlines in the key (however that happened..). I was able to fix it with the aws cli like this:
aws cli rm "s3://my_bucket/Icon"$'\r'
I also had versioning enabled, so I also needed to do this, for all the versions (versions ids are visible in the UI when enabling the version view):
aws s3api delete-object --bucket my_bucket --key "Icon"$'\r' --version-id <version_id>
I was in this situation recently, to list the items you can use:
aws s3api list-objects-v2 --bucket my_bucket --encoding-type url
the bad keys will come back url encoded like:
"Key": "%01%C3%B4%C2%B3%C3%8Bu%C2%A5%27%40yr%3E%60%0EQ%14%C3%A5.gif"
spaces became + and I had to change those to %20 and * wasn't encoded I had to replace those with %2A before I was able to delete them.
To actually delete them, I wasn't able to use the aws cli because it would urlencode the already urlencoded key resulting in a 404, so to get around that I manually hit the rest API with the DELETE verb.
I recently encountered this case. I had newline at the end of my bucket. The following command solved the matter.
aws s3 rm "bucket_name"$'\r' --recursive