I have uploaded files to s3 bucket with a UUID as a key for each file name ,
I have a requirement to keep the files key as the stored uuid but when download i need to have the downloaded file name as the actual file name eg: Foo.png
stored file on aws s3 -0e8221b9-9bf4-49d6-b0c0-d99e86f91f8e.png
Downloading file name should be : foo.bar
I have tried with setting Content-Disposition meta Data but still when downloading the file contains the uuid.
Perform the below changes and try.
Update Content-Disposition = attachment;filename="abc.csv". Please note the file name is case sensitive and if you are using CDN then it will take some time after you apply the changes. After you update the metadata then download the file using the OBJECT URL. Direct download is not working.. If I download the file using Object URL then the downloaded file name is abc.csv instead of test.csv.
Related
i want to download a file from Amazon S3 using only a certain unique id i can use from it's api, without using a folder or subfolder name. I created a folder/subfolder structure with hierarchy levels to organize the files.
The same of what I did in Google Drive API v3, regardless of which the folder or subfolder name or hierarchy level of folders the file was saved, i can download the file using only the fileid.
i haven't read yet about the file versioning docs since there are tons to read.
any help would greatly be appreciated. thank you.
You can't do this with S3. You need to know the bucket name (--bucket) and full key (--key) of the file you want to download. Since a given file can have multiple versions, you can also provide a version id (--version-id).
With PHP
how to put object by an external URL to amazon s3?
So suppose I had a URL: http://example.com/file.avi I want to be able to move it into my bucket without downloading the file to my desktop and re-uploading the file. Is this possible?
S3 only supports copying objects from another S3 bucket, or uploading of local files.
It is not possible to upload a resource located at an external URL.
See here for more details:
Put Object from remote resource in Amazon S3
You can do it using S3.php by tpyo https://github.com/tpyo/amazon-s3-php-class
Even it is not included in his ReadMe file, you can use the putObjectString() static function of it but you must convert first the url to string file by doing
$fileUrl = file_get_contents("http://www.somesite.com/imagesample.png");
S3::putObjectString($fileUrl, "yourBucket", "uploads/filenamehere.png");
More details: https://gist.github.com/neilmaledev/d255c42f1289a9ab9394121b7896d4d3
I'm trying to download GhTorrent dump from http://ghtorrent-downloads.ewi.tudelft.nl/mysql/mysql-2020-07-17.tar.gz which is about 127gb
I tried in the cloud but after 6gb it stops, I believe that there is a size limit for using curl
curl http://ghtorrent... | gsutil cp - gs://MY_BUCKET_NAME/mysql-2020-07-17.tar.gz
I cannot use Data Transfer as I need to specify the url, size in bytes (which I have) and hash MD5 which I don't have and I only can generate by having the file in my disk. I think(?)
Is there any other option to download and upload the file directly to the cloud?
My total disk size is 117gb sad beep
Worked for me with Storage Transfer Service: https://console.cloud.google.com/transfer/
Have a look on the pricing before moving TBs especially if your target is nearline/coldline: https://cloud.google.com/storage-transfer/pricing
Simple example that copies a file from a public url, to my bucket using a Transfer Job:
Create a file theTsv.tsv and specify the complete list of files that must be copied. This example contains just one file:
TsvHttpData-1.0
http://public-url-pointint-to-the-file
Upload the theTsv.tsv file to your bucket or any publicly accessible url. In this example I am storing my .tsv file on my bucket https://storage.googleapis.com/<my-bucket-name>/theTsv.tsv
Create a transfer job - List of object URLs
Add the url that points to the theTsv.tsv file in the URL of TSV file field;
Select the target bucket
Run immediately
My file, named MD5SUB was copied from the source url into my bucket, under an identical directory structure.
I am trying to add a file in s3-bucket in my AWS account using postman. see below screenshot.
I pass Host in the header as a divyesh.vkinds.com.s3.amazonaws.com where divyesh.vkinds.com is my bucket name. and in Body I am giving file as index.html as file type like image below.
but it is giving me The provided 'x-amz-content-sha256' header does not match what was computed.
error. I searched for it but can't find anything.
Please check content-header. Add Content-Type as text/plain and date in this format XX-XX-XXXX
I have also faced the same problem. The issue was that, postman does not calculate the SHA. It defaults to a SHA of empty string e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
So in the postman headers, add an explicit key x-amz-content-sha256. Caluclate the value of SHA256 for your file using a sha command and provide as the value. Below command works on linux flavors
shasum -a 256 index.html
Couple of other observations in the question.
You can change the Body as binary and choose the file you want to upload.
Provide the complete path including the file name in the upload URL. E.g. if you provide the URL as <your bucket name>.s3.<region>.amazonaws.com/test/index.html, the file will be copied to test directory in the bucket with name as index.html
I encountered this situation recently, and the issue was that I was copying an active log file which changed between when my side calculated the hash and when the file was actually uploaded. My solution was to copy the file to a temporary location, then upload that stable file.
I have 2 AWC accounts, each of them has one S3 bucket. I uploaded two same-size .CSV files to each of the S3 bucket.
When I try to Download or Download As, this file is downloaded as .CSV file in first account. BUT(!!) When I try to download this file from second account - it is downloading it as .TXT.
How can this happen? Both files are created in the same way: through Redshift UNLOAD query, that perform copying of selected data from Redshift to S3.
UPDATE:
Can it be because in this account for this document , **Server side encryption is equal to AWS-KMS?
I noticed that file, that converted from .csv to .txt has "Server side encryption: AWS-KMS", while .csv file that is downloaded as .csv - has "Server side encryption: NONE"
UPDATE: tried in different browsers - same result
Check the headers for each object in the AWS S3 console and compare the Content-Type values. Content-Type provides a hint to web browsers on what data the object contains.
If Content-Type does not exist or does not contain text/csv, add or modify the header in the S3 console or via your favorite S3 application such as CloudBerry.
John is right about the Content-Type not being text/csv. Sometimes, S3 will get it right and sometimes it won't. If you can't manually correct this yourself, you can run a Lambda function to do this for you everytime you upload a new object. You can use a Python 2.7 template Lambda function to download the object from the bucket, employ mimetypes library to guess_type for your S3 object, and then re-upload the file in the same bucket. You will need to trigger this function with S3 object upload and give it the necessary permissions (S3:GetObject).
P.S. This will work for files with any extension. If you know you are only going to upload .csv files, you can ignore the mimetypes and directly re-upload the object with
bucket.upload_fileobj(filename, key, ExtraArgs={'ContentType': 'text/csv'})
If the mimetypes cannot guess the typethen you might need to add the types, look at an example here https://www.programcreek.com/python/example/5209/mimetypes.add_type
Good Luck!
Here is scala solution (to specify content type):
val settingsLine: String = "csvdata1,csvdata2,csvdata3"
val settingsStream: InputStream = new ByteArrayInputStream(settingsLine.getBytes())
val metadata: ObjectMetadata = new ObjectMetadata()
metadata.setContentType("text/csv")
s3Client.putObject(bucketName, prefix, settingsStream, metadata)