I am using amazon sdk version 2
For presigned_url i use the below
url = signer.presigned_url(:get_object, bucket: bucket, key: key)
But how do i set the filename?( For users when they select save as..)
Need to add response_content_disposition and set the filename
signer = Aws::S3::Presigner.new
signer.presigned_url(:get_object, bucket: bucket, key: key, response_content_disposition: "attachment;filename=test.pdf")
If you add ${filename} to the key it will be replaced with the filename chosen when uploading the file.
Related
As title, is it a correct answer since I'm so sure whether we can put directory to a folder as the bucket name:
SELECT COUNT(NumberOfObjects) FROM SCHEMA("AWS/S3", BucketName,StorageType)
WHERE BucketName = 'pbr-data-lake-dev/data/raw/A' AND BucketName = 'pbr-data-lake-dev/data/raw/B' AND BucketName = 'pbr-data-lake-dev/data/raw/C'
GROUP BY BucketName
I have this code for Amazon Lambda:
const file_stream = s3.getObject({ Bucket: bucket, Key: filename });
This line of code would be used along with exports.handler = async (event) function.
In regards to the "Key" parameter, is the filename should be just the filename (ex. filename.ext), full directory for the filename (ex. https://link/to/a/file/filename.ext), or something else? (I am pretty much new in AWS S3 and Lambda)
Lets assume you have a bucket my-bucket and a file 'abc.txt' inside folder hierarchy like 2021/04/12.
So you can do like below to get the object:
s3.getObject({ Bucket: 'my-bucket', Key: '2021/04/12/abc.txt' }).promise();
You can check the key value in console as well just click on the file you want to process and you can see key value under Properties Object overview tab.
I have a scala jar which I am calling from AWS Glue job. My jar writes to write a DataFrame to an S3 bucket in another AWS account which has KMS Encryption turned on. I am able to write to the bucket but I am not able to add the destination bucket owner permission to access the files. I can achieve this if simply use Glue Writer but with straight Spark, it just not work. I have read all the documentation and I am setting following bucket policies in hadoop configuration.
def writeDataFrameInTargetLocation( sparkContext:SparkContext = null, dataFrame: DataFrame, location: String,
fileFormat: String,saveMode:String,encryptionKey:Option[String] = Option.empty,kms_region:Option[String]=Option("us-west-2")): Unit = {
if(encryptionKey.isDefined) {
val region = if(kms_region.isDefined) kms_region.getOrElse("us-west-2")
else
"us-west-2"
sparkContext.hadoopConfiguration.set("fs.s3.enableServerSideEncryption", "false")
sparkContext.hadoopConfiguration.set("fs.s3.cse.enabled", "true")
sparkContext.hadoopConfiguration.set("fs.s3.cse.encryptionMaterialsProvider", "com.amazon.ws.emr.hadoop.fs.cse.KMSEncryptionMaterialsProvider")
sparkContext.hadoopConfiguration.set("fs.s3.cse.kms.keyId", encryptionKey.get) // KMS key to encrypt the data with
sparkContext.hadoopConfiguration.set("fs.s3.cse.kms.region", region) // the region for the KMS key
sparkContext.hadoopConfiguration.set("fs.s3.canned.acl", "BucketOwnerFullControl")
sparkContext.hadoopConfiguration.set("fs.s3.acl.default", "BucketOwnerFullControl")
sparkContext.hadoopConfiguration.set("fs.s3.acl", "bucket-owner-full-control")
sparkContext.hadoopConfiguration.set("fs.s3.acl", "BucketOwnerFullControl")
}
else {
sparkContext.hadoopConfiguration.set("fs.s3.canned.acl", "BucketOwnerFullControl")
sparkContext.hadoopConfiguration.set("fs.s3.acl.default", "BucketOwnerFullControl")
sparkContext.hadoopConfiguration.set("fs.s3.acl", "bucket-owner-full-control")
sparkContext.hadoopConfiguration.set("fs.s3.acl", "BucketOwnerFullControl")
}
val writeDF = dataFrame
.repartition(5)
.write
writeDF
.mode(saveMode)
.option(Header, true)
.format(fileFormat)
.save(location)
}
You are probably using the S3AFileSystem implementation for the "s3" scheme (i.e. URLs of the form "s3://..."). You can check that by looking at sparkContext.hadoopConfiguration.get("fs.s3.impl"). If that is the case, then you actually need to set the hadoop properties for "fs.s3a.*" not "fs.s3.*".
Then the correct settings would be:
sparkContext.hadoopConfiguration.set("fs.s3a.canned.acl", "BucketOwnerFullControl")
sparkContext.hadoopConfiguration.set("fs.s3a.acl.default", "BucketOwnerFullControl")
The S3AFileSystem implementation is not using any of the properties under "fs.s3". You can see that by investigating the code related to the following hadoop source code link:
https://github.com/apache/hadoop/blob/43e8ac60971323054753bb0b21e52581f7996ece/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java#L268
I have multiple folders in an s3 bucket and each folder contains some .txt files. Now I want to fetch just 10 .txt files from a given folder using javascript API.
For eg: the path is something like this
s3bucket/folder1/folder2/folder3/id
Now folder id is the one containing multiple .txt files. There are multiple id folders inside folder3. I want to pass id and get 10 s3 objects which have id as prefix. Is this possible using listObjectsV2? How do I limit the response to just 10 objects.
____obj1.txt
______id1----|____obj2.txt
| _____obj3.txt
|_____ id2---|____obj4.txt
s3bucket/folder1/folder2/folder3-| ____obj5.txt
|_____ id3---|____obj6.txt
So if I pass
var params= {Bucket:"s3bucket",Key:"folder1/folder2/folder3/id1"}
I should get obj1.txt and obj2.txt in response.
Which S3 method are you using? I suggest to use listObjectsV2 to achieve your goal. A possible call might look as the following
const s3 = new AWS.S3();
const { Contents } = await s3.listObjectsV2({
Bucket: 's3bucket',
Prefix: 'folder1/folder2/folder3/id1',
MaxKeys: 10
}).promise();
To get the object values you need to call getObject on each Key.
const responses = await Promise.all((Contents || []).map(({ Key }) => (
s3.getObject({
Bucket: 's3bucket',
Key
}).promise()
)));
I have a private bucket, I want create a pre signed url that allows a user to upload a file to within the time limit and set the ACL to public read only.
When creating a PutObjectRequest like below it works fine I can PUT the file no problem. When I add ACL: aws.String("public-read"), I get the error 'signature doesn't match' and the PUT fails, here is a sample of the url the GO sdk is generating.
https://<MY-BUCKET>.s3.eu-west-2.amazonaws.com/<MY-KEY>?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=<AWS_ACCESS_KEY>/20170505/eu-west-2/s3/aws4_request&X-Amz-Date=20170505T793528Z&X-Amz-Expires=900&X-Amz-SignedHeaders=host;x-amz-acl&X-Amz-Signature=2584062aaa76545665bfed7204fcf0dfe233a45016f698e7e8a11c34a5a7921e
I have tried with the root aws user and a normal user.
I have tried with bucket policy and without, and with bucket policy and IAM policy of FULL S3 access and without. Basically all combinations. Any time I add the ACL field the signature error appears.
I am not sure if it's related to the GO SDK or to the AWS service. Can someone advice on what I am to do?
svc := s3.New(session.New(&aws.Config{Region: aws.String("eu-west-2")}))
req, _ := svc.PutObjectRequest(&s3.PutObjectInput{
ACL: aws.String("public-read"),
Bucket: aws.String("MY BUCKET NAME"),
Key: aws.String("MY KEY"),
})
str, err := req.Presign(15 * time.Minute)
It was and error on the aws service end, the url is not being signed.