So I have this issue that whenever I try to post a file to the s3 bucket with a pre-signed URL, the key for the metadata is being forced in lowercases?
I've looked at the Pre-signed URL it already sets the lowercase part when the URL has been genereted and Im wondering why? and how do I solve this issue?
I've tried to create a manual key-value pair in the s3 bucket on a file, where I clearly can set a key with capital letters as well?
const params = {
Bucket: 'buckets3',
Key: 'hoho-fileUpload-' + uuid.v4(),
Metadata: {"FooBar": "FooBar"},
Expires: 600
};
current output in the s3:
x-amz-meta-foobar: FooBar
Wishing output:
x-amz-meta-FooBar: FooBar
There is nothing you can do, AWS stores S3 metadata in lower case.
User-defined metadata is a set of key-value pairs. Amazon S3 stores
user-defined metadata keys in lowercase.
From: Object Meta Data and scroll to the bottom. It's the paragraph just above the Note.
On top of WaltDe's answered, I'd recommend you convert your metadata keys to kebab case when sending to aws s3 (foo-bar) so you can convert it back to pascal case in your lambda or code or wherever you use the metadata.
Related
I'm using an AWS Lambda function to create a file and save it to my bucket on S3, it is working fine. After executing the putObject method, I get a data object, but it only contains an Etag of the recently added object.
s3.putObject(params, function(err, data) {
// data only contains Etag
});
I need to know the exact URL that I can use in a browser so the client can see the file. The folder has been already made public and I can see the file if I copy the Link from the S3 console.
I tried using getSignedUrl but the URL it returns is used for other purposes, I believe.
Thanks!
The SDKs do not generally contain a convenience method to create a URL for publicly-readable objects. However, when you called PutObject, you provided the bucket and the object's key and that's all you need. You can simply combine those to make the URL of the object, for example:
https://bucket.s3.amazonaws.com/key
So, for example, if your bucket is pablo and the object key is dogs/toto.png, use:
https://pablo.s3.amazonaws.com/dogs/toto.png
Note that S3 keys do not begin with a / prefix. A key is of the form dogs/toto.png, and not /dogs/toto.png.
For region-specific buckets, see Working with Amazon S3 Buckets and AWS S3 URL Styles. Replace s3 with s3.<region>.amazonaws.com or s3-<region>.amazonaws.com in the above URLs, for example:
https://seamus.s3.eu-west-1.amazonaws.com/dogs/setter.png (with dot)
https://seamus.s3-eu-west-1.amazonaws.com/dogs/setter.png (with dash)
If you are using IPv6, then the general URL form will be:
https://BUCKET.s3.dualstack.REGION.amazonaws.com
For some buckets, you may use the older path-style URLs. Path-style URLs are deprecated and only work with buckets created on or before September 30, 2020. They are used like this:
https://s3.amazonaws.com/bucket/key
https://s3.amazonaws.com/pablo/dogs/toto.png
https://s3.eu-west-1.amazonaws.com/seamus/dogs/setter.png
https://s3.dualstack.REGION.amazonaws.com/BUCKET
Currently there are TLS and SSL certificate issues that may require some buckets with dots (.) in their name to be accessed via path-style URLs. AWS plans to address this. See the AWS announcement.
Note: General guidance on object keys where certain characters need special handling. For example space is encoded to + (plus sign) and plus sign is encoded to %2B. Also here.
in case you got the s3bucket and filename objects and want to extract the url, here is an option:
function getUrlFromBucket(s3Bucket,fileName){
const {config :{params,region}} = s3Bucket;
const regionString = region.includes('us-east-1') ?'':('-' + region)
return `https://${params.Bucket}.s3${regionString}.amazonaws.com/${fileName}`
};
You can do an another call with this:
var params = {Bucket: 'bucket', Key: 'key'};
s3.getSignedUrl('putObject', params, function (err, url) {
console.log('The URL is', url);
});
Scenario
I have Full text search requirement which can search inside the document. I am uploading documents in s3 bucket and encrypting it using envelope encryption.
Can we do full text search in encrypted document(in S3 bucket). If yes what are the rest API(NodeJS API) for the same.
Example => bucket1 =>Encrypted content in the files
bucket1/abc.pdf
bucket1/def.doc
bucket1/ghi.txt
and I want to search text like "I am from planet earth" in the above files.
I want in result file name(s) with above text.
Solution
I am reading following article:
aws article here
encryption of data at rest
Problem
Will it works if s3 bucket data is encrypted?
What will be the best solution for this scenario?
Elasticsearch does not search inside documents, you need to index the content of the documents inside elasticsearch to be able to perform searchs, it also does not support search on encrypted data, the data needs to be stored in clear text.
What you can do is configure SSL/TLS and authentication on Elasticsearch, so you only will be able to make requests if you use the correct certificate and a username and password.
Does anyone know how to decode the contents provided by amazon S3 in this field of the response? It looks like a hashed string, but I need to understand what does it contain in it? Does it use some commonly used hashing?
It looks like a hash, but the documentation says it's obfuscated, maybe it's salted.
NextContinuationToken is sent when isTruncated is true which means there are more keys in the bucket that can be listed. The next list requests to Amazon S3 can be continued with this NextContinuationToken. NextContinuationToken is obfuscated and is not a real key.
Example: 17z4MXisD8tDT/0+uf1UndaqeI3+K7vG8bso1NFBtbPq2gKRaS2Ax6
ioTgsamg0QOZt3V56dV4r0=
I have a system where after a file is uploaded to S3, a Lambda job raises a queue message and I use it to maintain a list of keys in a MySQL table.
I am trying to generate a pre-signed URL based on the records in my table.
i have two records currently
/41jQnjTkg/thumbnail.jpg
/41jQnjTkg/Artist+-+Song.mp3
Generating pre-signed URL using :
var params = {
Bucket: bucket,
Expires: Settings.UrlGetTimeout,
Key: record
};
S3.getSignedUrl('getObject', params);
The URL with thumbnail.jpg works perfectly fine, but the one with +-+ fails. The original file name on local disk was "Artist - Song.mp3". S3 replaced spaces with '+'. Now when I am generating a URL using the exact same filename that S3 uses, it doesn't work; I get a "Specified Key doesn't exist" error from S3.
What must I do to generate URLs consistently for all filenames?
I solved this after a little experimentation.
Instead of directly storing key that S3 provides in their S3 event message, I am first replacing '+' character with space (as they are originally on the disk) and then URL decoding it.
return decodeURIComponent(str.replace(/\+/img, " "));
Now generating a S3 Pre-Signed URL works as expected.
Before MySQL has the following records:
/41jQnjTkg/thumbnail.jpg
/41jQnjTkg/Artist+-+Song.mp3
Now:
/41jQnjTkg/thumbnail.jpg
/41jQnjTkg/Artist - Song.mp3
I personally feel there is an inconsistency with S3's api/event messages.
Had i generated a Signed URL directly using the Key that S3 itself provided in SQS event message, It wouldn't have worked. One must do this string replacement step & URL decoding on the key in order to use it to get a proper working url.
Not sure if this is by design or a bug.
The second file's name is coming to you form-urlencoded. The + is actually a space, and if you had other characters (like parenthesis) they would be percent-escaped. You need to run your data through a URL decoder before working with it further.
Side-note: if the only thing your Lambda function does is create an SQS message, you can do that directly from S3 without writing your own function.
AS the title says, what is a key in boto?
What does it encapsulate (fields, data structures, methods etc.)?
How does one access the file contents for files in an AWS bucket using a key/boto?
I was not able to find this information on their official documentation or on any other third party website. Could anybody provide this info?
Here are some examples of the usage of the key object:
def download_file(key_name, storage):
key = bucket.get_key(key_name)
try:
storage.append(key.get_contents_as_string())
except:
print "Some error message."
and:
for key in keys_to_process:
pool.spawn_n(download_file, key.key, file_contents)
pool.waitall()
In your code example - key is the object reference to the unique identifier within a bucket.
Think of buckets as a table in a database
think of keys as the rows in the table
you reference the key (better known as an object) in the bucket.
often in boto (not boto3) works like this
from boto.s3.connection import S3Connection
connection = S3Connection() # assumes you have a .boto or boto.cfg setup
bucket = connection.get_bucket('my_bucket_name_here') # this is like the table name in SQL, select OBJECT form TABLENAME
key = bucket.get_key('my_key_name_here') this is the OBJECT in the above SQL example. key names are a string, and there is a convention that says if you put a '/' in the name, a viewer/tool should treat it like a path/folder for the user, e.g. my/object_name/is_this is really just a key inside the bucket, but most viewers will show a my folder, and an object_name folder, and then what looks like a file called is_this simply by UI convention
Since you appear to be talking about Simple Storage Service (S3), you'll find that information on Page 1 of the S3 documentation.
Each object is stored and retrieved using a unique developer-assigned key.
A key is the unique identifier for an object within a bucket. Every object in a bucket has exactly one key. Because the combination of a bucket, key, and version ID uniquely identify each object, Amazon S3 can be thought of as a basic data map between "bucket + key + version" and the object itself. Every object in Amazon S3 can be uniquely addressed through the combination of the web service endpoint, bucket name, key, and optionally, a version. For example, in the URL http://doc.s3.amazonaws.com/2006-03-01/AmazonS3.wsdl, "doc" is the name of the bucket and "2006-03-01/AmazonS3.wsdl" is the key.
http://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html
The key is just a string -- the "path and filename" of the object in the bucket, without a leading /.