I'm looking to implement an AWS Lambda function that is triggered by the input of an audio file to my s3 bucket, and then concatenates this file with the previously uploaded file (already stored in the bucket), and outputs this concatenated file back to the bucket. I'm quite new to Lambda and I'm wondering, is possible to pass in a list of file names to process into a Lambda function to transcode? Or does Lambda only accept one read at a time?
When Lambda gets invoked by S3 directly, it will get an event that looks similar to this one from the docs:
{
"Records": [
{
"eventVersion": "2.1",
"eventSource": "aws:s3",
"awsRegion": "us-east-2",
"eventTime": "2019-09-03T19:37:27.192Z",
"eventName": "ObjectCreated:Put",
"userIdentity": {
"principalId": "AWS:AIDAINPONIXQXHT3IKHL2"
},
"requestParameters": {
"sourceIPAddress": "205.255.255.255"
},
"responseElements": {
"x-amz-request-id": "D82B88E5F771F645",
"x-amz-id-2": "vlR7PnpV2Ce81l0PRw6jlUpck7Jo5ZsQ"
},
"s3": {
"s3SchemaVersion": "1.0",
"configurationId": "828aa6fc-f7b5-4305-8584-487c791949c1",
"bucket": {
"name": "lambda-artifacts-deafc19498e3f2df",
"ownerIdentity": {
"principalId": "A3I5XTEXAMAI3E"
},
"arn": "arn:aws:s3:::lambda-artifacts-deafc19498e3f2df"
},
"object": {
"key": "b21b84d653bb07b05b1e6b33684dc11b",
"size": 1305107,
"eTag": "b21b84d653bb07b05b1e6b33684dc11b",
"sequencer": "0C0F6F405D6ED209E1"
}
}
}
]
}
It gives you basic meta information about the object that was uploaded.
There's no way to customize this event, but a Lambda function can query external resources for more information.
You could also use S3-Batching, but that's probably not designed for your use case.
Related
I get the following trigger event for some of the files that are uploaded to a location s3://bucket/folder1:
{
"Records": [
{
"eventVersion": "2.1",
"eventSource": "aws:s3",
"awsRegion": "eu-west-3",
"eventTime": "2022-07-26T08: 30: 03.280Z",
"eventName": "ObjectCreated:Put",
"userIdentity": {
"principalId": "principalId"
},
"requestParameters": {
"sourceIPAddress": "sourceIPAddress"
},
"responseElements": {
"x-amz-request-id": "JZJMKZAYX3HMQTY8",
"x-amz-id-2": "iXfsgUn5v1SQuR+YAacurX2qP+B7f39StWcWEyebkDbJzZzazygE9tABlKpg5hcW6lNOqZgEQ2jupDb26T9dww8fTG1O2Q0l"
},
"s3": {
"s3SchemaVersion": "1.0",
"configurationId": "NjViZmRiNmQtMTM0NS00NGZmLThlYjgtYjc4YWE4MWE2ZGU3",
"bucket": {
"name": "bucket",
"ownerIdentity": {
"principalId": "principalId"
},
"arn": "arn:aws:s3: : :bucket"
},
"object": {
"key": "folder1/",
"size": 0,
"eTag": "d41d8cd98f00b204e9800998ecf8427e",
"versionId": "qsENjqz.CmAFvvRg4z0.8ug4K0rZmegS",
"sequencer": "0062DFA60B3BF1F737"
}
}
}
]
}
Note that the key contains only prefix without file name.
Partial keys lead to 404 errors in the Lambda function:
event["Records"][0]["s3"]["object"]["key"]
[ERROR] ClientError: An error occurred (404) when calling the HeadObject operation: Not Found
For the files I upload via S3 CLI, the object key is correct: folder1/file1.txt. How does it happen that I receive some 'object keys' without filenames?
The event
"key": "folder1/",
"size": 0,
means that someone literally created an object called "folder1/" with size 0. One way this happens is if you use S3 Console and use Create folder button in the console.
AFAIK, object names on AWS S3 are always case sensitive and it is impossible to configure AWS S3 to be case insensitive.
So, is it possible to configure something like AWS Lambda in order to normalize uploaded file names to lower case? Or what is the best practice to perform this task with AWS S3?
Yes, this is easily done by having a Lambda function subscribe to your S3 PUT Event.
{
"Records": [
{
"eventVersion": "2.0",
"eventTime": "1970-01-01T00:00:00.000Z",
"requestParameters": {
"sourceIPAddress": "127.0.0.1"
},
"s3": {
"configurationId": "testConfigRule",
"object": {
"eTag": "0123456789abcdef0123456789abcdef",
"sequencer": "0A1B2C3D4E5F678901",
"key": "HappyFace.jpg",
"size": 1024
},
"bucket": {
"arn": bucketarn,
"name": "sourcebucket",
"ownerIdentity": {
"principalId": "EXAMPLE"
}
},
"s3SchemaVersion": "1.0"
},
"responseElements": {
"x-amz-id-2": "EXAMPLE123/5678abcdefghijklambdaisawesome/mnopqrstuvwxyzABCDEFGH",
"x-amz-request-id": "EXAMPLE123456789"
},
"awsRegion": "us-east-1",
"eventName": "ObjectCreated:Put",
"userIdentity": {
"principalId": "EXAMPLE"
},
"eventSource": "aws:s3"
}
]
}
You can then grab event.Records[0].s3.bucket.name and event.Records[0].s3.object.key to make a copyObject request to AWS
Once your file has been copied successfully, you can then delete the original file.
Just make sure your Lambda is configured for PUT events only, because if you set it to ALL events, both COPY and DELETE will also trigger your function, making you enter in an infinite recursion.
Background
I'm storing some files (objects) in S3 bucket.
Whenever any file gets deleted from S3 bucket my Lambda function gets triggered.
The lambda function needs to subtract the object size from DynamoDB.
Problem:
S3 deleteObject event does not send object Size
Sample S3 deleteObject event
{
"Records": [
{
"eventVersion": "2.0",
"eventTime": "1970-01-01T00:00:00.000Z",
"requestParameters": {
"sourceIPAddress": "127.0.0.1"
},
"s3": {
"configurationId": "testConfigRule",
"object": {
"sequencer": "0A1B2C3D4E5F678901",
"key": "HappyFace.jpg"
},
"bucket": {
"arn": "arn:aws:s3:::mybucket",
"name": "sourcebucket",
"ownerIdentity": {
"principalId": "EXAMPLE"
}
},
"s3SchemaVersion": "1.0"
},
"responseElements": {
"x-amz-id-2": "EXAMPLE123/5678abcdefghijklambdaisawesome/mnopqrstuvwxyzABCDEFGH",
"x-amz-request-id": "EXAMPLE123456789"
},
"awsRegion": "us-east-1",
"eventName": "ObjectRemoved:Delete",
"userIdentity": {
"principalId": "EXAMPLE"
},
"eventSource": "aws:s3"
}
]
}
Please help me find solution for my use case.
A way to do it is to enable versioning and check the metadata of the last version.
To avoid having the deleted version forever, you could setup an expiration policy or explicitly delete the version. I'd probably use both to catch cases where the event processor (the lambda function) fails and could not delete the file.
How can I get only Updated S3 Key object data from SNS topic subscription mail
{
"Records": [
{
"eventVersion": "2.0",
"eventSource": "aws:s3",
"awsRegion": "us-east-1",
"eventTime": "2018-01-27T01:24:43.660Z",
"eventName": "ObjectCreated:Put",
"userIdentity": {
"principalId": "AWS:AIDAJNQE6"
},
"requestParameters": {
"sourceIPAddress": "98.210.xxx.xxx"
},
"responseElements": {
"x-amz-request-id": "D464A8CB49C8D082",
"x-amz-id-2": "8g8K1G6b5sSUaxdZyshgxpk2wZR9hUSmj="
},
"s3": {
"s3SchemaVersion": "1.0",
"configurationId": "Radhikas3update",
"bucket": {
"name": "Radhika-test",
"ownerIdentity": {
"principalId": "AG737Y2"
},
"arn": "arn:aws:s3:::radhika-test"
},
"object": {
"key": "test.txt",
"size": 14,
"eTag": "12b9d85febf",
"sequencer": "005A6B"
}
}
}
]
}
How can I get only Updated S3 Key object data from SNS topic subscription mail
********Expecting Output*****************
"object": {
"key": "test.txt",
"size": 14,
"eTag": "12b9d85febf",
"sequencer": "005A6B"
}
Please help approach to get only key value in mail
There are no true 'directories' in S3, because S3 is an object store, and each object has a single key. An S3 bucket is basically a set of key-value pairs.
It is typical to use keys that are somewhat 'namespaced', to allow them to look like directories through. The AWS Console gives the impression of a directory filesystem by breaking up the keys by these namespace components.
For example, I might have a key like this: com/stackoverflow/nicholas.hauschild/secrets.txt, and the AWS console would give the impression that there is a 'file' called secrets.txt in the directory com/stackoverflow/nicholas.hauschild/.
In your example, I see no directory, as the key is simply test.txt.
If you were to use my example 'namespaced' key in your example json, it would look like this:
...
"object": {
"key": "com/stackoverflow/nicholas.hauschild/secrets.txt",
"size": 99999999999999,
"eTag": "12345",
"sequencer": "67890"
}
...
At this point you will see that you simply need to look at the key in your JSON and process the 'directory' from there its value.
I have followed this tutorial to create thumbnails of images to another bucket with AWS Lambda: http://docs.aws.amazon.com/lambda/latest/dg/walkthrough-s3-events-adminuser-create-test-function-upload-zip-test.html
I have done all the steps earlier in the tutorial but when I run the code below in Lambda test from the link above
{
"Records":[
{
"eventVersion":"2.0",
"eventSource":"aws:s3",
"awsRegion":"us-east-1",
"eventTime":"1970-01-01T00:00:00.000Z",
"eventName":"ObjectCreated:Put",
"userIdentity":{
"principalId":"AIDAJDPLRKLG7UEXAMPLE"
},
"requestParameters":{
"sourceIPAddress":"127.0.0.1"
},
"responseElements":{
"x-amz-request-id":"C3D13FE58DE4C810",
"x-amz-id-2":"FMyUVURIY8/IgAtTv8xRjskZQpcIZ9KG4V5Wp6S7S/JRWeUWerMUE5JgHvANOjpD"
},
"s3":{
"s3SchemaVersion":"1.0",
"configurationId":"testConfigRule",
"bucket":{
"name":"sourcebucket",
"ownerIdentity":{
"principalId":"A3NL1KOZZKExample"
},
"arn":"arn:aws:s3:::sourcebucket"
},
"object":{
"key":"HappyFace.jpg",
"size":1024,
"eTag":"d41d8cd98f00b204e9800998ecf8427e",
"versionId":"096fKKXTRTtl3on89fVO.nfljtsv6qko"
}
}
}
]
}
I get the error message
Unable to resize sourcebucket/HappyFace.jpg and upload to
sourcebucketresized/resized-HappyFace.jpg due to an error:
PermanentRedirect: The bucket you are attempting to access must be
addressed using the specified endpoint. Please send all future
requests to this endpoint. END RequestId: 345345...
I have changed the bucket name, eTag and image name. Do I need to change something else? My region are correct. Do I need to edit "principalId"? Where can I find it?
What is wrong.
In my case, the problem was the bucket region. In the example "us-east-1" is used, but my bucket is on "eu-west-1", so i had to change 2 things:
"awsRegion":"eu-west-1", in lambda test file
set region in my nodejs lambda code: AWS.config.update({"region": "eu-west-1"})
And of course you still need to set following values in in lambda test file:
name: 'your_bucket_name_here',
arn: 'arn:aws:s3:::your_bucket_name_here'
After this modifications it worked as expected
Your problem is about "endpoint".You must change "arn":"arn:aws:s3:::sourcebucket" to "arn":"arn:aws:s3:::(name_of_your_bucket)". Same to "name":"sourcebucket" to "name":"(name_of_your_bucket)".
In order to avoid more problems you must upload a jpg called HappyFace.jpg to your bucket or change in s3 put Test object code.
Regards
Try use this updated format (Please carefully configure the key, bucket name,arn and awsRegion to your own settings):
{
"Records": [
{
"eventVersion": "2.0",
"eventTime": "1970-01-01T00:00:00.000Z",
"requestParameters": {
"sourceIPAddress": "127.0.0.1"
},
"s3": {
"configurationId": "testConfigRule",
"object": {
"eTag": "0123456789abcdef0123456789abcdef",
"sequencer": "0A1B2C3D4E5F678901",
"key": "HappyFace.jpg",
"size": 1024
},
"bucket": {
"arn": "arn:aws:s3:::myS3bucket",
"name": "myS3bucket",
"ownerIdentity": {
"principalId": "EXAMPLE"
}
},
"s3SchemaVersion": "1.0"
},
"responseElements": {
"x-amz-id-2": "EXAMPLE123/5678abcdefghijklambdaisawesome/mnopqrstuvwxyzABCDEFGH",
"x-amz-request-id": "EXAMPLE123456789"
},
"awsRegion": "us-east-1",
"eventName": "ObjectCreated:Put",
"userIdentity": {
"principalId": "EXAMPLE"
},
"eventSource": "aws:s3"
}
]
}