Send AWS S3 notification only once for many files - amazon-web-services

I have enabled the S3 notification create:* event for SNS sending to my email from AWS CLI in my Windows servers.
However, there are many notification emails received from the SNS after I copied 10000+ local files such as xml, html, jpg to the S3. Because of creating the folder or the files, it is going to send the event with per creating file request to SNS.
I search the topic discussed in Edit event notification prefix and suffix and I think it just partially works on folder and file.
How can just receive one notification email that indicates it's starting the copy job?

How can just receive one notification email that indicates it's starting the copy job?
You can't. This is not how S3 notifications work. You would have to develop custom solution to that. Maybe instead of SNS, have a lambda function to get triggered. The function would somehow verify if this is the first file being uploaded, and send out email only for that file.

Related

Aws S3 event notification mail for différent user

we have an aws account with some s3 buckets that are used by several people. I would like to know if there is a way, and if so which ones, to configure an sns notification by email when a restore from s3 glacier deep archive is completed so that this email goes to the person who initiated the retrieval ?
Yes, there in fact there are multiple ways to accomplish this. What you would want to do is to create an Event Notification for your bucket. This can be found in the console by going into the bucket Properties and pressing the Create Event Notification button.
You will be prompted to select the type of action for which you want to send notifications. You would want to select s3:ObjectRestore:Completed:
You will need destination for your events. For sending emails, one option would be to create an SNS topic and use that as a destination. This topic can be integrated with Amazon Simple Email Service (SES) which will be able to send mails.
Other option would be to select a Lambda function as destination and use that to send emails and do other automations.
the problem is more, how as an admin can I configure the sending of mail to the user who initiated the restoration and only to him
In this case you would want to enable Event Bridge integration for your bucket. If you enable it, it will send events which contain requester who initiated the restore for an object (see event full event format here). Event Bridge can be also integrated with SNS or Lambda.

AWS S3 Event Notification for upload start

I have tried setting up S3 event notifications via SNS topic & was able to successfully get event notifications when objects are created. However in my use case we have large file uploads from the apps that we don't control. These uploads take time. We want to get notified when upload starts (in progress) as well.
I was not able to find any event type that corresponds to upload start!
For large files multipart uploads are used so we get "multipart upload complete" event but still we don't have a clue about when the upload started!
Is there any other way to detect the uploads (start) on AWS S3?
You can create an Amazon CloudWatch Events rule that triggers on CreateMultiPartUpload and sends a message to an Amazon SNS topic:
From CreateMultipartUpload - Amazon Simple Storage Service:
This action initiates a multipart upload and returns an upload ID.

How to send a S3 bucket notification on SNS with a presigned url to new object uploaded?

I would like to configure a bucket notification on an S3 bucket so that every time a notification from a file being uploaded, it sends a message to an SNS topic with a pre signed url in the message so that file can be downloaded when consumed from an email.
File gets uploaded -> notification sent to SNS containing presigned url to recently updated object -> to SNS topic -> to email -> to user
I have tried setting up a topic, but couldn't get a presigned url inside the message, is there a way of setting this up without using a lambda or queue or other AWS services as intermediates?
You can't do this without additional services.
S3 is a storage service, as such it doesn't to much computation if any.
The simplest way would be to create a Lambda that gets triggered on S3 put object, generates the presigned url and sends a message to an SQS or SNS topic.
You'll need to use a Lambda for this unfortunately, it shouldn't be too much additional work.

SNS with file attachment

I have SNS notification for a S3 bucket. I wanted to push that SNS notification with attachment
We have a S3 bucket namely like SDD-XXX-YYY. Once if any file is getting arrived in that bucket, we have configured SNS to receive email notification.
Now we need email notification with content of the file which we are receiving in S3 or as attachment in our email.
Example: We are receiving some "error.log" file S3, here we need to push the error.log in email or else content of error.log file.
Please help me to achieve it.
Do we need to write lambda here? or we can manage it in SNS itself?
The SNS S3 notification will only give you information about the newly created object.
You can see the structure of the event in Event Message Structure.
If you want to send the file via email you will have to write this logic in a Lambda function: you will use the information from the event (bucket and key) to download the object and then send it via email.
This function could subscribe to SNS or directly to S3.

Get Notified when upload is completed in Amazon S3 bucket

Is there a way by which I can get notified when a upload is completed in S3 Bucket? The requirement is that I need to provide link to users after uploading of a video is complete in the bucket. By default now I provide link after 30 minutes of start of video, whether video takes 5 minutes to upload or 40 minutes. So is there any way like any API that provides information that the upload has been completed?
Notifications can be triggered in Amazon S3 when any of the following occur:
s3:ObjectCreated:*
s3:ObjectCreated:Put
s3:ObjectCreated:Post
s3:ObjectCreated:Copy
s3:ObjectCreated:CompleteMultipartUpload
s3:ObjectRemoved:*
s3:ObjectRemoved:Delete
s3:ObjectRemoved:DeleteMarkerCreated
s3:ReducedRedundancyLostObject
Notifications can be sent via three destinations:
Amazon Simple Notification Service (SNS), which in-turn can send notifications via email, HTTP/S endpoint, SMS, mobile push notification
Amazon Simple Queueing Service (SQS)
Amazon Lambda (not currently available in all regions)
See: Configuring Amazon S3 Event Notifications
The most appropriate choice depends on your programming preference and how your app is written:
Use SNS to push to an HTTP endpoint to trigger some code in your app
Write some code to periodically check an SQS queue
Write a Lambda function in Node.js or Java
Once triggered, your code would then need to identify who uploaded the video, retrieve their user details, then send them an email notification. This would be easiest if you control the key (filename) of the object being uploaded, since this will assist in determining the user to notify.
You can use Amazon Lambda to post a message to Amazon SNS (or notify you any other way) when a file is uploaded to S3.
Setup an S3 trigger to your Lambda function. See this tutorial: http://docs.aws.amazon.com/lambda/latest/dg/walkthrough-s3-events-adminuser.html
Inside your Lambda function, send out your notification. You can use SNS, SES, SQS, etc.
There is no direct method that can tell that whether the upload is complete or not in S3 bucket. You can do a simple thing which I have followed after lot of research and it is working correctly.
Follow this link and read the size of file after every 30 seconds or so as per your requirement when the file size has not changed for two simultaneous readings once again check the size for surety because it might be due to network congestion that size might not have changed for two simultaneous readings.