Is it possible to receive notifications in a .NET Core application about bucket/object creation/deletion?
How to do it?
S3 bucket can generate SNS and SQS event notifications as well as trigger Lambda function on misc. events. Go to Bucket Properties->Events.
In your .NET code you'll need to react to these events, for instance by receiving SQS messages.
Amazon S3 Events can send a notification to:
An AWS Lambda function (Trigger): Does not appear relevant since your code is running elsewhere.
An Amazon SQS queue (Pull): Your application could regularly poll the Amazon SQS queue to retrieve a message, then act on that message.
An Amazon SNS topic (Push): Your application could subscribe to the Amazon SNS topic to receive the message via an HTTP endpoint. For example, this could point to your back-end web server.
If your application has a web server that is accessible from the Internet, then use the SNS push. Otherwise, your app will need to poll the SQS queue.
Related
I am guessing the answer would be a big No. But is there a way to publish SQS messages straight to an S3 Bucket.
I know the pattern is SQS -> Lambda ->S3. I was wondering is there a way to just publish from a SQS straight to a S3 Bucket.
Amazon SQS does not 'publish' messages.
Instead, apps can SendMessage() to an Amazon SQS queue, and then apps can request messages by calling ReceiveMessage(). Once they have finished processing the message, they call DeleteMessage().
I would like to know if it's possible to persist all unacknowledged messages from an SNS topic to an S3 file given a certain time window. These messages don't necessarily need to follow the original order in the S3 file, a timestamp attribute is enough.
If all you want is to save all messages published to your SNS topic in an S3 bucket, then you can simply subscribe to your SNS topic the AWS Event Fork Pipeline for Storage & Backup:
https://docs.aws.amazon.com/sns/latest/dg/sns-fork-pipeline-as-subscriber.html#sns-fork-event-storage-and-backup-pipeline
** Jan 2021 Update: SNS now supports Kinesis Data Firehose as a native subscription type. https://aws.amazon.com/about-aws/whats-new/2021/01/amazon-sns-adds-support-for-message-archiving-and-analytics-via-kineses-data-firehose-subscriptions/
There is no in-built capability to save messages from Amazon SNS to Amazon S3.
However, this week AWS introduced Dead Letter Queues for Amazon SNS.
From Amazon SNS Adds Support for Dead-Letter Queues (DLQ):
You can now set a dead-letter queue (DLQ) to an Amazon Simple Notification Service (SNS) subscription to capture undeliverable messages. Amazon SNS DLQs make your application more resilient and durable by storing messages in case your subscription endpoint becomes unreachable.
Amazon SNS DLQs are standard Amazon SQS queues.
So, if Amazon SNS is unable to deliver a message, it can automatically send it to an Amazon SQS queue. You can later review/process those failed messages. For example, you could create an AWS Lambda function that is triggered when a message arrives in the Dead Letter Queue. The function could then store the message in Amazon S3.
What's the easiest way to save/log every message published on a AWS SNS topic? I thought there might be a magic setting to automatically push them to S3 or a database, or maybe a database service supporting the HTTP destination automatically, but doesn't seem to be the case. Maybe it needs to be done via a Lambda function?
The purpose is just for basic diagnostics and debugging while setting up some SNS publishing. I don't really care about high scale or fast querying, just want to log and perform basic queries on all the activity for a few minutes at a time.
You can setup a trigger to push your SNS messages to SQS queue. Push is automatic and does not require any code.
According to the docs, SNS can publish to:
http – delivery of JSON-encoded message via HTTP POST
https – delivery of JSON-encoded message via HTTPS POST
email – delivery of message via SMTP
email-json – delivery of JSON-encoded message via SMTP
sms – delivery of message via SMS
sqs – delivery of JSON-encoded message to an Amazon SQS queue
application – delivery of JSON-encoded message to an EndpointArn for a mobile app and device.
lambda – delivery of JSON-encoded message to an AWS Lambda function.
So yes, a simple approach would be to trigger a lambda function to write to S3 or CloudWatch.
Can someone explain to me the advantage or disadvantage of using SNS -> Lambda vs. SNS -> SQS -> Lambda.
I'm looking to setup an architecture for pub/sub micro-service messaging, but having a queue in front of every Lambda seems excessive.
Unless something has changed, the question of whether to it makes more sense to deploy SNS → Lambda, or SNS → SQS → Lambda, is based on a premise with a significant flaw.
As is indicated in Supported Event Sources in the Lambda documentation, Lambda events can be sourced from S3, DynamoDB, Kinesis, SNS, SES, Cognito, CloudFormation, CloudWatch & Events (including Scheduled Events), AWS Config, Amazon Echo, and API Gateway.
And, of course, you can invoke them directly.
But SQS is not a supported Lambda event source.
Amazon SQS is a message queue service used by distributed applications to exchange messages through a polling model, and can be used to decouple sending and receiving components—without requiring each component to be concurrently available. By using Amazon SNS and Amazon SQS together, messages can be delivered to applications that require immediate notification of an event, and also persisted in an Amazon SQS queue for other applications to process at a later time.
Untill you don't want to decouple sending and receiving components and just want to achieve your use case in the question it will work in both case SNS- Lambda and SNS - SQS - Lambda.
I know I can configure an Amazon S3 bucket to publish events to a SQS topic and to a SNS topic.
But, is it possible to configure the bucket to publish the event to SQS first, and then, when the message has been sent to SQS, have the bucket publish the event to SNS (kind of publish these events synchronously)?
An Amazon S3 bucket can publish a notification to one of:
Amazon Simple Notification Service (SNS)
Amazon Simple Queue Service (SQS)
AWS Lambda
However, SNS can also send a message to SQS. (More accurately, SQS can be added as a subscriber to an SNS topic).
Therefore, you could choose to send the event to SNS, which can on-send the event to an SQS queue. This is a good way to "fork" the event, sending it to multiple SNS subscribers.