Is there a way to send the data from a Kinesis Data Firehose Delivery Stream that uses the Agent Event Streams to an SNS topic which then triggers a lambda? I know you can send data from SNS to firehose via subscribing a topic to it, but what about vice versa?
Currently I have a Kinesis Firehose Delivery Stream setup that sends data to S3, and my goal is to use this data to trigger a lambda that will then a message to SNS and then Microsoft teams...any help or just general direction would be really appreciated, thanks!
Related
How to create in AWS SQS something like 'direct' exchange in RabbitMQ:
1 message -> N receivers queues.
Each client app connects to server and creates its own queue, a publisher sends one messages to exchange (direct+routing key) and its sent to all N queues, then each user reads its own queue and the queue is emptied.
This can be done with a 'fan-out' pattern combining Amazon SNS and Amazon SQS:
Create all desired Amazon SQS queues
Create an Amazon SNS Topic
Subscribe all of the Amazon SQS queues to the Amazon SNS Topic
Send a message to the Amazon SNS topic -- this will be sent to all subscribing queues. Each queue will have its own copy of the message.
Make sure you use Amazon SNS raw message delivery to preserve the format of the initial message as it goes from Amazon SNS to the SQS queues.
See also: How to Fan-Out to Different SQS Queues Using SNS Message Filtering | by Lorenz Vanthillo | Better Programming
I would like to trigger on received SQS message a mechanism where it would pass the content of this message into other AWS services like DynamoDB, Kinesis, SNS.
Can it be done right now? Currently the only event I was able to capture from SQS in EventBridge was on SQS queue attribute change. If it is possible, can I manipulate this message before passing it to other streams?
In case it is not possible what are the alternatives? Lambda function which would trigger on received SQS message?
Yes you can trigger the lambda function when the new message recived in the SQS. Do processing on the message and then forward it to next service. If you fanout that message to multiple services then you can use SNS or if just want a particular targeted service then you can use Eventbridge too. I hope this helps.
I would like to know if it's possible to persist all unacknowledged messages from an SNS topic to an S3 file given a certain time window. These messages don't necessarily need to follow the original order in the S3 file, a timestamp attribute is enough.
If all you want is to save all messages published to your SNS topic in an S3 bucket, then you can simply subscribe to your SNS topic the AWS Event Fork Pipeline for Storage & Backup:
https://docs.aws.amazon.com/sns/latest/dg/sns-fork-pipeline-as-subscriber.html#sns-fork-event-storage-and-backup-pipeline
** Jan 2021 Update: SNS now supports Kinesis Data Firehose as a native subscription type. https://aws.amazon.com/about-aws/whats-new/2021/01/amazon-sns-adds-support-for-message-archiving-and-analytics-via-kineses-data-firehose-subscriptions/
There is no in-built capability to save messages from Amazon SNS to Amazon S3.
However, this week AWS introduced Dead Letter Queues for Amazon SNS.
From Amazon SNS Adds Support for Dead-Letter Queues (DLQ):
You can now set a dead-letter queue (DLQ) to an Amazon Simple Notification Service (SNS) subscription to capture undeliverable messages. Amazon SNS DLQs make your application more resilient and durable by storing messages in case your subscription endpoint becomes unreachable.
Amazon SNS DLQs are standard Amazon SQS queues.
So, if Amazon SNS is unable to deliver a message, it can automatically send it to an Amazon SQS queue. You can later review/process those failed messages. For example, you could create an AWS Lambda function that is triggered when a message arrives in the Dead Letter Queue. The function could then store the message in Amazon S3.
I have requirement to publish messages from SNS to kinesis. I have found that, it is not possible directly by subscribing same as SNS/SQS. I will need to write lambda to fetch from SNS and publish it to kinesis.
Is there any other way to publish records from SNS to kinesis directly?
Thanks
Amazon SNS is a publish/subscribe model.
Messages sent to SNS can be subscribed from:
http/s: delivery of JSON-encoded message via HTTP POST
email: delivery of message via SMTP
email-json: delivery of JSON-encoded message via SMTP
sms: delivery of message via SMS
sqs: delivery of JSON-encoded message to an Amazon SQS queue
application: delivery of JSON-encoded message to an EndpointArn for a mobile app and device.
lambda: delivery of JSON-encoded message to an AWS Lambda function.
Other options: See Otavio's answer below!
See: Subscribe - Amazon Simple Notification Service
Of these, the only ones that could be used to send to Amazon Kinesis would be to use AWS Lambda. You would need to write a Lambda function that would send the message to a Kinesis stream.
To clarify: Your Lambda function will not "fetch from SNS". Rather, the Lambda function will be triggered by SNS, with the message being passed as input. Your Lambda function will then need to send the message to Kinesis.
Your only other alternative is to change the system that currently sends the message to SNS and have it send the message to Kinesis instead.
Good news! As of January 2021, Amazon SNS has added support for message archiving and analytics via Kinesis Data Firehose subscriptions. You can now load SNS messages into S3, Redshift, Elasticsearch, MongoDB, Datadog, Splunk, New Relic, and more. The SNS documentation has the details.
I know I can configure an Amazon S3 bucket to publish events to a SQS topic and to a SNS topic.
But, is it possible to configure the bucket to publish the event to SQS first, and then, when the message has been sent to SQS, have the bucket publish the event to SNS (kind of publish these events synchronously)?
An Amazon S3 bucket can publish a notification to one of:
Amazon Simple Notification Service (SNS)
Amazon Simple Queue Service (SQS)
AWS Lambda
However, SNS can also send a message to SQS. (More accurately, SQS can be added as a subscriber to an SNS topic).
Therefore, you could choose to send the event to SNS, which can on-send the event to an SQS queue. This is a good way to "fork" the event, sending it to multiple SNS subscribers.