I have a requirement where I need to send some inbound http query parameters from api A by publishing it to SNS and other api B is subscribed to sns where it should get the query params.
Is there any way I can acheive it?
Note: I have a fixed message structure when I publish to SNS which I cannot change, since I have other endpoints who are already subscribing to SNS, depend on it.
As Michael mentioned, we can use lambda. but I do not want to use another service provided by aws. I want to acheive the same using SNS only. I heard about SNS message attributes, but not sure whether we can acheive the same using it.
Thanks.
SNS doesn't support customizing the HTTP transaction or otherwise modifying the message being delivered... but SNS does support subscriptions that target a Lambda function.
This means you can write a Lambda function that parses the SNS event payload... and then, instead of SNS contacting the target system over HTTPS, your Lambda function makes the HTTPS request to the target endpoint, customized as required. If the HTTPS request fails, or the endpoint returns an error, the Lambda function should throw an exception, so that Lambda can know to retry.
https://docs.aws.amazon.com/lambda/latest/dg/invoking-lambda-function.html#supported-event-source-sns
Related
I'm attempting to subscribe an SNS topic to a HTTPS endpoint I own. I'm reading the docs on how to process incoming messages from SNS and how the subscription confirmation needs to be done. I see two methods of confirmation:
https://docs.aws.amazon.com/sns/latest/dg/sns-http-https-endpoint-as-subscriber.html - Using the subscribeURL. We can perform a HTTP get request on the "SubscribeURL" attribute value and that would confirm the subscription.
Calling the ConfirmSubscription API - We pass the SNS Topic ARN and the token received when SNS sends a confirm subscription message to the SNS endpoint.
I'm trying to understand what's the difference between the two methods. The most obvious one to me was this - The choice of using the API will require AWS credentials since the request needs to be signed. But seems like the same call will succeed with just the HTTP GET request?
What's the best practice out there (if any) and/or which method is the one being followed by other folks using AWS/SNS?
There isn't a difference -- these two alternatives are in fact the same thing.
The SubscribeURL attribute is a pre-constructed (by the service) link to the ConfirmSubscription action on the SNS API endpoint.
The API accepts GET or POST. No signature is required in this case.
This call requires an AWS signature only when the AuthenticateOnUnsubscribe flag is set to "true".
https://docs.aws.amazon.com/sns/latest/api/API_ConfirmSubscription.html
Before SNS will talk to an endpoint, you need to prove that you control that endpoint. So your options are to write some code that can do it automatically (most of the SDKs support this) or to capture the token, and by returning it via the API call prove that you control it.
This is a one-time procedure, so you do not need to deploy any AWS credentials to your API endpoint - you can do it from a different system.
We generally build the confirmation handler into the application.
I have tried to do some R&D but i couldn't find anything useful the only thing that i found is lambda functions is the only way. I want to write a simple application that execute when the Simple Queue Service receive any message , but i couldn't find a way to do that till now, since i don't want to use lambda. for example if i receive some message on Simple Queue Service and while receiving each messages i can trigger a event that is not lambda but instead of that any HTTP-request.
I think your choices are:
use lambda (which you said you didn't want to use, but its probably the best solution)
use your own app running on ec2 or even on premise to consume the message and invoke the http endpoint
use SNS instead of SQS for message delivery - SNS supports http endpoints.
You can use Amazon SNS to send notification messages to one or more
HTTP or HTTPS endpoints. When you subscribe an endpoint to a topic,
you can publish a notification to the topic and Amazon SNS sends an
HTTP POST request delivering the contents of the notification to the
subscribed endpoint. When you subscribe the endpoint, you select
whether Amazon SNS uses HTTP or HTTPS to send the POST request to the
endpoint.
from here: https://docs.aws.amazon.com/sns/latest/dg/sns-http-https-endpoint-as-subscriber.html
A short answer to your question is No, (until today)
Let me tell you the sineros i faced.In general Queue triggering lambda is widely used and for that u have to make sure about proper concurrency (minimum 5) in place and also database I/O if u are performing and any DB calls . But I've a scenario where we cannot use "lambda as a triggering service" as our DB is onprem "ORACLE" so the choices are .
Push to "SNS" and make http "what ever applies"(to a container we have custom Kubernetes routed through NLB ).Also make sure you push a batch of messages as it might make more http noise.
2.Poll the queue and perform the operations.
SQS triggering => Lambda and lambda invoking state machine (step functions)
What would be the most efficient way to EMIT a POST requests (a webhook precisely) from AWS on a daily basis ?
Right away, I started by simply creating a cloudWatch rule with an event schedule CRON that would trigger an SNS publication "every day at 18h", then I created an SNS topic "AlertMyWebhook" with all POST endpoints as subscribers of the topic.
But.. SNS requires me to CONFIRM subscription of each endpoints... which I can't by definition, since the https endpoint is not mine (webhook = HOOK into someone ELSE'S WEBapp).
So now I am starting to think crazy stuff like having a dynamoDB table to store all webhooks endpoint URL, coupled with a lambda function to read the table, take each https endpoints, and send a POST request to them...
Frankly speaking: that doesn't make any sense to me.
Is there any way to avoid SNS confirmation ?
If not, how on earth would you do to "trigger a POST every day at 18h" without creating a monolithic-like architecture ?
AWS SNS and Lambda functions are integrated with each other so you can subscribe a Lambda function to your topic. When a message is posted to that topic the subscribed Lambda function is invoked with the Payload(published message).
Using this Payload as input for the Lambda function trigger the POST requests for the endpoints. A good way to do is make all the HTTPS POST endpoints as Environment variables in Lambda. So there is no code change in the Lambda function whenever a new POST endpoint need to be added as the Subscription endpoints. For more of How to integrate AWS SNS and Lambda look here.
Invoking Lambda functions using Amazon SNS notifications
The sample NodeJS code to invoke the POST request
AWS Lambda HTTP POST Request
I have a AWS Lambda function (triggered by an API call via API Gateway) that publishes messages to a collection of related IoT devices using an IoT topic based on the parameters received (the topic being a serial number that uniquely identifies a specific IoT device). I do this using boto3's IoT data-plane client method publish(). Is there a way for me to have this same Lambda function subscribe to the same IoT topic so that it is able to receive the response from the IoT device?
I am aware that the prescribed way for an IoT message to trigger a Lambda is to create an IoT Rule Action that calls a Lambda, however this doesn't really work for me (as far as I can tell) because what I want is for the original Lambda (the one triggered by the external API call to API Gateway) to listen for the IoT device's response and send back a HTTP response to the external caller based on that; this doesn't seem achievable if it's a separate Lambda that gets triggered when the IoT device responds on its IoT topic. Any ideas on how to do this?
Why does it have to be the original lambda? I'm guessing there's some state you want to save; in which case you should either persist it or pass it in the messages.
Though not pretty, this enables me to do what I posted in my question: I added an IoT rule which flicks the IoT device's response on to a SQS queue; the Lambda (that was invoked by API Gateway to handle the request at the start) polls that SQS queue for a message which it then consumes, using it to construct and send a response back via API Gateway to the external client that sent the HTTP POST request. To be sure, this is not a great way to use AWS Lambdas, but it seems the only way available to use Lambda + API Gateway to hook up HTTP request/response with IoT publish/subscribe.
As it currently stands AWS SNS provides functionality for retrial(Linear, Geometric and Exponential backoff) with HTTP/HTTPS endpoints in case of a 5XX response returned from the endpoint.
Because of this my application architecture changes and I forcefully need to insert a API gateway between my SNS and Lambda so that in case of a failure I can return a 5XX status from the API gateway and utilise the retrial functionality of SNS.
But there is nothing mentioned for retrial mechanism with AWS lambda. Is there any way I can use the SNS retrial facilities for non-HTTP based subscriptions?
Thanks
After a couple of hours of debugging and going through AWS documentation it seems that there is currently no way of getting exponential back of from AWS SNS for anything else apart from HTTP/HTTPS sources.
You can checkout the this.
As quoted in the documentation:
When a user calls the SNS Publish API on a topic that your Lambda
function is subscribed to, Amazon SNS will call Lambda to invoke your
function asynchronously. Lambda will then return a delivery status. If
there was an error calling Lambda, Amazon SNS will retry invoking the
Lambda function up to three times. After three tries, if Amazon SNS
still could not successfully invoke the Lambda function, then Amazon
SNS will send a delivery status failure message to CloudWatch.
Since there is a async invocation of the Lambda SNS will not care what the exit status of the lambda is. Hence, from the point of view of SNS, a successful invocation of the lambda is success enough and will not provide a failure event, hence no customised back off.
For now it seems, adding an HTTP endpoint is the only option.