What would be the most efficient way to EMIT a POST requests (a webhook precisely) from AWS on a daily basis ?
Right away, I started by simply creating a cloudWatch rule with an event schedule CRON that would trigger an SNS publication "every day at 18h", then I created an SNS topic "AlertMyWebhook" with all POST endpoints as subscribers of the topic.
But.. SNS requires me to CONFIRM subscription of each endpoints... which I can't by definition, since the https endpoint is not mine (webhook = HOOK into someone ELSE'S WEBapp).
So now I am starting to think crazy stuff like having a dynamoDB table to store all webhooks endpoint URL, coupled with a lambda function to read the table, take each https endpoints, and send a POST request to them...
Frankly speaking: that doesn't make any sense to me.
Is there any way to avoid SNS confirmation ?
If not, how on earth would you do to "trigger a POST every day at 18h" without creating a monolithic-like architecture ?
AWS SNS and Lambda functions are integrated with each other so you can subscribe a Lambda function to your topic. When a message is posted to that topic the subscribed Lambda function is invoked with the Payload(published message).
Using this Payload as input for the Lambda function trigger the POST requests for the endpoints. A good way to do is make all the HTTPS POST endpoints as Environment variables in Lambda. So there is no code change in the Lambda function whenever a new POST endpoint need to be added as the Subscription endpoints. For more of How to integrate AWS SNS and Lambda look here.
Invoking Lambda functions using Amazon SNS notifications
The sample NodeJS code to invoke the POST request
AWS Lambda HTTP POST Request
Related
I have a lambda function that notifies an SNS topic.
It currently sends an email each time I invoke the function so the lambda, the SNS piece and the email subscription are all working ok.
Now I want a Slack Integration.
I've created a channel called aws_int but I can't get it to receive the notification.
I am stuck on the need to 'verify the subscription'. I don't see how I would do this in the https connection to slack.
I don't understand how to do it manually using the console in aws because all the documentation keeps referring to using the 'Subscription Confirmation URL'
I can't find what that is.
but I cannot find what that is.
I found a post talking about the JSON being invalid: Confirming AWS SNS Topic Subscription for Slack Webhook but I don't understand how I would use that, maybe I could use Postman but I am not sure exactly how to format that POST.
So how can I integrate Slack and aws for an SNS topic, specifically how can I do the confirmation (which is required in order to activate it). This has to be a common need!
If I use the hook URL I get
Everything I am trying is just stacking up more Pending Confirmations, which btw I cannot delete...
Here's my postman attempt...
You really DON'T need Lambda. Just SNS and SLACK are enough.
I found a way to integrate AWS SNS with slack WITHOUT AWS Lambda or AWS chatbot. With this approach you can confirm the subscription easily.
Follow the video which show all the step clearly.
https://www.youtube.com/watch?v=CszzQcPAqNM
Steps to follow:
Create slack channel or use existing channel
Create a work flow with selecting Webhook
Create a variable name as "SubscribeURL". The name
is very important
Add the above variable in the message body of the
workflow Publish the workflow and get the url
Add the above Url as subscription of the SNS You will see the subscription URL in the
slack channel
Follow the URl and complete the subscription
Come back to the work flow and change the "SubscribeURL" variable to "Message"
The publish the
message in SNS. you will see the message in the slack channel.
It doesn't look like there is a way to confirm an SNS subscription to a Slack endpoint (email, webhook, whatever). If you want to use SNS as a decoupling layer, you'll need to add a lambda into the system, which can handle the confirmation process.
Lambda -> webhook -> SNS -> Lambda -> webhook -> Slack
Confirming AWS SNS Topic Subscription for Slack Webhook has some more information about this.
I would recommend just using the Lambda to send a message to slack, either calling another dedicated lambda, or just internally to the original one. There's a good tutorial here. If you really want to use SNS, then you could use your original lambda to trigger SNS, then have a new slack-dedicated lambda subscribe to the SNS topic.
I have tried to do some R&D but i couldn't find anything useful the only thing that i found is lambda functions is the only way. I want to write a simple application that execute when the Simple Queue Service receive any message , but i couldn't find a way to do that till now, since i don't want to use lambda. for example if i receive some message on Simple Queue Service and while receiving each messages i can trigger a event that is not lambda but instead of that any HTTP-request.
I think your choices are:
use lambda (which you said you didn't want to use, but its probably the best solution)
use your own app running on ec2 or even on premise to consume the message and invoke the http endpoint
use SNS instead of SQS for message delivery - SNS supports http endpoints.
You can use Amazon SNS to send notification messages to one or more
HTTP or HTTPS endpoints. When you subscribe an endpoint to a topic,
you can publish a notification to the topic and Amazon SNS sends an
HTTP POST request delivering the contents of the notification to the
subscribed endpoint. When you subscribe the endpoint, you select
whether Amazon SNS uses HTTP or HTTPS to send the POST request to the
endpoint.
from here: https://docs.aws.amazon.com/sns/latest/dg/sns-http-https-endpoint-as-subscriber.html
A short answer to your question is No, (until today)
Let me tell you the sineros i faced.In general Queue triggering lambda is widely used and for that u have to make sure about proper concurrency (minimum 5) in place and also database I/O if u are performing and any DB calls . But I've a scenario where we cannot use "lambda as a triggering service" as our DB is onprem "ORACLE" so the choices are .
Push to "SNS" and make http "what ever applies"(to a container we have custom Kubernetes routed through NLB ).Also make sure you push a batch of messages as it might make more http noise.
2.Poll the queue and perform the operations.
SQS triggering => Lambda and lambda invoking state machine (step functions)
I have a requirement where I need to send some inbound http query parameters from api A by publishing it to SNS and other api B is subscribed to sns where it should get the query params.
Is there any way I can acheive it?
Note: I have a fixed message structure when I publish to SNS which I cannot change, since I have other endpoints who are already subscribing to SNS, depend on it.
As Michael mentioned, we can use lambda. but I do not want to use another service provided by aws. I want to acheive the same using SNS only. I heard about SNS message attributes, but not sure whether we can acheive the same using it.
Thanks.
SNS doesn't support customizing the HTTP transaction or otherwise modifying the message being delivered... but SNS does support subscriptions that target a Lambda function.
This means you can write a Lambda function that parses the SNS event payload... and then, instead of SNS contacting the target system over HTTPS, your Lambda function makes the HTTPS request to the target endpoint, customized as required. If the HTTPS request fails, or the endpoint returns an error, the Lambda function should throw an exception, so that Lambda can know to retry.
https://docs.aws.amazon.com/lambda/latest/dg/invoking-lambda-function.html#supported-event-source-sns
As it currently stands AWS SNS provides functionality for retrial(Linear, Geometric and Exponential backoff) with HTTP/HTTPS endpoints in case of a 5XX response returned from the endpoint.
Because of this my application architecture changes and I forcefully need to insert a API gateway between my SNS and Lambda so that in case of a failure I can return a 5XX status from the API gateway and utilise the retrial functionality of SNS.
But there is nothing mentioned for retrial mechanism with AWS lambda. Is there any way I can use the SNS retrial facilities for non-HTTP based subscriptions?
Thanks
After a couple of hours of debugging and going through AWS documentation it seems that there is currently no way of getting exponential back of from AWS SNS for anything else apart from HTTP/HTTPS sources.
You can checkout the this.
As quoted in the documentation:
When a user calls the SNS Publish API on a topic that your Lambda
function is subscribed to, Amazon SNS will call Lambda to invoke your
function asynchronously. Lambda will then return a delivery status. If
there was an error calling Lambda, Amazon SNS will retry invoking the
Lambda function up to three times. After three tries, if Amazon SNS
still could not successfully invoke the Lambda function, then Amazon
SNS will send a delivery status failure message to CloudWatch.
Since there is a async invocation of the Lambda SNS will not care what the exit status of the lambda is. Hence, from the point of view of SNS, a successful invocation of the lambda is success enough and will not provide a failure event, hence no customised back off.
For now it seems, adding an HTTP endpoint is the only option.
I've created SNS topic
I've created API Gateway endpoint that invokes Lambda function
I've created topic HTTPS subscription that points to API Gateway endpoint
Problem: everything works fine when AUTH=none, but when i enabled AUTH=AWS_IAM, neither subscription nor messages are delivered to my lambda. They also wont show up in Lambda OR Gateway cloudwatch logs as it's usually the case with authentication errors.
Questions:
What's the identity delivered by HTTPS endpoint to AWS_IAM so it doesn't allows it ( my first thought was to relay SNS posters token but it doesn't seem be the case )
I couldn't find any way to associate HTTPS endpoint with any identity, is there a way?
There are lots of information about delivering SNS to SQS or Gateway to SNS, but couldn't find any information about achieving what i try to do.
Is there any method to debug AWS_IAM authentication problems? Documentation i've seen advices to "check priviliges" which is something i've been doing for many hours but i have no more ideas.
I'd be glad to hear any ideas from you, thanks.
As you may have seen in the docs, SNS can only do Basic/Digest Auth http://docs.aws.amazon.com/sns/latest/dg/SendMessageToHttp.html
There is a section in the docs about verifying the validity of the message but that is code you'd have to write yourself or lift from one of the SNS SDKs on the backend. There really isn't any way to get SNS to sign the request with AWS SigV4, unfortunately.
Why don't you let the Lambda function subscribe directly to the SNS topic (without going through API Gateway)?
That should be straightforward: https://docs.aws.amazon.com/sns/latest/dg/sns-lambda.html
Here is the complete link which will help you in solving your authentication problem. https://aws.amazon.com/premiumsupport/knowledge-center/iam-authentication-api-gateway/
If it's an "Check privileges" issue, then your IAM user doesn't have any sufficient access to the resources to make any changes.