I need to subscribe an AWS SNS topic to a HTTP endpoint that I do not control, but I am blocked by the anti-abuse measure in SNS that requires the subscribed endpoint to confirm the subscription. This would require modifying the web service.
Given that I don't have the ability to modify the web service, that is probably the end of this discussion. I will need to find another solution. But, I would like to explain my scenario in case there is a workaround:
The messages we are sending are logs.
It works like this:
log_source -> sns_topic
sns_topic -> log_endpoint*
log_endpoint* -> log_data_store
* means it is not controlled by us
AWS SNS is such a wonderful solution to our problem, it is a shame that we cannot leverage it because of this anti-abuse measure.
I believe we are stuck because we have no control with how AWS will form the magic confirmation URL, nor how the log_endpoint will process it.
If there is no workaround, perhaps there is an analogous AWS service that I could leverage?
Related
I need to create an API gateway to consume messages from an SNS. I see that there are some questions on the same, like this, which hasn't been answered directly.
The main issue I face is : How to make the API gateway subscribe to the SNS?
SNS supports HTTP/HTTPS endpoint subscriptions. So I don't see a reason why you could not subscribe API gateway https endpoint to SNS this way.
As Marcin has mentioned, it can be done. Mentioning the steps I used:
Create SNS, and a lambda
Whatever the language is, make sure to print the event in the log. (For JavaScript, use console.log(event), and for Python use print(event) etc)
Create an API Gateway (REST API), select a "New API", and create it.
In the API create a POST method, select lambda, and make sure to use it as a proxy.
Copy the trigger HTTPS link
Add a HTTPS subscription in the SNS, and use the trigger link here.
The confirmation would be pending now.
Go to the lambda, inside it's monitoring section, go to cloud watch logs.
Inside the logs, you will find the event object printed. Inside it, look for SubscribeURL, along which the URL would be mentioned.
Copy this URL, and go back to the SNS. Select your subscription and use the "Confirm Subscription" option, and paste this link there.
The status should come as "Confirmed" now.
As you know, we can publish to an SNS events via those channels:
HTTP
HTTPS
Email
Email-JSON
Amazon SQS
AWS Lambda
Platform application endpoint
SMS
Now let's say I want to test a published messsage.
According to the list, I can use a "request catcher service" for Http/s requests, but sometimes the info is too confidential to be published to third party services.
The email solution is too side effecty. Also writing lambda involves writing code.
Connecting an SQS to inspect messages is causing me to create a special SQS just for testing...
Question:
Is there any option to submit to an SNS and to inspect the message in logs or something like that? Or am I limited only to this list? I'm talking only for dev purposes.
I have a lambda function that notifies an SNS topic.
It currently sends an email each time I invoke the function so the lambda, the SNS piece and the email subscription are all working ok.
Now I want a Slack Integration.
I've created a channel called aws_int but I can't get it to receive the notification.
I am stuck on the need to 'verify the subscription'. I don't see how I would do this in the https connection to slack.
I don't understand how to do it manually using the console in aws because all the documentation keeps referring to using the 'Subscription Confirmation URL'
I can't find what that is.
but I cannot find what that is.
I found a post talking about the JSON being invalid: Confirming AWS SNS Topic Subscription for Slack Webhook but I don't understand how I would use that, maybe I could use Postman but I am not sure exactly how to format that POST.
So how can I integrate Slack and aws for an SNS topic, specifically how can I do the confirmation (which is required in order to activate it). This has to be a common need!
If I use the hook URL I get
Everything I am trying is just stacking up more Pending Confirmations, which btw I cannot delete...
Here's my postman attempt...
You really DON'T need Lambda. Just SNS and SLACK are enough.
I found a way to integrate AWS SNS with slack WITHOUT AWS Lambda or AWS chatbot. With this approach you can confirm the subscription easily.
Follow the video which show all the step clearly.
https://www.youtube.com/watch?v=CszzQcPAqNM
Steps to follow:
Create slack channel or use existing channel
Create a work flow with selecting Webhook
Create a variable name as "SubscribeURL". The name
is very important
Add the above variable in the message body of the
workflow Publish the workflow and get the url
Add the above Url as subscription of the SNS You will see the subscription URL in the
slack channel
Follow the URl and complete the subscription
Come back to the work flow and change the "SubscribeURL" variable to "Message"
The publish the
message in SNS. you will see the message in the slack channel.
It doesn't look like there is a way to confirm an SNS subscription to a Slack endpoint (email, webhook, whatever). If you want to use SNS as a decoupling layer, you'll need to add a lambda into the system, which can handle the confirmation process.
Lambda -> webhook -> SNS -> Lambda -> webhook -> Slack
Confirming AWS SNS Topic Subscription for Slack Webhook has some more information about this.
I would recommend just using the Lambda to send a message to slack, either calling another dedicated lambda, or just internally to the original one. There's a good tutorial here. If you really want to use SNS, then you could use your original lambda to trigger SNS, then have a new slack-dedicated lambda subscribe to the SNS topic.
I've created SNS topic
I've created API Gateway endpoint that invokes Lambda function
I've created topic HTTPS subscription that points to API Gateway endpoint
Problem: everything works fine when AUTH=none, but when i enabled AUTH=AWS_IAM, neither subscription nor messages are delivered to my lambda. They also wont show up in Lambda OR Gateway cloudwatch logs as it's usually the case with authentication errors.
Questions:
What's the identity delivered by HTTPS endpoint to AWS_IAM so it doesn't allows it ( my first thought was to relay SNS posters token but it doesn't seem be the case )
I couldn't find any way to associate HTTPS endpoint with any identity, is there a way?
There are lots of information about delivering SNS to SQS or Gateway to SNS, but couldn't find any information about achieving what i try to do.
Is there any method to debug AWS_IAM authentication problems? Documentation i've seen advices to "check priviliges" which is something i've been doing for many hours but i have no more ideas.
I'd be glad to hear any ideas from you, thanks.
As you may have seen in the docs, SNS can only do Basic/Digest Auth http://docs.aws.amazon.com/sns/latest/dg/SendMessageToHttp.html
There is a section in the docs about verifying the validity of the message but that is code you'd have to write yourself or lift from one of the SNS SDKs on the backend. There really isn't any way to get SNS to sign the request with AWS SigV4, unfortunately.
Why don't you let the Lambda function subscribe directly to the SNS topic (without going through API Gateway)?
That should be straightforward: https://docs.aws.amazon.com/sns/latest/dg/sns-lambda.html
Here is the complete link which will help you in solving your authentication problem. https://aws.amazon.com/premiumsupport/knowledge-center/iam-authentication-api-gateway/
If it's an "Check privileges" issue, then your IAM user doesn't have any sufficient access to the resources to make any changes.
I have a lambda function i'd like to invoke from the client-side. I was going to use the API Gateway, but it occurred to me that the queuing SNS affords might be handy.
After researching, it appears the only way to publish to SNS thru the Javsacript SDK is auth thru google/facebook or AWS Cognito. I'd like users (more specifically, events) to be able to push w/o auth'ng, so that's not an option.
The last option is hard-coding an AWS key. This is pretty explicitly discouraged in the docs, but after looking into it, it looks like I can create security provisions for a specific key and limit it to publishing only to one topic.
In other words, it'd ostensibly mimic a REST API, wouldn't it?
The only drawback I can think of is malicious spamming of the SNS. I know AWS API allows for rate-throttling, but couldn't find something similar on SNS.
So, 2 related question:
Is there a way to prevent malicious spam to an SNS topic?
are there other drawbacks to using SNS instead of an AWS API for invoking lambdas?
What queueing are you wanting to get from an SNS topic? I think you may be confusing SNS with SQS.
I see no advantage to using SNS->Lambda in this instance versus API->Lambda. I do however see several drawbacks to using SNS in this instance as it adds an unnecessary complication, as well as opens up unnecessary security risks.
You literally get no advantage to using SNS here, while you get several advantages to using API Gateway such as rate limiting and API key support. Not to mention that API Gateway endpoints are much easier to access from the browser than SNS topics. This is API Gateway's intended use, why try to hack together some method using SNS and hard coded AWS keys?