I have an API Gateway endpoint that takes the bodies of requests to it and places them in an SQS queue. The API Gateway has the ability to transform the request and add requester meta, like the user agent and ip address, to the message it eventually sends to SQS.
I'd like to change this from API Gateway to SNS so that the requester would publish to an SNS topic that feeds into SQS or directly to SQS.
The issue i'm having with this is that while I can get the useragent from the requester pre-send, i can't get the ip of the user without making a call to an endpoint and having the endpoint return the ip it observed.
Is it possible for the aws SNS/SQS api to append the ip of the request to the messages they receive?
SQS actions SendMessage, SendMessageBatch and SetQueueAttributes will process messages as they are received. There is no SQS/SNS configuration that could be used to modify the message. It would make sense to use SQS directly but in my opinion using SNS instead of API Gateway won't make it any better from cost/performance/implementation point of view. API Gateway appears to be your best option.
Related
quick question. I am planning to minimize use of resources by passing SNS topic as a source to EventBridge instead of passing it thru Lambda, then from EventBridge to Firehose. I tried but it is not working. Is it possible, or Lambda is the only way to make it as a source?
Its not possible. The only valid subscriptions to a SNS topic are:
For the http protocol, the (public) endpoint is a URL beginning with http://.
For the https protocol, the (public) endpoint is a URL beginning with https://.
For the email protocol, the endpoint is an email address.
For the email-json protocol, the endpoint is an email address.
For the sms protocol, the endpoint is a phone number of an SMS-enabled device.
For the sqs protocol, the endpoint is the ARN of an Amazon SQS queue.
For the application protocol, the endpoint is the EndpointArn of a mobile app and device.
For the lambda protocol, the endpoint is the ARN of an AWS Lambda function.
For the firehose protocol, the endpoint is the ARN of an Amazon Kinesis Data Firehose delivery stream.
But SNS can push directly to Firehose, so maybe just do this.
Has something changed since the last answer? Now it is possible to set aws.sns as source for an EventBridge rule (as shown in the below picture), but it does not seem to trigger the rule anyway... The question is, why do the AWS team let the users select aws.sns as source if it cannot work?
I have tried to do some R&D but i couldn't find anything useful the only thing that i found is lambda functions is the only way. I want to write a simple application that execute when the Simple Queue Service receive any message , but i couldn't find a way to do that till now, since i don't want to use lambda. for example if i receive some message on Simple Queue Service and while receiving each messages i can trigger a event that is not lambda but instead of that any HTTP-request.
I think your choices are:
use lambda (which you said you didn't want to use, but its probably the best solution)
use your own app running on ec2 or even on premise to consume the message and invoke the http endpoint
use SNS instead of SQS for message delivery - SNS supports http endpoints.
You can use Amazon SNS to send notification messages to one or more
HTTP or HTTPS endpoints. When you subscribe an endpoint to a topic,
you can publish a notification to the topic and Amazon SNS sends an
HTTP POST request delivering the contents of the notification to the
subscribed endpoint. When you subscribe the endpoint, you select
whether Amazon SNS uses HTTP or HTTPS to send the POST request to the
endpoint.
from here: https://docs.aws.amazon.com/sns/latest/dg/sns-http-https-endpoint-as-subscriber.html
A short answer to your question is No, (until today)
Let me tell you the sineros i faced.In general Queue triggering lambda is widely used and for that u have to make sure about proper concurrency (minimum 5) in place and also database I/O if u are performing and any DB calls . But I've a scenario where we cannot use "lambda as a triggering service" as our DB is onprem "ORACLE" so the choices are .
Push to "SNS" and make http "what ever applies"(to a container we have custom Kubernetes routed through NLB ).Also make sure you push a batch of messages as it might make more http noise.
2.Poll the queue and perform the operations.
SQS triggering => Lambda and lambda invoking state machine (step functions)
I have a requirement where I need to send some inbound http query parameters from api A by publishing it to SNS and other api B is subscribed to sns where it should get the query params.
Is there any way I can acheive it?
Note: I have a fixed message structure when I publish to SNS which I cannot change, since I have other endpoints who are already subscribing to SNS, depend on it.
As Michael mentioned, we can use lambda. but I do not want to use another service provided by aws. I want to acheive the same using SNS only. I heard about SNS message attributes, but not sure whether we can acheive the same using it.
Thanks.
SNS doesn't support customizing the HTTP transaction or otherwise modifying the message being delivered... but SNS does support subscriptions that target a Lambda function.
This means you can write a Lambda function that parses the SNS event payload... and then, instead of SNS contacting the target system over HTTPS, your Lambda function makes the HTTPS request to the target endpoint, customized as required. If the HTTPS request fails, or the endpoint returns an error, the Lambda function should throw an exception, so that Lambda can know to retry.
https://docs.aws.amazon.com/lambda/latest/dg/invoking-lambda-function.html#supported-event-source-sns
What's the easiest way to save/log every message published on a AWS SNS topic? I thought there might be a magic setting to automatically push them to S3 or a database, or maybe a database service supporting the HTTP destination automatically, but doesn't seem to be the case. Maybe it needs to be done via a Lambda function?
The purpose is just for basic diagnostics and debugging while setting up some SNS publishing. I don't really care about high scale or fast querying, just want to log and perform basic queries on all the activity for a few minutes at a time.
You can setup a trigger to push your SNS messages to SQS queue. Push is automatic and does not require any code.
According to the docs, SNS can publish to:
http – delivery of JSON-encoded message via HTTP POST
https – delivery of JSON-encoded message via HTTPS POST
email – delivery of message via SMTP
email-json – delivery of JSON-encoded message via SMTP
sms – delivery of message via SMS
sqs – delivery of JSON-encoded message to an Amazon SQS queue
application – delivery of JSON-encoded message to an EndpointArn for a mobile app and device.
lambda – delivery of JSON-encoded message to an AWS Lambda function.
So yes, a simple approach would be to trigger a lambda function to write to S3 or CloudWatch.
As far as I know, there is no way to directly send an SNS failed delivery receipt(of an SMS) to a custom HTTP endpoint; the only way I can think of is an intermediate CloudWatch log, which triggers a Lambda, which calls my API gateway URL.
What would be the most efficient way to capture an SNS failed delivery receipt at an HTTP endpoint?
Yeah that sounds right, although you should be able to directly trigger the Lambda with the SNS event (without CloudWatch log).
http://docs.aws.amazon.com/sns/latest/dg/application-event-notifications.html