quick question. I am planning to minimize use of resources by passing SNS topic as a source to EventBridge instead of passing it thru Lambda, then from EventBridge to Firehose. I tried but it is not working. Is it possible, or Lambda is the only way to make it as a source?
Its not possible. The only valid subscriptions to a SNS topic are:
For the http protocol, the (public) endpoint is a URL beginning with http://.
For the https protocol, the (public) endpoint is a URL beginning with https://.
For the email protocol, the endpoint is an email address.
For the email-json protocol, the endpoint is an email address.
For the sms protocol, the endpoint is a phone number of an SMS-enabled device.
For the sqs protocol, the endpoint is the ARN of an Amazon SQS queue.
For the application protocol, the endpoint is the EndpointArn of a mobile app and device.
For the lambda protocol, the endpoint is the ARN of an AWS Lambda function.
For the firehose protocol, the endpoint is the ARN of an Amazon Kinesis Data Firehose delivery stream.
But SNS can push directly to Firehose, so maybe just do this.
Has something changed since the last answer? Now it is possible to set aws.sns as source for an EventBridge rule (as shown in the below picture), but it does not seem to trigger the rule anyway... The question is, why do the AWS team let the users select aws.sns as source if it cannot work?
Related
I have an API Gateway endpoint that takes the bodies of requests to it and places them in an SQS queue. The API Gateway has the ability to transform the request and add requester meta, like the user agent and ip address, to the message it eventually sends to SQS.
I'd like to change this from API Gateway to SNS so that the requester would publish to an SNS topic that feeds into SQS or directly to SQS.
The issue i'm having with this is that while I can get the useragent from the requester pre-send, i can't get the ip of the user without making a call to an endpoint and having the endpoint return the ip it observed.
Is it possible for the aws SNS/SQS api to append the ip of the request to the messages they receive?
SQS actions SendMessage, SendMessageBatch and SetQueueAttributes will process messages as they are received. There is no SQS/SNS configuration that could be used to modify the message. It would make sense to use SQS directly but in my opinion using SNS instead of API Gateway won't make it any better from cost/performance/implementation point of view. API Gateway appears to be your best option.
I have a requirement where I need to send some inbound http query parameters from api A by publishing it to SNS and other api B is subscribed to sns where it should get the query params.
Is there any way I can acheive it?
Note: I have a fixed message structure when I publish to SNS which I cannot change, since I have other endpoints who are already subscribing to SNS, depend on it.
As Michael mentioned, we can use lambda. but I do not want to use another service provided by aws. I want to acheive the same using SNS only. I heard about SNS message attributes, but not sure whether we can acheive the same using it.
Thanks.
SNS doesn't support customizing the HTTP transaction or otherwise modifying the message being delivered... but SNS does support subscriptions that target a Lambda function.
This means you can write a Lambda function that parses the SNS event payload... and then, instead of SNS contacting the target system over HTTPS, your Lambda function makes the HTTPS request to the target endpoint, customized as required. If the HTTPS request fails, or the endpoint returns an error, the Lambda function should throw an exception, so that Lambda can know to retry.
https://docs.aws.amazon.com/lambda/latest/dg/invoking-lambda-function.html#supported-event-source-sns
What's the easiest way to save/log every message published on a AWS SNS topic? I thought there might be a magic setting to automatically push them to S3 or a database, or maybe a database service supporting the HTTP destination automatically, but doesn't seem to be the case. Maybe it needs to be done via a Lambda function?
The purpose is just for basic diagnostics and debugging while setting up some SNS publishing. I don't really care about high scale or fast querying, just want to log and perform basic queries on all the activity for a few minutes at a time.
You can setup a trigger to push your SNS messages to SQS queue. Push is automatic and does not require any code.
According to the docs, SNS can publish to:
http – delivery of JSON-encoded message via HTTP POST
https – delivery of JSON-encoded message via HTTPS POST
email – delivery of message via SMTP
email-json – delivery of JSON-encoded message via SMTP
sms – delivery of message via SMS
sqs – delivery of JSON-encoded message to an Amazon SQS queue
application – delivery of JSON-encoded message to an EndpointArn for a mobile app and device.
lambda – delivery of JSON-encoded message to an AWS Lambda function.
So yes, a simple approach would be to trigger a lambda function to write to S3 or CloudWatch.
We are working on SNS to HTTPS API integration between 2 teams. So, another team has SNS topic configured in us-east-1 region and our HTTPS API is running in us-west-2 region.
Just wanted to check if we can process SNS messages generated from us-east-1 and a different AWS account in us-west-2 region.
I read somewhere (can't remember the blog post link now), that if SNS messages are generated in us-east-1, they have to be processed in us-east-1 only. If we try to process in us-east-2, message signature verification will fail and throws an invalid TopicArn exception.
Please guide if this is correct.
So, another team has SNS topic configured in us-east-1 region and our HTTPS API is running in us-west-2 region.
When SNS is publishing to an HTTPS endpoint, the endpoint can be anywhere on the Internet. It doesn't even need to be in AWS at all. As long as your HTTPS endpoint is accessible from the Internet and has a valid SSL certificate (matches the hostname, not expired, signed by an accredited certificate authority, not self-signed), that is all SNS will require.
The destination region and destination AWS account lose all meaningfulness in such a setup. There is no constraint, here.
If the topic is in us-east-1 then all requests sent to SNS must be sent to the us-east-1 endpoint (e.g. Subscribe or Publish) but even then, they can be sent from anywhere on the Internet.
SNS can be configured across regions via HTTPS endpoint of lambda by attaching the API Gateway to that lambda and copying the HTTPS URL of same.
After getting the HTTPS URL of the lambda may be whichever region. It may be that you just add in SNS service subscription part in SNS service by creating the subscription in the service.
After that, just publish a message. You will see entries in your Cloudwatch logs if you print an event in lambda. You will see something like "Subscribe URL". Copy that URL and paste it in the SNS service where you will be able to see "Pending on Subscription". This will work with latest amazon SNS service.
I tried and tested it to verify that this works.
I'm relatively new to AWS, but I am trying to figure out how to get AWS to receive emails. According this post How to configure email accounts like support#xyz.com or feedback#xyz.com on AWS SES only handles outbound email.
What I am hoping to achieve is the ability to filter aliases. For example, if the alias is "xyz12alias", then any email sent to "xyz12alias#mydomain.co", can see the email and process the content appropriately. Which in my case will be storing it in account associated with the filter.
Can anybody direct me to a strategy or service within AWS that would allow me to implement inbound email on Amazon AWS?
https://postmarkapp.com/inbound appears to give me what I want, but is there anything within the AWS framework itself? Are there alternate services to postmarkapp?
Thanks.
Amazon Simple Email Service just introduced incoming e-mail support:
https://aws.amazon.com/about-aws/whats-new/2015/09/amazon-ses-now-supports-inbound-email/
In addition to offering a scalable, cost-effective email-sending
platform, Amazon SES can now accept your incoming emails. You can
configure Amazon SES to deliver your messages to an Amazon S3 bucket,
call your custom code via an AWS Lambda function, or publish
notifications to Amazon SNS. You can also configure Amazon SES to drop
or bounce messages you do not want to receive. If you choose to store
your messages in Amazon S3, Amazon SES can encrypt your mail using AWS
Key Management Service (KMS) before writing it to the bucket.
You configure all of these actions by defining receipt rules, which
you set up by using the Amazon SES console or the Amazon SES API.
Receipt rules enable a single message to trigger multiple actions.
Your rules can be as broad or as specific as you choose because you
can configure them to apply to specific email addresses or entire
domains.
You can also use receipt rules to control which messages Amazon SES
can accept on your behalf. Another filtering method is to set up
custom IP address block lists and allow lists. If you know that you
don’t want to receive mail originating from a particular IP address
range, simply add it to your account's IP address block list. You can
also override block lists by adding IP address ranges to your allow
list, which provides fine-grained control over your inbound email
traffic.
You'd have to set up your own server; that's the way to handle it using AWS. They don't provide anything other than their bulk email delivery service. A few links below:
http://jeffreifman.com/how-to-install-your-own-private-e-mail-server-in-the-amazon-cloud-aws/
http://cerebellumstrategies.com/2012/04/15/amazon-linux-postfix-dovecot/
Update: there is now a solution available in AWS, as referenced in the comments below.
Still doesn't appear to be possible on SES. I'd recommend looking at Mandrill and Sendgrid though.
http://mandrill.com/features/
https://sendgrid.com/docs/API_Reference/Webhooks/parse.html
Here is how to use Amazon and any virtual server to deliver email from SES to a local IMAP account.
This plan is about stable operations: every step is under our control.
Have SES receive emails
And deliver to an S3 bucket and send notification to a SNS topic
Subscribe to that notification with HTTPS protocol
Use aws/aws-php-sns-message-validator and write a small PHP script to reveive notifications from SNS through HTTPS
SNS only sends messaged ID-s, put those in a file
Install incron on your server to start a shell script that downloads those messages from the S3 bucket by s3cmd and delivers to the local IMAP account by sendmail
Use any IMAP server, I use Courier IMAP
Read your messages with e.g. Rainloop webmail
Use SES also for sending outgoing emails
When using AWS SES inbound mail support, main challenge is that, it fits well for programmatic tasks, but hardly human readable since it receives in raw message delivered format.
You you can setup a S3 bucket, SNS Topic (Which could again links to Email, SMS & etc.), Lambda to forward the inbound mail with the same format it receives to take any programmatic actions based on the content of the email.
If you want read the message in human readable format, you need to write your own code to do the formatting. For example using Serverless lambda SES forwarder includes, NodeJS code runs in Lambda to convert raw email to human readable format and forward to recipient email.