Is it possible to auto send/push the messages in Amazon SQS to DynamoDB? I wish to send my messages to SQS and for period of time I want to send this to DynamoDB. Another service should fetch the DynamoDB table and send it as email using SES.
Kindly help me out to achieve this. I will be using it for the User notification purpose from a Social networking site.
Thanks.
There is no AWS mechanism to automatically publish SQS messages to DynamoDB; but you can use an AWS Lambda event source mapping to automatically pull SQS messages and invoke a Lambda function, and it's pretty straightforward to write a Lambda function that writes those messages to DynamoDB. (Here's an example using Node.js: https://docs.aws.amazon.com/sdk-for-javascript/v2/developer-guide/dynamodb-example-table-read-write.html.)
Yes I agree Hyangelo, you can do this with Simple Workflow Service (SWF).
SWF will give you a control feature over your application enabling you to distribute and execute different services or tasks when you want.
Here is the link to the documentation: http://aws.amazon.com/swf/
Sounds like a workflow system from how you describe what you want, have you considered Simple Workflow Service?
SQS can't be processed w/o pulling messages.
You can either use SWF to solve your use-case OR use SNS.
SNS<=>SQS binding is free by AWS.
Send your messages to SNS, bind your SNS with SQS & lambda-function.
On triggering lambda function - you can create dynamodb-record and send it to another SNS2.
Bind SNS2 <=> SES which will trigger the email.
checkout: https://aws.amazon.com/premiumsupport/knowledge-center/lambda-sns-ses-dynamodb/
Related
I would normally handle the task of sending an email after a new DynamoDB entry with Lambda and SES but I'm required to not use Lambda for it.
There's a 'Contact us' section in the website and the email needs to be sent every time a new entry is made. We use API Gateway to post the data to DyanmoDB
Is there a way to carry this out without Lambda?
It's not possible without writing code. Furthermore you may probably want to tailor each email to make it more personal to the user, thus Lambda is a must.
You could design something using EventBridge Pipes which can trigger when a new user is added and can have SNS as a destination which could trigger an email. But that email may not be customizable nor may it send to people not subscribed to the topic
DynamoDB triggers Lambda functions by feeding the records of a DynamoDB Stream to the Lambda function. That is by far the easiest way to process updates to a DynamoDB table, but you can also write other code that processes a DynamoDB outside of Lambda: https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Streams.html
If you really want to do it without lambda you can use the following pattern and glue some AWS services together.
You can consume the dynamodb stream with eventbridge pipes and send it to a stepfunction and there you use the sdk integration to call SES. Instead of stepfunction you can directly use SNS for simpler setups.
You will not have to use lambda for this setup. Further you can transform your message either in the pipe or in the stepfunction.
Be aware that eventbirdge pipes has currently no AWS CDK L2 construct which might make it harder to configure if you use CDK.
I want to build a pub/sub messaging system into my services that are hosted on Amazon Web Services, and creating SQS queues that subscribe to SNS topics seems like the obvious direction to take, but I can't get it working at all.
So far my code looks for the topics and the queues at startup and creates anything that's missing. This seems to work, I can see the SNS topic and the SQS queues in the AWS management console, and I can see that the queue is subscribed to the topic, but when I publish messages to the topic nothing ends up in the queue.
Even if I manually publish a message using the 'Publish' button in the management console the queue is still empty.
I changed the permissions on both the topic and the queue to 'everyone can do everything' just to eliminate this possibility. My code receives a message ID in response to the publish and there are no errors, every API call returns a 200 (OK) status.
Where can I go from here to figure out why it's not working?
The SNS --> SQS link has a few gotchas:
"Amazon SNS isn't currently compatible with FIFO queues." per the note on their Subscribing an Amazon SQS Queue to an Amazon SNS Topic Tutorial
You have to fiddle with the IAM permissions (see the page on Sending Amazon SNS Messages to Amazon SQS Queues)
You can't send messages to encrypted queues (see their Server-Side Encryption page)
It would definitely have been easier to figure this out if all this info were consolidated into a single page. The killer for me was #3 - perhaps one of these will be the solution to your issue.
A couple of options -
Enable CloudTrail and monitor the logs
View the CloudWatch logs to identify any permissions issues
Open a ticket with AWS support.
Ideally, you wouldn't be creating the resources in your application but instead decouple those into CloudFormation or at a minimum CLI scripts. If you require the ability to dynamically create these resources, using the AWS IoT Message Broker may be a better option since it supports ephemeral messaging resources - http://docs.aws.amazon.com/iot/latest/developerguide/iot-message-broker.html
I had a similar issue with SQS subscriptions. It turned out that if I create the subscription from the SQS editor it works, but if I create it from the SNS creation screen it accepts the message but never forwards it to the queue.
To get more detailed information about specific failures you can turn on
SNS "Delivery status logging".
We are planning to use AWS SES with SQS. The idea is to push the email content to SQS then SES fetch the queue item and send the emails. We are using PHP.
We just want the idea to implement this without hampering the webserver performance.
SES cannot fetch stuff from SQS.
You will have to have a compute tier in between (either on EC2 or Lambda or some other server), which could poll the SQS and then call SES.
Scheduled Lambda can be ideal for this use case.
Out of curiosity, who is putting the message in SQS and do tou have any specific use case for putting the message in SQS. Why can't the routine that places the message on SQS, simply call the SES and get the email sent
Answer is summed up in previous answer but still I want to put a word for it.
Flow that I implemented was like this: Invoking sender lambda of sqs -> sqs queue triggers reciever lambda of ses -> ses sends mail.
The answer to why it is to be done? To avoid bottlenek at SES.
A Lambda function can process items from multiple queues (using one Lambda event source for each queue). You can use the same queue with multiple Lambda functions.
Other ways to maximize throughput of SES given in docs: https://docs.aws.amazon.com/ses/latest/dg/troubleshoot-throughput-problems.html
Is that possible if the user create a ticket in freshdesk that needs to be trigger the AWS lambda function.
That shouldn't be that hard. I would like to recommend using the following architecture
FreshDesk Ticket Trigger
FreshDesk Ticket Trigger Handler Published Message to SNS Topic
AWS Lambda Configured to SNS Topic as Event Source
AWS Lambda Code Accepts the SNS topic message (as Input) and performs the necessary processing
The advantages of using SNS rather directly calling Lambda are
Reducing the exposure of AWS API to only SNS topic and completely sealing rest of the API (IAM Privileges)
Possibility of Fan-Out Architecture [Multiple Lambda Functions can listen to the same SNS topic - near zero configuration]
For anyone landing on this topic.
It's possible with Freshdesk Marketplace app. With onTicketCreate product event, any actions can be written to execute with a Serverless function. It's completely run in Freshworks platform cloud.
If required, it can call your AWS Lambda.
I need to load a SQS queue on the Amazon Web Services to load test an application which consumes the messages from the SQS queue.
Is it possible to load the SQS queue using JMeter?
Are there any better options available to load the SQS queue than using JMeter?
I haven't tried it, but you should be able to send the messages through HTTP requests.
I created JMeter plugin that use AWS SDK to publish events in SNS Standard or FIFO Topic:
https://github.com/JoseLuisSR/awsmeter
You just need AWS IAM user with programmatic access, download JMeter and install awsmeter plugin.
If you have questions or comments let me know.
Thanks.