Background:
I have created notification on my GCP bucket so whenever I am receiving any file in the bucket, it will create notification and sends it to pub/sub topic. I have created spring boot app that received message and processes that file.
Problem:
if i am running single instance of spring boot app that pull message from gcp subscription, then its working fine but when i am running multiple instances then each app is receiving notification and starts processing file, but i want only one instance to process the file.
Please help to suggest.
Related
I have two services, one is the producer (Service A) and one is a consumer (Service B). So Service A will produce a message which will be published to Amazon SQS service and then it will be delivered to Service B as it has subscribed to the queue. So, this works fine until I have a single instance of Service B.
But when I start another instance of Service B, such that now there are 2 instances of Service B, both of which are subscribing to the same queue, as it is the same service, I observe that the messages from SQS are now being delivered in round-robin fashion. Such that at a given time, only one instance of Service B receives the message that is published by Service A. I want that when a message is published to this queue, it should be received by all the instances of Service B.
How can we do this? I have developed these services as Springboot applications, along with Spring cloud dependencies.
Please see the diagram below for reference.
If you are interested in building functionality like this, use SNS, not SQS. We have a Spring BOOT example that shows how to build a web app that lets users sign up for email subscriptions and then when a message is published, all subscribed emails get the message.
The purpose of this example is to get you up and running building a Spring BOOT app using the Amazon Simple Notification Service. That is, you can build this app with Spring BOOT and the official AWS Java V2 API:
Creating a Publish/Subscription Spring Boot Application
While your message may appear to be read in a round robbin fashion, they are not actually consumed in a round robin. SQS works by making all messages available to any consumer (that has the appropriate IAM permissions) and hides the message as soon as one consumer fetches the message for a pre-configured amount of time that you can configure, effectively "locking" that message. The fact that all of your consumer seem to be operating in a round robin way is most likely coincidental.
As others have mentioned you could use SNS instead of SQS to fanout messages to multiple consumers at once, but that's not as simple a setup as it may sound. If your service B is load balanced, the HTTP endpoint subscriber will point to the Load Balancer's DNS name, and thus only one instance will get the message. Assuming your instances have a public IP, you could modify your app so that it self-registers as an HTTP subscriber to the topic when the application wakes up. The downsides here are that you're not only bypassing your Load Balancer, you're also losing the durability guarantees that come with SQS since an SNS topic will try to send the message X times, but will simply drop the message after that.
An alternative solution would be to change the message hiding timeout setting on the SQS queue to 0, that way the message is never locked and every consumer will be able to read it. That will also mean you'll need to modify your application to a) not process messages twice, as the same message will likely be read more than once by the time it has finished processing and b) handle failure gracefully when one of the instance deletes the message from the queue and other instances try to delete that message from the queue after that.
Alternatively, you might want to use some sort of service mesh, or service discovery mechanism so that instances can communicate between each other in a peer-to-peer fashion so that one instance can pull the message from the SQS queue and propagate it to the other instances of the service.
You could also use a distributed store like Redis or DynamoDB to persist the messages and their current status so that every instance can read them, but only one instance will ever insert a new row.
Ultimately there's a few solutions out there for this, but without understanding the use-case properly it's hard to make a hard recommendation.
Implement message fanout using Amazon Simple Notification Service (SNS) and Amazon Simple Queue Service (SQS). There is a hands-on Getting Started example of this.
Here's how it works: in the fanout model, service A publishes a message to an SNS topic. Each instance of service B has an associated SQS queue which is subscribed to that SNS topic. The published message is delivered to each subscribed queue and hence to each instance of service B.
I used Aws IoT Device management to register my laptop as a device and run a node js script on my laptop.
My laptop will publish the message to a topic and it also can subscribe to a topic.
But what I want is, aws cloud send msg to my deivce( which is my laptop), periodically, if my laptop doesn't give the responses back to the cloud, cloud will know that my device is dead.
But how to send msg to my device and wait for the response? I read the tutorial about the jobs and shadow in aws IoT device management, none of them are sending something to device, they are just existing on the cloud and device will go and grab them.
How to make aws IoT device management automatically send msg or publish the msg to the device? All I know is go to the "Test" page and publish to the topic the manually.
Okay, so you're trying to use AWS IoT to create a monitoring system to detect if your device (laptop) is alive or not.
Sending data from the cloud -> laptop is not needed. You can do this by only sending data from your laptop -> cloud.
Configure the Node.JS script on your laptop to periodically send messages to a topic in AWS IoT. Configure an AWS IoT Rule on that topic that will publish a metric to AWS CloudWatch. This metric will represent your laptop sending a heartbeat to the cloud.
You can additionally configure an alarm in AWS CloudWatch to watch this metric and perform an action if the alarm threshold is ever breached. For your use case you can probably set the alarm to breach when it's missing data points because that'll mean your laptop has stopped sending messages.
We plan to run a java application on Elastic BeanStalk. It is noting but a file retriever, processor, transformer and a mapper. It would retrieve the file from S3 and map it to an RDS DB. Question is how do I trigger this application running on beanstalk on file arrival on S3 bucket and also on demand ?
Thanks and regards,
Kunal
You can send an event to SNS (notification topic) on S3 file upload.
Then I see two options:
To the SNS you can hook up a lambda or http invocation, however in that case you will need to handle failures or availability issues.
If your application is running on EC2 I'd suggest to send the upload event to SQS (queue service) and your application can poll for queue messages
also on demand
for that you need to expose a an interfate or service from your application. You did not specify what your application is, so it's really up to you to define what 'on demand' is
I have two buckets created (bucket1 bucket2) and two SNS notification topics (alert_bkp_1 and alert_bkp_2) created and configured in the S3 event property.
Setup is for receiving email alerts whenever a Put occurs inside the bucket.
I'm encountering two problems with this setting:
Bucket 1
Bucket1 receives .tar files of 4 servers within a scheduled backup window in the servers crontab, but I receive email notifications of only 3.
Details:
When I send manually I get notification of all 4 servers normally.
The 4 backups are being stored diary in the bucket normally.
Bucket 2
Bucket 2 is receiving the .dump type backup files normally, but does not receive the email notifications.
Detail:
When I send manually I get notification of the 2 servers normally.
The 2 backups are being stored diary in the bucket normally.
I have already made several changes in the configuration of events of S3, but without success, I do not know what to do, someone would have any tips to help me with this problem?
For my web app I will need a separate instance of EC2 to process CPU-intensive things, and things that can be queued so they don't encumber the web serving instance, like image resizing, sending email....
When you create an AWS Elastic Beanstalk instance it asks you to choose between a "web" or "worker" environment. From my understanding it's in the worker environment that I will process those kind of tasks.
What's the role of SQS in this context? I read it is only about sending "messages" and this kind of stuff, but how will I get my image resized with a "message"?
Should I create specific, distinct code for the worker instance, to handle image resizing, and then use SQS to order it to process the image? Then can SQS pass image from web instance, to worker instance? I completely miss the main concept.
A queuing service (such as Amazon SQS) is used to store messages for later retrieval. Think of it like a TODO list -- you add items to the queue, and later you retrieve an item from the queue and take action on the item.
For example, let's say users upload images to a website and you wish to generate thumbnails from those images. Your website will store the image in Amazon S3, then push a message into an SQS queue. The message would include a reference to the image in S3 and details about the user.
Then, your Elastic Beanstalk worker will request a message from the queue and process the image. It would retrieve the image from S3, resize it, store it in an S3 bucket, then perhaps email the user to say that the job is finished. The worker then exits, and Elastic Beanstalk will trigger a new worker to read the next message from the queue and do it all again.
So, yes -- you will create the worker code. Elastic Beanstalk will trigger the worker with the SQS message. SQS itself doesn't trigger anything -- it is actually Elastic Beanstalk that retrieves the message and runs the worker.
See: Elastic Beanstalk worker environments