I have two nodes in a Akka cluster.
I subscribe to all ClusterDomainEvent of the cluster with:
cluster.subscribe(
self,
InitialStateAsEvents,
classOf[ClusterDomainEvent])
When one of the two nodes is down, I receive a Unreachable event and I start to receive some logs every few seconds that warn me as following:
Association with remote system [akka.tcp://application#127.0.0.1:2554] has failed
When the down node come back, the logs stop, so it is detected that the node is reachable again but I still don't get a ReachableMember event.
What am I missing? Why should I do in order to receive this cluster event?
The way of doing it is to subscribe to cluster events with classOf[ReachabilityEvent]
So
cluster.subscribe(
self,
InitialStateAsEvents,
classOf[MemberEvent],
classOf[ReachabilityEvent])
Related
I have an SQS queue which contains messages that need not be consumed in order. This queue is mostly for decoupling purpose. I have 2 EC2 hosts that I would want to poll this queue. The processing of each message takes time. While one of my EC2 instance is processing a message, can my other EC2 poll the next message from the queue?
If this cannot be done, then is using an SQS an incorrect approach here? Should I instead configure an autoscaling group of EC2 instances and load balance the incoming requests among the EC2 instances?
Yes it is possible, when a instance grabs the message it is put in " Messages in flight" status. this is not available to other instances polling the queue.
Efectly reserving that message for that consumer.
more info here https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/features-capabilities.html
I am getting familiar with queue services in Amazon.
SQS is pull based not push based, so I have to have an EC2 instance pulling out the messages from the queue.
Are those instances EC2 AMI VM? or when I created an sqs queue ... do I have to associate to a special EC2 instance?
Why we can lose an EC2 instance when they are reading queues?
Any computer on the Internet can make a ReceiveMessage() API call. This could be an Amazon EC2 instance, or an AWS Lambda function, or a container or even the computer under your desk.
The typical architecture is that some 'worker' code is running somewhere, and it polls the Amazon SQS queue to ask for a message. If a message is available, the worker then processes the message and then deletes the message.
So, simply include the code to 'pull' the message within the program that will process the message.
I have a autoscalling in AWS, that basically do:
Run a python process script.py
This script get messages from sqs queue to process
My autoscalling is configured to start/terminate instance based on # of avaliables messages in queue. But sometimes when i processing something in machines and my # of messages, my autoscaling trigger to terminate instances, so i end losting message in the middle of processing.
I starting trying to handler signals but does not seem to be working.
My main goal is:
If i know that my instance will be terminate soon, i will wait finishing my current processes (i will not get any new message) and them i send a signal "OK" to AWS to shutdown the instance.
Is there anyway to archive this? I'm not using load balancing because i manually get the messages from queue.
you can use AWS autoscaling lifecycle hooks, they will put your ec2 instance in wait state before terminating it and deliver a message to SNS or cloudwatch that your instance is ready to terminate you can finish your already processed message in the mean time, i found an interesting blog post explaining the use case similar to yours .
AWS autoscaling lifecycle hooks
I have made two ec2 instances listening to one particular aws sqs queue. Now both the instances are connected via load balancer. The problem is , as both the instances are listening on the same queue, isn't the message going to be received by both the instances at the same time. If it is received on both the instances at the same time, how to prevent duplication of the transactions?
Your architecture does not describe a normal usage pattern for Amazon SQS.
Amazon Simple Queue Service (SQS) is a queueing service where you can create a queue and send messages to a queue. Amazon SQS retains the message for up to 14 days.
Applications can then request to receive a message from the queue. This makes the message invisible while the application is processing the message. Once the application finishes processing the message, to tells SQS to delete the message from the queue. If the application fails while processing a message, Amazon SQS will make the message visible again after a period of time so that it can be processed by another application server.
Things to note:
SQS does not send messages. Rather, applications request to receive a message from the queue. Think of the apps as "retrieving" a message rather than SQS transmitting a message.
The instances retrieving the messages should not be sitting behind a Load Balancer. There is nothing sent to a Load Balancer in this process. Instead, the instances themselves connect to SQS and request a message.
If you wish to send a message to multiple systems at the same time, you can use Amazon Simple Notification Service (SNS).
According to this page, Shared Topic Subscription in WSO2, message delivery to subscribers sharing a client id will be done in round robin order. This article only shows a single MB instance. I am wondering how delivery is managed when you have a cluster of MB instances where there are multiple subscribers sharing a client id across the cluster. Is MB capable of round-robin delivery across all nodes?
WSO2 Message broker is distributed broker. It has slot based delivery model [1]. Slot creation and slot dispatch happen in coordinator node in cluster. Each node has slot delivery worker running to deliver messages to local subscriptions of node.
So when there are multiple subscriptions across in cluster sharing same subscription id, then local subscriptions of particular node will get messages in round robin order.
Due to slot architecture, it guarantee that none of two subscriptions will get same message. Because slot contain particular range of message id set.
Example: Let's say there is two node cluster where node1 and node2. Assume node1 is the coordinator node. There is topic call topic1. Publisher1 send 1000 messages to topic1 in node1 and there are two subscribers in each node call subcriber1, subscriber2 from node1 and subscriber3, subscriber4 from node2. Coordinator node will create slots when messages are publishing and dispatch to subscribers running node on demand. This happen through thrift communication. So all local subscribers in node1 and node2 will get messages round-robin order.
Hope you might understand high level architecture.
[1] More details - https://docs.wso2.com/display/MB300/Architecture