I have an Active-Passive multi region architecture on AWS. We have cross-region replication for our RDS and DynamoDB; however, I'm not sure what to do with our messages in SQS when we failover. Based on the documentation there isn't a built-in feature for cross region replication.
Based on my research, the 2 solutions are;
Use fan-out with SNS so messages will be sent to both regions.
Have the secondary region applications get its messages from the primary region SQS queues.
The fan out pattern will not work for us because we need to come up with a way to determine if the message has already been processed by the primary region. We only have one active region at a time.
I was hoping for a more elegant solution than having the secondary region applications accessing the primary region messages. It's more expensive and introduce a little more complexity.
Is there a better way to accomplish having the secondary region continue to process messages where the primary region left off?
Related
I'm trying to understand DynamoDB replication & failover strategies but do not find any articles on the web which clarifies them. I understand cross-region replication can be achieved by DynamoDB with Global Tables but I also understand this is a multi-active table setup, meaning there are multiple active tables and multiple replica table. Is there a setup with single-active table and multiple replicas? I briefly read about this in this article but do not find any mentions anywhere else including AWS documentation.
I'm also trying to understand failover strategies for both cases - Is there a DynamoDB Java Client which can failover across AZs in case of issues in one AZ for both reads & writes?
DynamoDB Global Tables are always active-active but you can treat it as active-passive if you prefer. Many people do. That's useful if you want to use features like condition expressions, transactions, or do any non-idempotent wheres where you could have the same item being written around the same time in both regions with the second write happening before the first replicates, because this would cause the first write to be effectively lost.
To do this you just route your write traffic to one region, and to failover you decide when it's time to write to another. The failover region is always happy to be an active region if you'll let it.
As for AZs, DynamoDB is a regional service meaning it crosses at least 3 AZs always and would keep operating fine even if a full AZ were to be down. You don't have to worry about that.
Is there a setup with single-active table and multiple replicas
Unfortunately there is no such single active and multiple replica setup for cross region in dynamodb using global tables, so failover strategy will be for multiple active tables and multiple replica tables! - Source - docs
Fro failover strategies
According to docs
If a single AWS Region becomes isolated or degraded, your application can redirect to a different Region and perform reads and writes against a different replica table.
This means this is seamingless smooth process which happends by default ofcourse you can add custom logic when to redirect
As SQS is distribute queue, so does it replicate messages in the same region or different region? Looking at architecture at the AWS
docs, it shows the message being replicated, but does it replicate in the same region or different regions?
Use case:
I'm setting up queue in region X, but it might be accessed in a region at other end of world. So if there are two workers running one in region X and one in region Y, does both get data from same region X queue or can it be region X and region Y got data from region near to them.
Like X got a message from region X and before the time this info reaches region Y to update queue, then another worker take from replicated region Y queue and reads same message.
P.S :- I know SQS in at least once semantics. But I want to know semantics in the above use case.
SQS is a regional service, that is highly available within a single region. There is no cross-region replication capability. You can definitely access the queue from different regions, just initialize the sqs client with the correct destination region.
As a standard practice for AWS services, the data resides within the region that you create the service in.
There are exceptions, but these will require you as the user to perform an action to allow such as copying an AMI, or enabling S3 replication.
If the queue is being consumed in multiple regions, it will always access the regional endpoint of the SQS queue rather than that of the current region.
As SQS is a queueing service, if you have workers distributed across regions the likelihood is that the item is removed from the queue and processed in a single region (although the exact definition would be it is delivered at least once).
If you're trying to have the message consumed in multiple regions, it would be better to consider a fanout based approach whereby each regions workers would consume from their own SQS queue as opposed to sharing one.
For more information take a look at the https://aws.amazon.com/getting-started/hands-on/send-fanout-event-notifications/ documentation.
I am trying to achieve a point in a system architecture on top of AWS infrastructure where a message is published from a data processor and sent to multiple subscribers (clients). This message would contain information that some - but not all - clients would want to receive.
Very similar question > Routing messages from Amazon SNS to SQS with filtering
So to do this message filtering I have turned to the message FilterPolicy functionality provided by SNS using one topic. Currently the system is reaching a point in time that clients have more granular and specific filtering rules so now I am reaching the filtering limits of the AWS SNS.
See more about the SNS filter policy here https://docs.aws.amazon.com/sns/latest/dg/sns-subscription-filter-policies.html (section “Filter policy constraints”)
One example of my limitation is the amount of filter values in a policy, on above link it states 150 values. Right now my subscribers would be interested in receiving messages with a specific attribute value. Although this one attribute could have several hundreds or thousands of different values.
I can not also group this attributes since they represent a non-sequential identity.
I seek some guidance over on a architectural solution that would allow me to keep using AWS SNS. I am limited to use some of the AWS infrastructure services, so no RabbitMQ for me.
Generic information :-i am designing solution for one of IOT problem approach in which data is continuously streaming from plc(programmable logic controller),plc have different tags these tags are representation of telemetry data and data will be continuously streaming from these tags, each of devices will have alarm tags which will be 0 or 1 , 1 means there is an equipment failure
problem statement:- i have to read the alarm tag and raise a ticket if any of alarm tag value is 1 and i have to stream these alerts to dashboard and also i have to maintain the ticket history too,so the operator can update the ticket status too
My solution:- i am using aws IOT , i am getting data in dynamo db then i am using dynamo db stream to check if any new item is added in alarm table and if it will trigger lambda function (which i have implemented in java) lambda function opens a new ticket in relational database using hibernate.
problem with my approach:-the aws iot data is continuously streaming in alarm table at a very fast rate and this is opening a lot of connection before it can be closed that's taking my relational database down
please let me know if other good design approach can i adopt?
USE Amazon Kinesis Analytics to process streaming data. Dynamodb isn't suitable for this.
Read more here
Below image will give you an idea for same
Just a proposal....
From lambda, do not contact RDS,
Rather push all alarms in AWS SQS
then you can have one another lambda scheduled for every minute using AWS CloudWatch Rules that will pick all items from AWS SQS and then insert them in RDS at once.
I agree with raevilman's design of not letting Lambda contact RDS directly.
Since creating a new ticket is not the only task you Lambda function is doing, you are also streaming these alerts to a dashboard. Depending on the streaming rate and the RDS limitations, you may want to split these tasks in multiple queues.
Generic solution: I'd suggest you can push the alarm to a fanout exchange and this exchange will in turn push the alarm to one or more queues as required. You can then batch the alarms and perform multiple writes together without performing connect/disconnect cycle multiple times.
AWS specific Solution: I haven't used SQS so can't really comment on it's architecture. Alternatively, you can create an SNS Topic and publish these alarms to this topic. You can then have SQS queues as subscribers to this topic which in turn will be used for Ticketing and Dashboard purpose independent of each other.
Here again, from Ticketing queue, you can poll messages using Lambda or your own scheduler in batch and process tickets(frequency depending on how time critical alarms are).
You may want to read this tutorial to get some pointers.
You can control number of lambda function concurrency. And this will reduce the number of lambdas that get spinned up based on the dynamo events. Thereby reducing the connections to RDS.
https://aws.amazon.com/blogs/compute/managing-aws-lambda-function-concurrency/
Ofcourse , this will throttle the dynamo events.
I would like to create SQS queues in 2 different AWS regions. Is there a way to setup synchronization between both queues? When data is read off a queue in either region , message must not be available for consumption. If one the region goes down , then consumer must start reading from the next message in the available region? Does AWS support this out of the box or are there any patterns available to support this use case?
No, this is not a feature of Amazon SQS.
It would be quite difficult to implement because you cannot request a specific message off a queue. So, if a message is retrieved in one region, there is no way to delete that message in a different region. You would need to operate a database to keep track of the messages, which sort of defeats the whole purpose of the queue.
Amazon SQS is a multi-AZ service that can survive failure of an Availability Zone, but resides in a single region.
You can use Amazon SNS to fan out messages to multi SQS queues, even in multiple different regions. Details here: Sending Amazon SNS messages to an Amazon SQS queue or AWS Lambda function in a different Region.
However this results in duplicate messages across those regions, this does not satisfy your requirement
When data is read off a queue in either region , message must not be available for consumption