Can SQS scale up to 1,000,000 queues for a single account? - amazon-web-services

Question

I need a messaging service that allows me to create a channel for each user in order to facilitate real-time notifications. If I have somewhere between 100,000 and 1 million users, does it make sense to create an SQS queue for each of these users?
According to the SQS pricing documentation it would only cost $0.40 to create 1 million queues, but would I run into scaling problems?
Also, is there a way to set an expiration date on a queue? If a user deletes their account, then their queue no longer needs to exist.

Answer 1

Creating queues is not an issue here. Polling or even long polling the queue is going to be really expensive for you. In order to process real-time notifications, you need to poll every queue, 1M of them for lets say every 5 seconds.
Based on SQS Pricing, Price per 1 Million Requests after free tier is $0.00000040 per request.
That means you will be calling the ReceiveMessage API for about:
1000000 queues * 17280 (1 day in seconds / 5 seconds) = 17280000000 times.
Which is about $6912.00 per day for the worst case scenarios.
You need to architect the solution in a better way.

Answer 2

"a channel for each user in order to facilitate real-time notifications" - you don't need a dedicated queue per user for this - you can do this with one main messaging queue, and depending on your traffic patterns, probably a few overflow queues to deal with ultra-high-traffic users.
"One queue" you say? "How on earth will that scale to 1M users?"
The number of users doesn't matter. What matters is that your message consumption can keep up with message production (incoming messages per second). If you can do that, it'll seem like realtime to your users.
Message consumption can scale to as high as you're willing to spend - just spawn up a thread to handle each incoming message (use a thread pool!)
of course, you'll need to limit each host to X processing threads, based on how many it can handle (hence 'as high as you're willing')
the overflow queues are to keep costs under control - if you're scaled to handle 10K messages per second, you don't want a user to come along and send you 1M messages per second, knocking out your service and the rest of your customers - throttle them to some reasonable limit, and process the rest of those messages at a lower priority.
"But... millions." - Yes. SQS can handle a lot. And a single multi-tenant queue will scale much better in terms of architecture and cost than multiple single-tenant channels ever will.

Answer 3

Most AWS resource volumes are limited, and while I'm not finding any account limits on numbers of queues, I may have missed it or it may just not be published. I definitely wouldn't be excited about the queue per notification destination architecture you're pitching here, if my co-workers brought it to me. I would be concerned about the cost of putting the same notification to all the listeners' queues, and then reading them back out.
What you're describing sounds more like pub sub. Or, if you want better delivery guarantees, maybe a stream like kinesis or kafka. I've also heard of folks using Redis to implement this kind of thing.

Answer 4

Could you potentially design a queue consumer that pauses after a certain period of idle time to prevent unnecessary API calls?
Something like:
const AWS = require('aws-sdk');
// Set the AWS region
AWS.config.update({ region: 'YOUR_REGION' });
// Set the parameters for the consumer
const IDLE_TIMEOUT = 300; // Stop Polling after 300 seconds of idle time
const POLLING_INTERVAL = 10; // Poll the queue every 10 seconds
// Create an SQS client
const sqs = new AWS.SQS();
// Set a flag to control the Polling loop
let isPolling = false;
function startPolling() {
isPolling = true;
}
function stopPolling() {
isPolling = false;
}
async function processMessage(message) {
// Do something with the message here
console.log(message);
}
async function pollQueue() {
// Set the idle timer to 0
let idleTimer = 0;
while (isPolling) {
// Check if the idle timer has reached the timeout
if (idleTimer > IDLE_TIMEOUT) {
// Stop Polling
stopPolling();
break;
}
// Poll the queue for messages
const params = {
QueueUrl: 'YOUR_QUEUE_URL',
MaxNumberOfMessages: 10,
WaitTimeSeconds: POLLING_INTERVAL,
};
const data = await sqs.receiveMessage(params).promise();
// Get the messages from the response
const messages = data.Messages || [];
// Process the messages
for (const message of messages) {
await processMessage(message);
// Delete the message from the queue
const deleteParams = {
QueueUrl: 'YOUR_QUEUE_URL',
ReceiptHandle: message.ReceiptHandle,
};
await sqs.deleteMessage(deleteParams).promise();
// Reset the idle timer
idleTimer = 0;
}
// Increment the idle timer
idleTimer += POLLING_INTERVAL;
// Sleep for the Polling interval
await new Promise((resolve) => setTimeout(resolve, POLLING_INTERVAL * 1000));
}
}
// Start the Polling loop when the startPolling function is called
startPolling();
pollQueue();
This way you could only activate the consumer after some process begins, and avoid constant polling of the queue when the service is inactive.