I have a web service that i am using to submit a text message to be sent as a bulk sms. I have been given a url that i should submit the text message to for sending.
I have the numbers in a csv file that i am reading in this way
$row = 1;
if (($handle = fopen("27k.csv", "r")) !== FALSE) {
while (($data = fgetcsv($handle, 1000, ",")) !== FALSE) {
$num = count($data);
$row++;
for ($c=0; $c < $num; $c++) {
echo $data[$c] . '<br/>';
}
}
fclose($handle);
}
The messages are being submitted one by one to web service and sending 30,000 records is taking hours.
Is there a programming construct in any language that i can use to be able to make 30,000 concurrent requests to the web service instead of sending one text at a time?.
Yes! There are many concurrency constructs that could help you:
multiple threads
event loop (single thread)
multi process job queue
Each of the above has its own tradeoffs. With threading memory synchronization has to be considered (often simplified by passing data between threads with queues).
Event loops are optimized for IO (should be perfect for your use case) but representing futures and wrapping head around asynchronous operations, yielding, and what state the event loop is in in any given time can be maddening.
Interprocess job queue has operational overhead of managing a queue and a pool of workers.
While any of the above solutions should allow for X number of concurrent requests, it's generally good practice to bound your resource pools (ie limit concurrency). If you launch 30,000 concurrent requests (can your machine even launch that many???) You could easily overwhelm a server. If the launch is driven by an end user or an uncontrolled action, you could easily have X * 30000 requests being launched at any time.
Also what happens if a single request fails? will you have to retry? What about when you get rate limited?
Related
Our current implementation is to open one database connection outside of the Lambda handler. When the backing Lambda container terminates, the connection is then left open/idle.
Can I make the new container close the previous old container's database connection?
Are there any hooks available like an onContainerClose()?
How can we close the previous open connection which cannot be used anymore, when the Lambda cold starts?
In the background, AWS Lambda functions execute in a container that isolates them from other functions & provides the resources, such as memory, specified in the function’s configuration.
Any variable outside the handler function will be 'frozen' in between Lambda invocations and possibly reused. Possibly because depending on the volume of executions, the container is almost always reused though this is not guaranteed.
You can personally test this by invoking a Lambda with the below source code multiple times & taking a look at the response:
let counter = 0
exports.handler = async (event) => {
counter++
const response = {
statusCode: 200,
body: JSON.stringify(counter),
};
return response;
};
This also includes database connections that you may want to create outside of the handler, to maximise the chance of reuse between invocations & to avoid creating a new connection every time.
Regardless of if the Lambda function is reused or not, a connection made outside of the handler will eventually be closed when the Lambda container is terminated by AWS. Granted, the issue of "zombie" connections are much less when the connection is reused but it is still there.
When you start to reach a high number of concurrent Lambda executions, the main question is how to end the unused connections leftover by terminated Lambda function containers. AWS Lambda is quite good at reliably terminating connections when the container expires but you may still run into issues getting close to your max_connections limit.
How can we close the previous open connection which cannot be used anymore, when the Lambda cold starts?
There is no native workaround via your application code or Lambda settings to completely getting rid of these zombie connections unless you handle opening and closing them yourself, and take the added duration hit of creating a new connection (still a very small number).
To clear zombie connections (if you must), a workaround would be to trigger a Lambda which would then list, inspect & kill idle leftover connections. You could either trigger this via an EventBridge rule operating on a schedule or trigger it when you are close to maxing out the database connections.
These are also great guidelines to follow:
Ensure your Lambda concurrent executions limit does not exceed your database maximum connections limit: this is to prevent the database from maxing out connections
Reduce database timeouts (if supported): limit the amount of time that connections can be idle & left open, for example in MySQL tweaking the wait_timeout variable from the default 28800s (8 hour) to 900 seconds (15 minutes) can be a great start
Reduce the number of database connections: try your best to reduce the connections you need to make to the database via good application design & caching
If all else fails, look into increasing the max connections limit on the databe
I have a large number of tasks - N, needs to be distributed to multiple http worker nodes via load balancer. Though there exists multiple nodes - n, combining all nodes we have a max-concurrency setting - x.
Always
N > x > n
One node can run those tasks in multiple threads. Mean time consumption for each task is about 50 sec to 1 min. Using Web-Client to distribute tasks and Mono response from Workers.
There exists a distributor and designed the process as follows:
1. Remove a task from queue.
2. Send the task via POST request using Web-Client and subscribe immediately with a subscriber instance
3. Holt new subscription when max concurrency reached to x
4. When any one of the above distributed task completes it calls on-accept(T) method of the subscriber.
5. If task queue is not empty, remove and send the next task / (x+1) task.
6. Keep track of total number completed tasks.
7. If all tasks completed & queue empty set Completable Future object as complete
8. Exit
The above process works fine. Tested with N=5000, n=10 & x=25.
Now the confusion is in this design we always have x number of concurrent subscriptions. As soon as one ends we create another until all tasks are completed. What is the impact of this in large scale production environment? If number of concurrent subscription (the value of x > 10,000) increases via the HTTP(s) load balancer is that going to have serious impact on performance and network latency? Our expected production volume will be something like below:
N=200,000,000
n=100
x=10,000
I will be grateful if some one with knowledge of Reactor and Web-Client expertise comment of this approach. Our main concern is having too many concurrent subscriptions.
We're designing C# scheduled task (runs every few hours) that will run on AWS ECS instances that will grab batched transaction data for thousands of customers from an endpoint, modify the data then send it on to another web service. We will be maintaining the state of the last successful batch in a separate database (using some like created date of the transactions). We need the system to be scalable so as more customers are added we add additional ECS containers to process the data.
There are the options we're considering:
Each container only processes a specific subset of the data. As more customers are added more contains are added. We would need to maintain a logical separation of what contains are processing what customers data.
All the containers process all of the customers. We use some kind of locking flags on the database to let other processes know that the customers data is being processed.
Some other approach.
I think that option 2 is probably the best, but it adds a lot of complexity regarding the locking and unlocking of customers. Are there specific design patterns I could be pointed towards if that if the correct solution?
In both scenarios an important thing to consider is retries in case processing for a specific customer fails. One potential way to distribute jobs across a vast number of container with retries would be to use AWS SQS.
A single container would run periodically every few hours and be the job generator. It would create one SQS queued item for each customer that needs to be processed. In response to items appearing in the queue a number of "worker" containers would be spun up by ECS to consume items from the queue. This can be made to autoscale relative to the number of items in the queue to quickly spin up many containers that can work in parallel.
Each container would use its own high performance concurrent poller similar to this (https://www.npmjs.com/package/squiss) to start grabbing items from the queue and processing them. If a worker failed or crashed due to a bug then SQS will automatically redeliver and dropped queued items that worker had been working on to a different worker after they time out.
This approach would give you a great deal of flexibility, and would let you horizontally scale out the number of workers, while letting any of the workers process any jobs from the queue that it grabs. It would also ensure that every queued item gets processed at least once, and that none get dropped forever in case something crashes or goes wrong.
I want to create a web application were a client calls a REST Webservice. This returns OK-Status for the client (with a link to the result) and creates a new message on an activeMQ Queue. On the listeners side of the activeMQ there should be worker who process the messages.
Iam stucking here with my concept, because i dont really know how to determine the number of workers i need. The workers only have to call web service interfaces, so no high computation power is needed for the worker itself. The most time the worker has to wait for returning results from the called webservice. But one worker can not handle all requests, so if a limit of requests in the queue is exceeded (i dont know the limit yet), another worker should treat the queue.
What is the best practise for doing this job? Should i create one worker per Request and destroying them if the work is done? How to dynamically create workers based on the queue size? Is it better to run these workers all the time or creating them when the queue requiere that?
I think a Topic/Suscriber architecture is not reasonable, because only one worker should care about one request. Lets imagine of 100 Requests per Minute average and 500 requests on high workload.
My intention is to get results fast, so no client have to wait for it answer just because not properly used ressources ...
Thank you
Why don't you figure out the max number of workers you'd realistically be able to support, and then make that number and leave them running forever? I'd use a prefetch of either 0 or 1, to avoid piling up a bunch of messages in one worker's prefetch buffer while the others sit idle. (Prefetch=0 will pull the next message when the current one is finished, whereas prefetch=1 will have a single message sitting "on deck" available to be processed without needing to get it from the network but it means that a consumer might be available to consume a message but can't because it's sitting in another consumer's prefetch buffer waiting for that consumer to be read for it). I'd use prefetch=0 as long as the time to download your messages from the broker isn't unreasonable, since it will spread the workload as evenly as possible.
Then whenever there are messages to be processed, either a worker available to process the next message (so no delay) or all the workers are processing messages (so of course you're going to have to wait because you're at capacity, but as soon as there's a worker available it will take the next message from the queue).
Also, you're right that you want queues (where a message will be consumed by only a single worker) not topics (where a message will be consumed by each worker).
I have heard that there is a limit for a server for the requests number it can process.
So if the requests from the client are large than the number people will queue the requests.
So I have two problems:
1 When
How to decide if it is necessary to queue the requests? How to measure the largest number?
2 How
If the queue is unavoidable, so where should be the queue done?
For a J2EE application using spring web mvc as the framework, I want to know if the queue should be put in the Controller or the Model or the DAO?
3 Is there a idea which can avoid the queue but keeping providing the service?
First you have to establish your limit at the server actually is. Its likely that its a limit on the frequency of messages, ie. maybe your limited to sending 10 requests a second. If thats the case then your would need to keep a count of how many messages you've sent out in a second, then before you send out a request check to see if you will breach this limit, if this is true then you must make the thread wait until the second is up. If not your free to send the request. This thread would be reading from a queue of outbound messages.
If the server limit is determined in an other way, i.e. dynamically based on its current load, which sounds like it might be in your case, there must be a continuous feed of request limits which you must process to determine the current limit. Once you have this limit you can process the requests in the same way as mentioned in the first paragraph.
As for where to put the queue and the associated logic, i'd put it in the controller.
I don't think there is a way to avoid the queue, you are forced to throttle your requests an therefore you must queue your outbound requests internally so that they are not lost, and will be processed at some point in the future.