I have a autoscalling in AWS, that basically do:
Run a python process script.py
This script get messages from sqs queue to process
My autoscalling is configured to start/terminate instance based on # of avaliables messages in queue. But sometimes when i processing something in machines and my # of messages, my autoscaling trigger to terminate instances, so i end losting message in the middle of processing.
I starting trying to handler signals but does not seem to be working.
My main goal is:
If i know that my instance will be terminate soon, i will wait finishing my current processes (i will not get any new message) and them i send a signal "OK" to AWS to shutdown the instance.
Is there anyway to archive this? I'm not using load balancing because i manually get the messages from queue.
you can use AWS autoscaling lifecycle hooks, they will put your ec2 instance in wait state before terminating it and deliver a message to SNS or cloudwatch that your instance is ready to terminate you can finish your already processed message in the mean time, i found an interesting blog post explaining the use case similar to yours .
AWS autoscaling lifecycle hooks
Related
I have an SQS queue which contains messages that need not be consumed in order. This queue is mostly for decoupling purpose. I have 2 EC2 hosts that I would want to poll this queue. The processing of each message takes time. While one of my EC2 instance is processing a message, can my other EC2 poll the next message from the queue?
If this cannot be done, then is using an SQS an incorrect approach here? Should I instead configure an autoscaling group of EC2 instances and load balance the incoming requests among the EC2 instances?
Yes it is possible, when a instance grabs the message it is put in " Messages in flight" status. this is not available to other instances polling the queue.
Efectly reserving that message for that consumer.
more info here https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/features-capabilities.html
In my architecture when I receive a new file on S3 bucket, a lambda function triggers an ECS task.
The problem occurs when I receive multiple files at the same time: the lambda will trigger multiple instance of the same ECS task that acts on the same shared resources.
I want to ensure only 1 instance is running for specific ECS Task, how can I do?
Is there a specific setting that can ensure it?
I tried to query ECS Cluster before run a new instance of the ECS task, but (using AWS Python SDK) I didn't receive any information when the task is in PROVISIONING status, the sdk only return data when the task is in PENDING or RUNNING.
Thank you
I don't think you can control that because your S3 event will trigger new tasks. It will be more difficult to check if the task is already running and you might miss execution if you receive a lot of files.
You should think different to achieve what you want. If you want only one task processing that forget about triggering the ECS task from the S3 event. It might work better if you implement queues. Your S3 event should add the information (via Lambda, maybe?) to an SQS queue.
From there you can have an ECS service doing a SQS long polling and processing one message at a time.
I am getting familiar with queue services in Amazon.
SQS is pull based not push based, so I have to have an EC2 instance pulling out the messages from the queue.
Are those instances EC2 AMI VM? or when I created an sqs queue ... do I have to associate to a special EC2 instance?
Why we can lose an EC2 instance when they are reading queues?
Any computer on the Internet can make a ReceiveMessage() API call. This could be an Amazon EC2 instance, or an AWS Lambda function, or a container or even the computer under your desk.
The typical architecture is that some 'worker' code is running somewhere, and it polls the Amazon SQS queue to ask for a message. If a message is available, the worker then processes the message and then deletes the message.
So, simply include the code to 'pull' the message within the program that will process the message.
So I have this scenario where an Amazon EC2 instance in an Auto Scaling group will be terminated. My problem is that I don’t want it terminated until it has finished whatever it’s doing.
If I hook up a lambda, the lambda would check a metric, if this metric is > 0 then it needs to wait 60 seconds.
I have this done already, the problem is it may take more than the Max timeout for lambdas of 15 minutes, to finish the jobs it’s processing.
If I read correctly, the lifecycle notification is only sent once, so this lambda won’t work for me.
Is there any other way of doing this?
Here is how I would try to approach this problem (this is a needs a POC, the answer is theoretical):
Create an Auto Scaling Group
Put a lifecycle hook on this ASG like described here, sending notification to Amazon SNS
Create a launch script for instances which will do the following
subscribe to SNS on instance launch and start SNS listener script
SNS listener will wait for instance termination message, do whatever necessary to wait until instance is ready to terminate, including sending heartbeats if termination needs more than 1h and completing lifecycle hook (described here). This should also handle unsubscription from SNS.
I was trying to setup a periodic task (using cron.yaml) in EB worker environment which is using a FIFO SQS queue. When cron job tries submit job to SQS, it fails because it does not have message group id which is required for FIFO queue.
Is there a way around this? (Apart from using some other scheduling mechanism or using general queue)
scheduler: dropping leader, due to failed to send message for job
'italian-job', because: The request must contain the parameter
MessageGroupId. (Aws::SQS::Errors::MissingParameter)
Update: As a work around, I created a cloud watch trigger to start a lambda which sends messages to SQS queue.