I went through AWS RDS WAIT documentation but couldn't find any way to wait for RDS Proxy to get created.
Is there any hack or workaround to wait explicitly before going to the next command in sequence?
Waiter is just a basic loop, which periodically (e.g. every 10 s) checks a status of a resource. For RDS proxy you would use describe-db-proxies to check Status. When Status is available the loop would finish.
Related
In my architecture when I receive a new file on S3 bucket, a lambda function triggers an ECS task.
The problem occurs when I receive multiple files at the same time: the lambda will trigger multiple instance of the same ECS task that acts on the same shared resources.
I want to ensure only 1 instance is running for specific ECS Task, how can I do?
Is there a specific setting that can ensure it?
I tried to query ECS Cluster before run a new instance of the ECS task, but (using AWS Python SDK) I didn't receive any information when the task is in PROVISIONING status, the sdk only return data when the task is in PENDING or RUNNING.
Thank you
I don't think you can control that because your S3 event will trigger new tasks. It will be more difficult to check if the task is already running and you might miss execution if you receive a lot of files.
You should think different to achieve what you want. If you want only one task processing that forget about triggering the ECS task from the S3 event. It might work better if you implement queues. Your S3 event should add the information (via Lambda, maybe?) to an SQS queue.
From there you can have an ECS service doing a SQS long polling and processing one message at a time.
I have a use case where I would submitting dynamic number of jobs to the cluster, hence opting to submit jobs via SDK from a lambda and not add submit jobs as a task in step function. The EMR cluster would be used once a week and hence want to opt for onDemand variant.
Looks like "auto-terminate" parameter is not supported when creating cluster from Step Functions. As per the doc, The field Instances.KeepJobFlowAliveWhenNoSteps is mandatory, and must have the Boolean value TRUE.
Is there an alternative way to terminate cluster after all jobs are completed?
You have few options to terminate the cluster, but it depends on your scenerio.
Since you are using Lambda, you can check for the state of cluster periodically and if its is WAITING, you can terminate the cluster with the ID. You can also make a CloudWatch event with AWS Lambda function to check if EMR cluster is Idle. you can find a good answer for this specific approach here and the code implementation by the same user here
A very naive and stupid thing but can work is to deliberately submit a failing step as the final step and use 'TERMINATE_CLUSTER' on option key ActionOnFailure while submitting with add_job_flow_steps()
Update on your question:
would there be potential race condition where in EMR cluster could
terminate after its started and before jobs got submitted?
The waiting time between the cluster staring and jobs submission/first job running isnt same, you can have a logic around deciding maximum idle time threshold for cloudwatch
I have two elastic beanstalk environments.
One is the 'primary' web server environment and the other is a worker environment that handles cron jobs.
I have 12 cron jobs, setup via a cron.yaml file that all point at API endpoints on the primary web server.
Previously my cron jobs were all running on the web server environment but of course this created duplicate cron jobs when this scaled up.
My new implementation works nicely but where my cron jobs fail to run as expected the cron job repeats, generally within a minute or so.
I would rather avoid this behaviour and just attempt to run the cron job again at the next scheduled interval.
Is there a way to configure the worker environment/SQS so that failed jobs do not repeat?
Simply configure a CloudWatch event to take over your cron, and have it create an SQS message ( either directly or via a Lambda function ).
Your workers will now just have to handle SQS jobs and if needed, you will be able to scale the workers as well.
http://docs.aws.amazon.com/AmazonCloudWatch/latest/events/ScheduledEvents.html
Yes, you can set the Max retries parameter in the Elastic Beanstalk environment and the Maximum Receives parameter in the SQS queue to 1. This will ensure that the message is executed once, and if it fails, it will get sent to the dead letter queue.
With this approach, your instance may turn yellow if there are any failed jobs, because the messages would end up in the dead letter queue, which you can simple observe and ignore, but it may be annoying if you are OCD about needing all environments to be green. You can set the Message Retention Period parameter for the dead letter queue to something short so that it will go away sooner though.
An alternative approach, if you're interested, is to return a status 200 OK in your code regardless of how the job ran. This will ensure that the SQS daemon deletes the message in the queue, so that it won't get picked up again.
Of course, the downside is that you would have to modify your code, but I can see how this would make sense if you don't care about the result.
Here's a link to AWS documentation that explains all of the parameters.
When running a step (for example loading data) on my AWS EMR cluster via the terminal, is it possible to automatically return a message in my terminal when the step has finished? Instead of having to check it myself every several minutes?
AFAIK, you can only wait for the EMR cluster to terminate using aws-cli. If you need status of each tasks, I think you need to write something custom. May be the Cloudwatch can be a choice as well. As there are EMR metrics by default sent to cloudwatch. More details here. Hope this helps.
I'm creating bunch of EBS snapshots as part of AWS Lambda. I need to capture events when these snapshots complete so I can create an ec2 instance based on these.
I could use snapshot waiter but this polls and sometimes snapshot creation can take long time. I don't want Lambda to keep running for a while and plus the maximum time for Lambda seems to be five minutes. I looked at CloudWatch and AWS Config to see if I can capture snapshot events but had no luck.
There is now a new Event when Snapshots are completed in AWS Cloudwatch Events:
http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-cloud-watch-events.html
You are correct -- there is no notification event that signifies completion of an EBS Snapshot. Instead, you would need to check the status until the status changes to completed.
You are also correct that AWS Lambda functions can run for a maximum of 5 minutes and having a Lambda function waiting on an external process is not a good architecture.
Instead, you could break-up the architecture:
Have your existing process trigger the EBS Snapshot(s) and then push a message into an SQS queue
Schedule a Lambda function (eg every 5 minutes) to check the SQS queue. If a message exists:
Retrieve details about the instance and snapshot(s) from the message
Check the status of the snapshot(s)
If the status is completed, perform the next step in the process
The down-side is that the scheduled Lambda function will trigger even when there are no messages in the queue. The Lambda function will exit very quickly (cost: 100ms).
The alternative is to run a cron script on an Amazon EC2 instance (or on any computer connected to the Internet). A t2.nano instance is about 15.6c per day, which might be more expensive than a schedule Lambda function. If you already have an instance being used, then there would be no additional cost.