I have multiple jobs that runs on ansible tower, Need to report on how many time a given job has been executed in a hour. how do I pull this through api ?
Related
I was wondering if it is possible to run large number of "jobs" (or "pipeline" or whatever is the right way) to execute some modelling tasks in parallel.
So what I planned to do is to do a ETL process and EDA done and after that when the data is ready, I would like to fire 2000 modelling jobs. We have 2000 products and each job can start with a data (SELECT * FROM DATA WHERE PROD_ID='xxxxxxxxx') and my idea is to run these training jobs in parallel (there is no dependency between them - so it makes sense to me).
First of all - 1) Can it be done in AWS SageMaker? 2) What would be the right approach? 3) Any special considerations I need to be aware of?
Thanks a lot in advance!
it's possible to run this on SageMaker, with SageMaker pipelines that will orchestrate a SageMaker Processing job, followed by a Training job. You can define the PROD_ID as a String parameter to the SageMaker Pipeline, then run multiple pipelines executions concurrently (default soft limit is 200 concurrent executions).
As you have a very high numbers of jobs (2K) which you want to run in parallel, and perhaps optimize compute usage, you might also want to look at AWS Batch, which allows you to queue up tasks, for a fleet of instances that starts containers to perform these jobs. AWS Batch also support Spot instances which could reduce your instance cost by 70%-90%. Another advantage of AWS Batch is that jobs reuse the same running instance (only container stop/start), while in SageMaker there's a ~2 minute overhead to start the instance per job. Additionally, AWS Batch also takes care of retries and allowing you to chain all 2,000 jobs together and run a "finisher" job when all jobs have completed.
Limits increase - For any service, you'll need to increase your service quota limits. It can be done from the console "Quotas" for most services, or by contacting AWS support. Some services has hard limits.
I'm hoping to configure some form of alerting for AWS Glue Jobs when they run longer than a configurable amount of time. These Glue jobs can be triggered at any time of day, and usually take less than 2 hours to complete. However if this exceeds the 2 hour threshold, I want to get a notification for this (via SNS).
Usually I can configure run time alerting in CloudWatch Metrics, but I am struggling to do this for a Glue Job. The only metric I can see that could be useful is
glue.driver.aggregate.elapsedTime, but it doesn't appear to help. Any advice would be appreciated.
You could use the library for that. You just need the job run id and then call getJobRun to get the execution time. Based on that you can then notify someone / some other service.
I am trying to understand the implementation of AWS lambda scheduling.
Suppose I created a function and schedule it for some future time.
Then does AWS is running some cron jobs every minute or second to check all the scheduled tasks and run them if match?
Or they are using some different method?
Basically, I want to implement a similar functionality where the user can schedule a task and my system then complete it at a scheduled time.
Thanks for answering.
We are using a task-scheduler to run a series (sequence) of tasks/jobs (SOAP calls) no our site, every night. But, we want to use AWS services such as Step-Function and Lambda to achieve above requirement, as Task-scheduler seems less reliable.
Challenge with Lambda is, 15 min. max timeout. As some of our jobs take more than 1 hour to process, I am having trouble figuring out which service could suffice the request.
I am also looking into AWS Fargate, as an alternative.
Any suggestions/edits are welcome, on which AWS services I could use to run jobs which take up to 1 hour or more.
You could look into AWS Batch which is meant for long running jobs. You can use store your powershell scripts in a Docker container and schedule them to run. They will run until completion.
I have to run a migration job, processing about 2 million records and the processing step for each should take about 10-30 secs. Is AWS Batch the right solution? Based on my understanding, it will launch containers for each job submitted.
Is it more efficient to run a poller on containers running in ECS and process all records?