I'm moving stuff from Azure to AWS, and the only thing I'm really gonna miss is the webjobs, where I can schedule command line jobs.
I know I can achieve somewhat the same with task scheduler or windows services, but I do also like the way webjobs shows logs and that stuff...
Do anybody know a tool like that, that can run windows command line apps on AWS?
Checkout AWS Lambda. It is a new service from AWS.
AWS Lambda, compute service that runs your code in response to events and automatically manages the compute resources for you, making it easy to build applications that respond quickly to new information.
Lambda vs WebJobs
Related
I have an ECS fargate task defined in aws. I would like to run it occasionally as needed.
Is there an easy way to do this?
I have terraform code that defines it as a scheduled task that is disabled. Whenever I want to run it, my procedure is:
Modify the terraform file to enable the task and set the scheduled execution time for five minutes from now.
Deploy the terraform and wait for the task to run.
Undo the terraform changes and redeploy.
This procedure works, but is quite inconvenient. Surely there is a better way to run one-off tasks? I've tried going through the aws web console but it's even worse.
If you want to stick with using the scheduler to run the task, then something like your current process is the only way to achieve that. However it sounds like you don't really want to have the task run on a set schedule at all, instead you only want to run it when needed.
The most direct way to trigger an ECS task to run, is via the RunTask API, which you can trigger from the AWS CLI (which you could wrap in a shell script), or one of the AWS SDKs.
You can try with Lambda. There is a project I wrote Python code with boto3 to run a lot of different tasks in AWS and I'm pretty sure Lambda can solve your problem.
I have an AWS CLI invocation (in this case, to launch a configured EMR cluster to do some steps and then shut down) but I'm not sure how to go about running it daily.
I guess one way to do it is an EC2 micro instance running a cron job, or an ECS task in a micro that launches the command, but that all seems like it might be overkill. It looks like there's also a way to do it in Lambda, but rom what I can tell it'd be kludgy.
This doesn't have to be a good long-term solution, something that's suitable until I can do it right (Data Pipelines) would work just fine.
Suggestions?
If it is not a strict requirement to use the AWS CLI, you can use one of the AWS SDK instead to programmatically invoke Lambda.
Schedule a CloudWatch Rules using cron
When configured, the CloudWatch Rules will trigger a Lambda function
Implement a Lambda function that calls EMR using one of the supported SDKs (e.g. the EMR class in the AWS JavaScript SDK)
Make sure that you have the IAM configuration in place
Full example is available in the Schedule AWS Lambda Functions Using CloudWatch Events
Kludgy? Yes, configuration is needed, however if you take into account the amount of work required to launch EC2 / ECS (and make sure that it re-launches in the event of failure), I'd say it evens out.
Not sure about the whole task that you are doing, but to avoid doing it:
Manually
Avoid another set up for resources in AWS (as you mentioned)
I would create a simple job in a Continuous Integration (CI) server like jenkins,bamboo,circleci ..... (list can go on). I would assume that you might already have a CI server running, why not use it?
I was wondering what would be the recommended way to host a long running scheduled task in AWS.
Currently we have an EC2 instance (windows) in charge of triggering our app every few hours. This task will take between 1-3 hours depending on the number of items to process.
Lambda does not seem to be appropriate since my task is too long
Found this topic about Hangfire Scheduled Jobs in .NET Core 2 Web app hosted in AWS. Seems good but outside of AWS.
Any suggestions?
Thx
Seb
I would recommend AWS Step Functions. Very easy to implement. Part of the AWS Serverless Platform.
AWS Step Functions
AWS Step Functions makes it easy to coordinate the components of
distributed applications and microservices using visual workflows.
Building applications from individual components that each perform a
discrete function lets you scale and change applications quickly. Step
Functions is a reliable way to coordinate components and step through
the functions of your application. Step Functions provides a graphical
console to arrange and visualize the components of your application as
a series of steps. This makes it simple to build and run multistep
applications. Step Functions automatically triggers and tracks each
step, and retries when there are errors, so your application executes
in order and as expected. Step Functions logs the state of each step,
so when things do go wrong, you can diagnose and debug problems
quickly. You can change and add steps without even writing code, so
you can easily evolve your application and innovate faster.
Using AWS Lambda with Scheduled Events would let you create an AWS Lambda that will respond to a scheduled event. This Lambda could then trigger your app. Your app doesn't need to be in a Lambda itself.
I've setup my State Machine in the AWS console (I'm using it as a asynchronous workflow platform to trigger a few lambdas). Anyway, I want to trigger state machine with some input from a java service running on an EC2 instance. But I can't find anyway to do this. In fact it looks like the only triggering mechanism is via cloudwatch? Is that right? That doesn't seem right?
Thanks any help would help
In the AWS SDK for Java you have both AWSStepFunctionsClient.startExecution() and AWSStepFunctionsAsyncClient.startExecutionAsync()
I am new to AWS and have a question about an application I'm trying to write. I have a bunch of data that sits within Amazon RDS. On a periodic basis, I would like a small snippet of code to run against this data and in certain situations have notifications sent where appropriate. Of all the AWS services, what is the best architecture for this?
Thanks
You could use a simple cron job running on an EC2 instance. The cron job could run a script (PHP, Perl, whatever) to go fetch the data and then do something with it (notify people, generate reports etc)
Does that help?
See here for details on getting started with a Linux instance: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EC2_GetStarted.html
You could achieve the same results using a Windows machine and Scheduled Tasks. Here's the getting started guide for Windows instances: http://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/EC2Win_GetStarted.html
You can use Scheduled Lambda service driven by CloudWatch events. It acts as a resilient cron.