I have an AWS Python lambda function that connects to a DB, checks data integrity and send alerts to a slack channel(that's already done).
I want to execute that lambda every XX minutes.
What's the best way to do it?
You can build this with AWS EventBridge.
The documentation contains an example for this exact use case:
Tutorial: Schedule AWS Lambda Functions Using EventBridge
Related
I am new to AWS ecosystem. I'm building a (near) real-time system, where data comes from external API. The API is updated every 10 seconds, so I would like to consume and populate my Kinesis pipeline as soon as new data appears.
However, I'm not sure which tool use for that. I did a small research and, I think, I have two options:
AWS lambda which is triggered every 10 seconds and puts data on Kinesis
AWS StepFunction
What is the standard approach for a given use case?
AWS Step functions is created by Lambda functions. That is, each step in a workflow is actually a Lambda function. You can think of a workflow created by AWS Step Functions as a chain of Lambda functions.
If you are not familiar with how to create a workflow see this AWS tutorial:
Create AWS serverless workflows by using the AWS SDK for Java
(you can create a Lambda function in any supported programming language. This one happens to use Java).
Now, to answer your question, using a workflow to populate a Kinesis data stream is possible. You can build a Lambda function that gathers data (using logic in your Lambda function), and then invoke the putRecord operation of Kinesis to populate the data stream. You can create a scheduled event that fires off every x min based on a CRON expression.
If you do use a CRON expression, you can use the AWS Step Functions API to fire off the workflow. That is, create another Lambda function that is scheduled to fire say every 10 mins. Then in this Lambda funciton, use the Step Functions API to invoke the workflow. Now the workflow can populate the Kinesis data stream with data.
I'd like to peform the following tasks on a regular basis (e.g. every day at 6AM) using AWS:
get new set of data using API. This dataset is updated on a daily basis.
run a python script that would process the obtained dataset by the means of several python libraries like matplotlib, pandas, plotly
automatically send the output of the script, which would be a single pdf file or a html dashboard, via email to a group of specified recipients
I know how to perform all of the above items locally - my goal is to automate this routine. I'm new to AWS and would appreciate some advice on how to perform these tasks in a straightforward way. Based on the reading I did so far, it looks like the serverless approach may be able to do the job and also reduce the complexity, but I'm not sure which functionalities exactly I should use.
For scheduling you can use aws event bridge.
You can schedule AWS lambda or AWS Step Functions both of these are serverless :).
You can have 3 lambdas
To get the data and save it in S3/dynamo (if you want to persist the data)
Processor lambda and save the report to S3.
Another lambda to send email using AWS SES which will read the report from S3 and send it.
If you don't want to use step function you can start your lambda from S3 put event or you can trigger one lambda from another lambda using aws-sdk.
So there are different approaches you can take.
First off, I would create a Lambda. You can schedule the function to run on a cron job.
If the Message you want to send is small:
I would create a SNS Topic with a email fan out.
Inside your lambda you can then transform the data and send out via SNS.
Otherwise:
I would use SES and send a mail via the SES SDK.
i can write a lambda function which will send email to a specific user using SES . what i want to do is to make some kind of schedule task which will trigger this lambda function . the lambda function should accept user-email as function parameter as different user have different email and this lambda function should be trigger in different time for different user . how can achieve this ? i already did some digging about SNS, SQS, CloudeWatch, step function and i got confused . can anyone help me here ?
Here is an AWS tutorial that shows you how to write a Lambda function that uses the Lambda runtime API. This document shows you how to invoke other AWS services from the Lambda function (such as Amazon DynamoDB). In this example, user data is located in a database, not passed to the Lambda function (this is a minor diff from what you described).
The Lambda function is invoked on a regular schedule using scheduled events. This tutorial walks you through all steps.
Creating scheduled events to invoke Lambda functions
Following are some of the options;
Option - 1)
a) An observer = A lambda to check connect and check in mongodb for list of user's, for whom subscription/trial period is about to end. Have list of emails and part of message body such as date and other details which will be used in drafting email (in json format with key as email) of such users
b) A command executer = A lambda which will take list of emails and mapped contents. Iterate over this list and send email to the user's from the list
c) Schedule a Serverless Workflow using AWS step functions use/connect lambda a) and b) in this
Additional information: Refer Error handling in Step Functions for negative test cases
OR
Option - 2) I am not sure whether this is available for the setup of mongodb you have;
a) Using MongoDB Scheduled Triggers
b) MongoDB Scheduled Triggers integrated via. AWS Eventbridge mapped to AWS Lambda with logic to send email.
OR
Option 3) Combine observer (1.a) and command executer (1.b) logic within single lambda. Schedule this lambda as per your requirements using AWS cloudwatch event rule
Create CloudWatch trigger for each user if you have a small amount of users. Otherwise trigger the function frequently, say, every minutes by CW and save the last time for each user in a dynamo table.
Quick question: Is it possible to trigger the execution of a Step Function after an SQS message was sent?, if so, how would you specify it into the cloudformation yaml file?
Thanks in advance.
The first think to consider is this: do you really need to use SQS to start a Step Functions state machine? Can you use API gateway instead? Or could you write your messages to a S3 bucket and use the CloudWatch events to start a state machine?
If you must use SQS, then you will need to have a lambda function to act as a proxy. You will need to set up the queue as a lambda trigger, and you will need to write a lambda that can parse the SQS message and make the appropriate call to the Step Functions StartExecution API.
I’m on mobile, so I can’t type up the yaml right now, but if you need it, I can try to update with it later. For now, here is detailed walkthrough of how to invoke a Step Functions state machine from Lambda (including example yaml), and here is walkthrough of how to use CloudFormation to set up SQS to trigger a Lambda.
EventBridge Pipes (launched at re:Invent 2022) allows you to trigger Step Functions State Machines without need for a Lambda function.
https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-pipes.html
You can find an example here:
https://github.com/aws-samples/aws-stepfunctions-examples/blob/main/sam/demo-trigger-stepfunctions-from-sqs/template.yaml
I need to be able to programmatically create amazon lambda schedule triggers. Like to execute a function every five minutes. I can easily do it with a console, but I need many in different environments, so I need to do it in a script. Java or python or even a cli call will do.
Any ideas?
Thanks
You can use Amazon CloudWatch Events to achieve it.
Take a look at AWS Lambda documentation. These pages show you how to trigger a Lambda function on a schedule:
Using AWS Lambda with Scheduled Events
Run an AWS Lambda Function on a Schedule Using the AWS CLI
You can use this package in Python to run AWS CLI commands.