I have an awslambda.Function that I've configured using the Go CDK.
I want to trigger this lambda every night at 2 AM UTC.
To this end, I am trying to define a CloudWatch Event alike the one from the below JavaScript example:
const newLambda = new lambda.Function(this, 'newLambda',{
runtime: lambda.Runtime.PYTHON_3_8,
code: lambda.Code.fromAsset('functions'),
handler: 'index.handler',
});
const eventRule = new events.Rule(this, 'scheduleRule', {
schedule: events.Schedule.cron({ minute: '0', hour: '1' }),
});
eventRule.addTarget(new targets.LambdaFunction(newLambda))
I believe that I need to use awsevents.NewRule, but I cannot figure out how to pass my lambda
where the "WHAT GOES HERE?" placeholder is written. Note that I am using v2 of the aws-cdk.
event := awsevents.NewRule(stack, aws.String("TriggerDownloadOrdersToS3LambdaEvent"), &awsevents.RuleProps{
Schedule: awsevents.Schedule_Cron(&awsevents.CronOptions{Hour: aws.String("2")}),
Targets: &[]awsevents.IRuleTarget{WHAT GOES HERE?},
})
Does anyone know how I can define a recurring trigger for my lambda using the Go CDK?
You can create an IRuleTarget using the awseventstargets module:
&awsevents.RuleProps{
Schedule: awsevents.Schedule_Cron(&awsevents.CronOptions{Hour: aws.String("2"), Minute: aws.String("0")}),
Targets: &[]awsevents.IRuleTarget{awseventstargets.NewLambdaFunction(lambda, nil)},
},
Related
I created an AWS Lambda function in C# .NET 6.0. This Lambda reaches into our on-premises SQL Server database and retrieves information. I get a count of things that are overdue (can be 0), and I have a class with a Count property that I serialize to JSON. I then log that JSON data to CloudWatch. I also setup an EventBridge scheduled event to run the Lambda once per day. All of that is running just fine.
What I'd like to happen is that logging an overdue Count > 0 sends the count to an SNS topic we have already setup which forwards to Slack. Where I'm having trouble is getting an alarm/metric/filter combination to consistently trigger. My Metric filter is set to use {.count > 0} and that returns the expected Log Group entries when I tested it in the console. My CloudWatch code in AWS CDK is as follows:
new MetricFilter(this, 'OverdueFilter', {
logGroup: logGroup,
metricNamespace: METRIC_NAMESPACE,
metricName: METRIC_NAME,
filterPattern: FilterPattern.numberValue('$.count', '>', 0),
metricValue: '1'
});
const metric = new Metric({
namespace: METRIC_NAMESPACE,
metricName: METRIC_NAME,
dimensionsMap: {
FunctionName: functionName
},
statistic: "Sum",
period: Duration.minutes(30)
});
const alarm = new Alarm(this, 'OverdueAlarm', {
metric: metric,
evaluationPeriods: 1,
actionsEnabled: true,
alarmName: 'overdue-alarm',
alarmDescription: 'Alarm - Overdue',
comparisonOperator: ComparisonOperator.GREATER_THAN_THRESHOLD,
treatMissingData: TreatMissingData.MISSING,
threshold: 0
});
const topic = Topic.fromTopicArn(this, 'SnsTopic', chatbotTopicArn);
alarm.addAlarmAction(new SnsAction(topic));
My sense is that I have some issue with the metric's period vs. how often the Lambda runs. If the Lambda runs once per day (which means that the overdue count is logged once per day), I want the alarm to run (or not) based on the Count value > 0.
My system design
I have created 4 Glue Jobs: testgluejob1, testgluejob2, testgluejob3 and common-glue-job.
EventBridge rule detects SUCCEEDED state of glue jobs such as testgluejob1, testgluejob2, testgluejob3.
After getting Glue Job's SUCCEEDED notification, Glue Trigger run to start common-glue-job.
Problem
I want to use the jobname string in common-glue-job script as parameter
Is it possible to pass parameters to Glue Workflow or Trigger from EventBridge?
The things I tried
Trigger can pass parameters to common-glue-job
https://docs.aws.amazon.com/ja_jp/AWSCloudFormation/latest/UserGuide/aws-resource-glue-trigger.html
Type: AWS::Glue::Trigger
...
Actions:
- JobName: prod-job2
Arguments:
'--job-bookmark-option': job-bookmark-enable
If set Run Properties for Glue Workflow, I cat get it from common-glue-job by using boto3 and get_workflow_run_properties() function. But I have no idea how to put Run Properties from EventBridge by CFn
https://docs.aws.amazon.com/glue/latest/dg/workflow-run-properties-code.html
I set Target InputTransformer of EventBridge Rule, but I'm not sure how to use this value in common-glue-job.
DependsOn:
- EventBridgeGlueExecutionRole
- GlueWorkflowTest01
Type: AWS::Events::Rule
Properties:
Name: EventRuleTest01
EventPattern:
source:
- aws.glue
detail-type:
- Glue Job State Change
detail:
jobName:
- !Ref GlueJobTest01
state:
- SUCCEEDED
Targets:
-
Arn: !Sub arn:aws:glue:${AWS::Region}:${AWS::AccountId}:workflow/${GlueWorkflowTest01}
Id: GlueJobTriggersWorkflow
RoleArn: !GetAtt 'EventBridgeGlueExecutionRole.Arn'
InputTransformer:
InputTemplate: >-
{
"--ORIGINAL_JOB": <jobName>
}
InputPathsMap:
jobName : "$.detail.jobName"
Any help would be greatly appreciated.
If I understand you correctly, you already have information in EventBridge event, but cannot access it from your Glue job. I used the following workaround to do this:
You need to get an event ID from Glue workflow properties
event_id = glue_client.get_workflow_run_properties(Name=self.args['WORKFLOW_NAME'],
RunId=self.args['WORKFLOW_RUN_ID'])['RunProperties']['aws:eventIds'][1:-1]
Get all NotifyEvent events for the last several minutes. It's up to you to decide how much time can pass between the workflow start and your job start.
response = event_client.lookup_events(LookupAttributes=[{'AttributeKey': 'EventName',
'AttributeValue': 'NotifyEvent'}],
StartTime=(datetime.datetime.now() - datetime.timedelta(minutes=5)),
EndTime=datetime.datetime.now())['Events']
Check which event has an enclosed event with the event ID we get from Glue workflow.
for i in range(len(response)):
event_payload = json.loads(response[i]['CloudTrailEvent'])['requestParameters']['eventPayload']
if event_payload['eventId'] == event_id:
event = json.loads(event_payload['eventBody'])
In event variable you get full content of the event that triggered workflow.
I have the following two equivalent lambda functions:
setNotification(date, text):
exports.lambdaHandler = async (event, context) => {
const params = {
Entries: [
{
Source: "com.aws.message-event-lambda",
EventBusName: "",
DetailType: "message",
Detail: JSON.stringify({
title: event.detail.title,
text: event.detail.text,
}),
},
],
};
await eventbridge.putEvents(params).promise();
};
sendNotification(text)
Currently I am using Event bridge to trigget th sendNotification function from the setNotification function, but it triggers the function immediatley.
How can I trigger the sendNotification function at a sppecific date defined by the setNotification function?
Currently I see the following 2 options:
Create code inside the setNotification function that creates a scheduled rule on the EventBridge
Stop using EventBridge and use step functions.
I would like to know what is the correct approach between these two or if there is a better approach which i havent found.
I figured it out, you need a different architecture including a lambda function called by a cron expression on the eventbridge that checks a DB for entries to then send notifications to.
More information on scheduling systems on AWS in the following link:
https://aws.amazon.com/blogs/architecture/serverless-scheduling-with-amazon-eventbridge-aws-lambda-and-amazon-dynamodb/
Hi i have been trying many possibilities, but now i would need some help.
I am using aws-cdk to create architecture by code and so far things have going well. Now i am running into this issue:
The final policy size is bigger than the limit (20480)
In understand what it means, but i have no idea how to solve it.
I am creating a lambdafunction to handle all requests:
const router = new lambda.Function(this, apiName + '-handler-temp', {
runtime: LambdaRuntime, // execution environment
code: lambda.Code.fromAsset('bin/lambda'), // code loaded from "lambda" directory
handler: 'index.handler', // file is "index", function is "handler"
vpc: vpc,
environment: {
DB_HOST: props?.rdsEndpoint as string,
DB_USER: props?.rdsDbUser as string,
DB_PASS: props?.rdsDBPass as string,
DB_PORT: props?.rdsPort as string,
DB_DIALECT: props?.rdsDbSchema as string,
DB_DATABASE: props?.rdsDBName as string,
},
layers: [layer],
timeout: Duration.seconds(30),
memorySize: 1024,
})
and the LambdaRestApi is defined like this:
const api = new LambdaRestApi(this, apiName, {
handler: router,
proxy: false,
cloudWatchRole: false,
description: 'API for Backend',
deployOptions: {
stageName: 'prod',
},
domainName: domainProperties,
})
I am creating Endpoints where i am using 23 times addMethod.
e.g.
const user = api.root.addResource('user')
user.addMethod(HttpMethod.POST)
user.addMethod(HttpMethod.GET)
user.addMethod(HttpMethod.PATCH)
since only one lambda is used to be invoked from apigateway, i am curious, how i can get control of only one policy to be used for lambda execution and it is not creating a new one every time.
I also tried to add property
role: role to the lambda function with this role definition:
const role = new Role(this, apiName + 'ApiGWPermissions', {
assumedBy: new ServicePrincipal('apigateway.amazonaws.com'),
})
role.addToPolicy(
new PolicyStatement({
resources: ['*'],
actions: ['lambda:InvokeFunction'],
})
)
but then i am running into different errors.
Has someone solved this riddle?
Cheers
As suggested in the CDK issue which Ian Walters mentioned, stripping the generated method permissions solved this for me. I'm using .Net but I'd expect that the approach should work for all language implementations. This function removes the permissions:
public void StripMethodPermissions(ConstructNode node) {
foreach (var child in node.Children) {
if (child is Amazon.CDK.AWS.APIGateway.Method) {
var method = ((Amazon.CDK.AWS.APIGateway.Method)child);
var permissions = method.Node.Children.Where(c => c is Amazon.CDK.AWS.Lambda.CfnPermission);
foreach (var permission in permissions) {
child.Node.TryRemoveChild(permission.Node.Id);
}
}
if (child.Node.Children.Length > 0) StripMethodPermissions(child.Node);
}
}
I'm using the technique like this:
var api = new RestApi(this, "MyRestApi", new RestApiProps {
RestApiName = "MyRestApiV1"
});
var handlerLambda = new Function(this, "RequestHandler", new FunctionProps {
Runtime = Runtime.DOTNET_CORE_3_1,
...
});
// Add resources and methods which use the handlerLambda here
// Remove all generated permissions
StripMethodPermissions(api.Root.Node);
// Add a single invoke permission for the lambda
handlerLambda.GrantInvoke(new ServicePrincipal("apigateway.amazonaws.com"));
Thanks to nija-at for showing the way
#Elliveny has the correct answer. Here is a code snitbit for Python which does the same thing (as I cannot post formatted code in comments):
from aws_cdk import (
aws_lambda as _lambda,
aws_events as events,
aws_iam as iam,
core,
)
for child in self.node.children:
if isinstance(child, events.Rule):
for eventChild in child.node.children:
if isinstance(eventChild, _lambda.CfnPermission):
child.node.try_remove_child(eventChild.node.id)
Remember that if you do this, you still need to grant invoke on your lambda for the "events.amazonaws.com" ServicePrincipal. Something like:
my_lambda.add_permission(
"RuleInvoke",
principal=iam.ServicePrincipal("events.amazonaws.com"),
action="lambda:InvokeFunction",
source_arn=f"arn:aws:events:{core.Aws.REGION}:{core.Aws.ACCOUNT_ID}:rule/your-rule-name-here*",
)
I hit a similar issue. There is a CDK issue that might help resolve it if addressed,
https://github.com/aws/aws-cdk/issues/9327
Its also worth noting that by default lambda integrations have allowTestInvoke set to true, which pretty much is going to double the policy document size.
I'm not sure if you can alter the integration options for the lambda with LambdaRestApi though, I'm using RestApi directly.
A short term fix might be to use RestApi rather than LambdaRestApi and create the lambda integration directly with the allowTestInvoke option set to false.
The other thing I did that helped was to just create more than one lambda that worked the same way, but got attached to different routes (e.g. same code, permissions etc, just different logical id) to also reduce the policy document size a bit.
I'm a bit pressed for development time hence the work-arounds. Personally, I think the right solution would be to fix it in the CDK and propose a PR such that LambdaRestApi just did what the user would expect, wild-card permission for the lambda for all
You can increase the memory size of lambda by doing this
Type: AWS::Serverless::Function
Properties:
CodeUri: src/promoCodes/
Role: !GetAtt FunctionRole.Arn
Handler: promoCodesListCsvPDF.promoCodesListCsvPDF
Policies:
- AWSLambdaExecute # Managed Policy
- AWSLambdaBasicExecutionRole # Managed Policy
MemorySize: 512
Layers:
- !Ref NodeDependenciesLayer
Events:
promoCodesListCsvPDFEvent:
Type: Api
Properties:
Path: /api/promo-codes/csv-pdf
Method: POST
This tutorial shows how to Invoke a Lambda from CodePipeline passing a single parameter:
http://docs.aws.amazon.com/codepipeline/latest/userguide/how-to-lambda-integration.html
I've built a slackhook lambda that needs to get 2 parameters:
webhook_url
message
Passing in JSON via the CodePipeline editor results in the JSON block being sent in ' ' so it can't be parsed directly.
UserParameter passed in:
{
"webhook":"https://hooks.slack.com/services/T0311JJTE/3W...W7F2lvho",
"message":"Staging build awaiting approval for production deploy"
}
User Parameter in Event payload
UserParameters: '{
"webhook":"https://hooks.slack.com/services/T0311JJTE/3W...W7F2lvho",
"message":"Staging build awaiting approval for production deploy"
}'
When trying to apply multiple UserParameters directly in the CLoudFormation like this:
Name: SlackNotification
ActionTypeId:
Category: Invoke
Owner: AWS
Version: '1'
Provider: Lambda
OutputArtifacts: []
Configuration:
FunctionName: aws-notify2
UserParameters:
- webhook: !Ref SlackHook
- message: !Join [" ",[!Ref app, !Ref env, "build has started"]]
RunOrder: 1
Create an error - Configuration must only contain simple objects or strings.
Any guesses on how to get multiple UserParameters passing from a CloudFormation template into a Lambda would be much appreciated.
Here is the lambda code for reference:
https://github.com/byu-oit-appdev/aws-codepipeline-lambda-slack-webhook
You should be able to pass multiple UserParameters as a single JSON-object string, then parse the JSON in your Lambda function upon receipt.
This is exactly how the Python example in the documentation handles this case:
try:
# Get the user parameters which contain the stack, artifact and file settings
user_parameters = job_data['actionConfiguration']['configuration']['UserParameters']
decoded_parameters = json.loads(user_parameters)
Similarly, using JSON.parse should work fine in Node.JS to parse a JSON-object string (as shown in your Event payload example) into a usable JSON object:
> JSON.parse('{ "webhook":"https://hooks.slack.com/services/T0311JJTE/3W...W7F2lvho", "message":"Staging build awaiting approval for production deploy" }')
{ webhook: 'https://hooks.slack.com/services/T0311JJTE/3W...W7F2lvho',
message: 'Staging build awaiting approval for production deploy' }