I have a requirement where I need to trigger my lambda function when all of the glue crawlers have run & my data is ready in redshift to be queried.
I have setup the following AWS cloudwatch rule but it triggers the lambda if any of the crawlers have succeeded.
{
"detail-type": [
"Glue Crawler State Change"
],
"source": [
"aws.glue"
],
"detail": {
"crawlerName": [
"crw-raw-recon-the-hive-ces-cashflow",
"crw-raw-recon-the-hive-ces-position",
"crw-raw-recon-the-hive-ces-trade",
"crw-raw-recon-the-hive-ces-movement",
"crw-raw-recon-the-hive-ces-inventory"
],
"state": [
"Succeeded"
]
}
}
Now my question is there a way I could enforce the lambda to be triggered only when all of them have succeeded?
Also, I am not sure if redshift generates any similar events when it receives data.
Related
I have a setup wherein I need to trigger a lambda function when my glue crawler has run and data is ready in redshift. Is there a way to create such a trigger?
Edit:
I added an Event bridge rule for crawler state change, that works and triggers the lambda function but it triggers when any of my crawlers are running. I want to isolate it to trigger only after a specific crawler is run. I tested with the code below but it doesn't seem to pick my crawler name. Is there any other way to specify the crawler name in the rule or am I making a syntactical error?
{
"source": ["aws.glue"],
"detail-type": ["Glue Crawler State Change"],
"eventName": "crawler_name",
"detail": {
"state": ["Succeeded"]
}
}
Solution: Add an EvenBridge rule with the following Event Pattern
{
"source": ["aws.glue"],
"detail-type": ["Glue Crawler State Change"],
"detail": {
"crawlerName": ["newton_pfi_new_raw_to_source"],
"state": ["Succeeded"]
}
}
I have the following event pattern that I want my lambda function to be triggered from but i'm unable to find any documentation to allow this to be set in the zappa_settings.json file. I'm only able to find SNS and EventBridge cron expressions.
{
"source": ["aws.ec2"],
"detail-type": ["AWS API Call via CloudTrail"],
"detail": {
"eventSource": ["ec2.amazonaws.com"],
"eventName": ["RunInstance"]
}
}
How can I get my lambda function to create and trigger from this event?
I can add an SNS topic as a trigger to the lambda function and point EventBridge at the topic. Not perfect as the UI allows EventBridge to trigger Lambda functions directly but it works. Just need to add the following into the zappa_settings file and setup the EventBridge outside zappa.
"events": [
{
"function": "auto_tagger.lambda_handler",
"event_source": {
"arn": "arn:aws:sns:eu-west-1:1234567890:trigger-test",
"events": [
"sns:Publish"
]
}
}
]
I have an AWS lambda function that provides configuration objects. Whenever this lambda function is updated(i.e deployed again), I need to trigger another lambda function, that detects changes in those config objects and takes some action. How do I monitor this lambda deployment, which cloud watch event do I need to subscribe to?
I assume that your lambda deployments are not managed by CodeDeploy. If so, I would recommend looking into creating a CloudTrial trial.
Once CT trial is created with default settings, it will monitor all management API calls to your lambda function. One of them is UpdateFunctionCode. Thus you can create a CloudWatch rule for AWS API Call via CloudTrail. The rule would be triggered on the function update API call.
Example of such a rule:
{
"source": [
"aws.lambda"
],
"detail-type": [
"AWS API Call via CloudTrail"
],
"detail": {
"eventSource": [
"lambda.amazonaws.com"
],
"eventName": [
"UpdateFunctionCode"
]
}
}
Then you can trigger a second lambda, based on the captured update event of the first function.
Based on #Marcin's suggestion, used console log to print the event. The below rule helped to filter a specific function;
{
"source": [
"aws.lambda"
],
"detail-type": [
"AWS API Call via CloudTrail"
],
"detail": {
"eventSource": [
"lambda.amazonaws.com"
],
"eventName": [
"UpdateFunctionCode20150331v2"
],
"responseElements": {
"functionName": [
"myFunction"
]
}
}
}
I want to be able to setup an AWS CloudWatch event rule that will trigger to an SNS topic whenever one of my Step Functions completes (either success or failure). I do not want this to run for all Step Functions, but there will be an indeterminate number of them based on a common name prefix. Ideally, I'd like to be able to do something like this, but it appears that wildcards are not allowed in Event Patterns. Are there any creative ways to work around this?
{
"source": [
"aws.states"
],
"detail-type": [
"Step Functions Execution Status Change"
],
"detail": {
"status": [
"FAILED",
"SUCCEEDED"
],
"stateMachineArn": [
"arn:aws:states:us-west-1:123456789012:stateMachine:Prefix-*"
]
}
}
Wildcards are not supported in Cloudwatch event rule according to AWS official forum.
You will have to add all the arn's in the state machine ARN list. To do it easily you may write a script that does the following:
Get all the state machine names with specific prefix.
Update the Cloudwatch Event Rule to include all the state machine arn's with specific prefix.
My solution is below:
{
"source": ["aws.states"],
"detail-type": ["Step Functions Execution Status Change"],
"detail": {
"status": ["SUCCEEDED", "FAILED"],
"stateMachineArn": [ { "prefix": "arn:aws:states::us-west-1:123456789012:stateMachine:prefix-" } ]
}
}
I am trying to make a rule trigger on any API call for creation, but I haven't had any success.
I have another rule that triggers whenever and ec2 instance is running which works, but this rule does not trigger for RunInstances although I see the cloudtrail logs with RunInstances in the API log.
I have made a CloudWatch log from the cloudtrail and see the events that should be triggering it, but they don't. Is there a step I am missing? What are the necessary components to have CloudWatch properly trigger on API calls?
The rule mentioned:
{
"source": [
"aws.cloudtrail"
],
"detail-type": [
"AWS API Call via CloudTrail"
],
"detail": {
"eventSource": [
"cloudtrail.amazonaws.com"
],
"eventName": [
"CreateUserPool",
"CreateImage",
"CreateCacheCluster",
"RunInstances",
"CreateActivation",
"RunJobFlow ",
"CreateVault",
"CreateDeliveryStream",
"CreateStream",
"CreateCluster",
"CreateDBInstance",
"CreateHostedZone",
"CreateBucket",
"CreateLaunchConfiguration",
"CreateStack",
"CreateEnvironment",
"CreateWorkspaces"
]
}
}
To add more detail here is the other rule I used to test CloudWatch
{
"source": [
"aws.ec2"
],
"detail-type": [
"EC2 Instance State-change Notification"
],
"detail": {
"state": [
"running"
]
}
}
This is the log of the startInstances and the following Lambda function running.
Cloudtrail of the the lambda function specifically which functions normally.
Here is the API call which should also initiate it according to the rule, but it does not.
Hopefully these images make it clearer as to what I am having trouble with.
P.S. I didn't know how much info I should consider confidential, so I over censored
I misunderstood the rule I created. I thought "AWS API calls from Cloudtrail" when using the cloudtrail Service meant all the information stored in Cloudtrail. That does not seem to be the case.
As I mention in my comment, I am looking for other solutions for my problem, but I will make a separate question for that. Thanks!