Serverless Framework AWS StepFunction not working - amazon-web-services

I have a problem that so far I'm unable to identify the root cause.
I have an AWS step machine that should be invoked once a file is uploaded to an S3 bucket.
So far when I upload the file to the S3 bucket, the lambda function that is defined in the StartAt key (StartAt: ImgUploadedEvent) starts as I can see in the lambda logs.
Here is the code:
stepFunctions:
stateMachines:
ValidateImageStateMachine:
loggingConfig:
level: ALL
includeExecutionData: true
destinations:
- Fn::GetAtt: [ StepFuncLogGroup, Arn ]
definition:
Comment: "This state function validates the images after users upload them to S3"
StartAt: ImgUploadedEvent
States:
ImgUploadedEvent:
Type: Task
Resource:
Fn::GetAtt: [ImgUploaded, Arn]
End: true
Below is the lambda function that is declared as the start of the StepMachine
This lambda function as I can see from the logs indeed get called once I modified an Object in S3
functions:
ImgUploaded:
handler: src/stepfunctions/imageWasUploadedEvent.handler
events:
- s3:
bucket: !Ref AttachmentsBucket
existing: true
iamRoleStatements:
- Effect: "Allow"
Action:
- "states:StartExecution"
Resource:
- "*"
To check that the Step Function was working I created a log group and added it to the Step Function.
resources:
Resources:
StepFuncLogGroup:
Type: AWS::Logs::LogGroup
Properties:
LogGroupName: /aws/stepfunctions/${self:service}-${self:provider.stage}
I see this cloud watch log group correctly associated with the Step function in the AWS Console.
However when I upload an object to S3, I do see in the logs of the lambda functions that it was invoked, but I can not see any logs on the Step Function log group
My question is:
Is the Step Function indeed working and it is just an issue with the logs in the Step Function?
Or it is that the Step Function itself is not working and the lambda function is working just a lambda function totally independent of the Step Function?
What I have to do so the lambda function gets trigger as part of the Step Function?
BR

After studying how StepFunctions work I finally arrived to the conclusion that this is a wrong pattern.
In no place on the documentation of the plugins or Amazon it says that what I did is a pattern
StepFunctions can be started by events on CloudWatch and those events could be originated on a change on S3. That is not what I did here
There is no a direct link between an action on S3 and a StepFunction
The link in AWS documentation could confuse someone to think otherwise. Here is the title Starting a State Machine Execution in Response to Amazon S3 Events
But the state machine does not start because an S3 events but because the CloudWatch log that this event generates
A lambda proxy function is a good way of invoking an State Machine. This is a easy to use pattern and very common as we can use it with SQS etc
So the correct response to this question is
The state machine does not start because it is never invoked. We just called a lambda function that was used as the StartAt for an StateMachine. This does mean I invoked the State Machine.
That is the reason why there is no logs for the state machine meanwhile there are correct logs for the lambda function
Hope this response helps
I will add more details and reference to this response
BR

Or it is that the Step Function itself is not working and the lambda function is working just a lambda function totally independent of the Step Function?
To verify whether lambda was invoked as part of step function or not can't you just check execution history from the step function console. Also unless you have explicitly configured s3 to publish events to lambda, your lambda will not be automatically invoked upon uploading files to s3.
What I have to do so the lambda function gets trigger as part of the Step Function?
To be able to call trigger step function on file-upload to s3 you can follow this tutorial: https://docs.aws.amazon.com/step-functions/latest/dg/tutorial-cloudwatch-events-s3.html

Related

How to programmatically set up EventsBridge events for Lambdas

I have set up 2 lambda functions, deployed with AWS SAM. The first one uses the JS AWS SDK to run putRule and putTarget to trigger the second lambda with a cron job. When I run the first lambda, I see both the rule and target correctly set up in EventsBridge.
I also create the following permission for the second Lambda in my AWS SAM template
InvokePermission:
Type: AWS::Lambda::Permission
Properties:
FunctionName: !Ref MyLambda
Action: lambda:InvokeFunction
Principal: 'events.amazonaws.com'
and can see the Policy in the console
The only result I see of this cron event (at the timestamp I've chosen for the rule) is a failed invocation of the second Lambda, and CloudWatch doesn't provide any useful information
Any idea of why this is failing or how to retrieve any error? Might "events.amazonaws.com" be the wrong Principal for that?
I am looking into EventSourceMapping but I can't see my case anywhere in the docs

How to pass parameters in lambda by cloud custodian?

I've one scenario like I want to invoke one lambda function by cloud custodian and want to pass newly created bucket name to that lambda function. is there any way to pass parameters to the lambda function from the custodian event? Thanks
-- below is my cloud custodian policy:-
policies:
name: lambda-s3-configure-standards-real-time
resource: aws.lambda
description: |
This policy is triggered when a new S3 bucket is created and it will invoke another lambda.
mode:
type: cloudtrail
events:
- CreateBucket
role: some-role
timeout: 200
actions:
- type: invoke-lambda
function: Lambda-function-name
Have you tried checking the lambda function's event dictionary that is passed as the payload to your invoked lambda function? I believe the bucket name should already be present as part of the payload readily.
As an example, when the bucket is created via console your payload should contain the Bucket Name at $.detail.requestParameters.bucketName

Enable Amazon S3 bucket to Trigger Lambda function using SAM

I want to trigger a Lambda function whenever a file is uploaded to an Amazon S3 bucket with a certain prefix and suffix using SAM. Right now I'm using this code but it's giving error
"The ARN is not well formed(Service: Amazon S3; Status Code: 400; Error Code: InvalidArgument"
Edit:
This is working but it's not giving an option to add suffix or prefix.
In the NotificationConfiguration you're simply using !Ref HelloWorld to reference your function, however, as the documentation for AWS::Serverless::Function states:
When the logical ID of this resource is provided to the Ref intrinsic function, it returns the resource name of the underlying Lambda function.
If we look at the documentation for the LambdaConfiguration it states:
The Amazon Resource Name (ARN) of the AWS Lambda function that Amazon S3 invokes when the specified event type occurs.
If you simply change the !Ref HelloWorld to !GetAtt HelloWorld.Arn it should pass in the correct value.
Beware however of the remark that is made on the NotificationConfiguration documentation that if you create the bucket at the same time as you're creating the notification configuration, you might end up with a circular dependency, since you also need to add a AWS::Lamdba::Permission (or use the Events of AWS::Serverless::Function) to allow S3 to invoke your lambda function.

Unable to add cloudfront as trigger to lambda function

Hi I've followed this instruction try to resize image with Cloudfront and lambda#edge. When I tried to test the resized image, I keep getting the error message below:
The Lambda function associated with the CloudFront distribution is
invalid or doesn't have the required permissions.
So I checked the lambda functions created by cloud formation provided by the article I mentioned in the beginning, and I found there's no trigger in it.
I've tried to set it manually but getting the error message below:
CloudFront events cannot be associated with $LATEST or Alias. Choose
Actions to publish a new version of your function, and then retry
association.
I followed the instruction in the error message; publish, and add Cloudfront as trigger but it seems there's no way to apply it. It's still running the one without Cloudfront as the trigger.
Is there any way to set Cloudfront as trigger and make this work properly?
For people Googling "The Lambda function associated with the CloudFront distribution is invalid or doesn't have the required permissions":
I got that error and struggled to debug it. It turned out there were some programmatic errors inside my Lambda that I had to resolve. But, how do you debug it if, when hitting Cloudfront you keep getting "The Lambda function associated with the CloudFront distribution is invalid or doesn't have the required permissions". That, and there's nothing inside the Cloudwatch logs.
My Lambda was defined in Cloudformation inside a AWS::Lambda::Function's ZipFile attribute. I ended up going to the Lambda service inside AWS and creating a Lambda test payload corresponding to my Cloudfront event as documented here: Lambda#Edge Event Structure. Then, I could debug the Lambda inside the Lambda console without having to hit Cloudfront or having to navigate to Cloudwatch logs.
I see a couple of you guys stating that the root cause of the issue was not a permissions issue and an issue with your code. Which is likely the correct root cause. Cloud front tends to use a 403 error for everything even a basic 404 will show up as a 403 in most cases.
I have also seen some of the comments above stating that you could not find any logs associated with the error in lambda. I think this is most likely because you guys are looking for the logs on us-east-1 and dont live on the east coast of the USA. The logs will be in your local region where they are executed. So choose the region in closest proximity to where you are sitting and you will likely find the log group there.
For other ppl suffering from the poor quality of dev articles from aws blog; I found it's due to the wrong S3 bucket policy. The article says:
ImageBucketPolicy:
Type: AWS::S3::BucketPolicy
Properties:
Bucket: !Ref ImageBucket
PolicyDocument:
Statement:
- Action:
- s3:GetObject
Effect: Allow
Principal: "*"
Resource: !Sub arn:aws:s3:::${ImageBucket}/*
- Action:
- s3:PutObject
Effect: Allow
Principal:
AWS: !GetAtt EdgeLambdaRole.Arn
Resource: !Sub arn:aws:s3:::${ImageBucket}/*
- Action:
- s3:GetObject
Effect: Allow
Principal:
AWS: !GetAtt EdgeLambdaRole.Arn
Resource: !Sub arn:aws:s3:::${ImageBucket}/*
It turns out you have to grant the permissions to allow other actions besides of GetObject and PutObject, because it needs to create folders in the bucket.
Simply the problem is resolved by changing it to s3:*
For me, the missing cloud front trigger on the lambda screen was because I was not in us-east-1 region
I ran into the same error message with no log in CloudWatch. I finally noticed that my Python runtime handler was index.handler while my index.py defined lambda_handler. After changing my Python runtime handler to index.lambda_handler, the error went away. HTH.
If you found this answer googling "The Lambda function associated with the CloudFront distribution is invalid or doesn't have the required permissions", this can be caused if your function is not wired correctly from cloudformation. For example given yaml:
Code: ./src/ # or CodeUri ./src/
Handler: foo.bar
Double check that ./src/foo.js has exports.bar = function...
When I changed "Include body" in Lambda Function Trigger from "Yes" to "No" it started working.
I had to delete and create CloudFront trigger again to change that setting.
just reading an article from here.
If you create a lambda in one region and use it with cloudfront (and later be requested by user in other edge-region), the issue is due to lambda does not have enough cloudwatch log permission.
Check this, all credits go to author.
https://dev.to/aws-builders/authorizing-requests-with-lambdaedge-mjm

AWS Lambda and S3 trigger event data

I have a lambda function that I need to run eveytime there is a change in my s3 Bucket. I have added the trigger and it is working just fine, but I was wondering if there is any way to limit the scope the lambda function is to be run... for example Instead of running over the entire bucket, it runs only in the folder (inside the bucket) that change has been made?! or something like that..!
You can specify rules:
- s3:
bucket: photos
event: s3:ObjectCreated:*
rules:
- prefix: uploads/
- suffix: .jpg
See the functions/events/s3 section in the yml definition.
Per this AWS announcement, you can add prefix or suffix restrictions for S3 event triggers.