Is it possible to spin up infrastructure such as a SQS queue in front of a Lambda function using AWS SAM without a API gateway?
I only see options to sam local invoke "Lambda" -e event.json and sam local start-api
When I run my lambda which is trying to read messages from the message queue it is not finding the Message Queue URL as referenced below:
NotificationFunction:
Type: AWS::Serverless::Function # More info about Function Resource: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#awsserverlessfunction
Properties:
Handler: index.handler
Runtime: nodejs8.10
Role: !Sub ${ConsumerLambdaRole.Arn}
Timeout: 10
Environment: # More info about Env Vars: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#environment-object
Variables:
NODE_ENV: 'dev'
MANDRILL_API_KEY: 'PUyIU3DAxJzIlt7KbBE5ow'
SQS_URL: !Ref MessageQueue
MessageQueue:
Type: AWS::SQS::Queue
Properties:
VisibilityTimeout: 60
RedrivePolicy:
deadLetterTargetArn: !Sub ${DeadLetterQueue.Arn}
maxReceiveCount: 10
# this is where any failed messages will go to
DeadLetterQueue:
Type: AWS::SQS::Queue
As the comment suggests, sam local invoke will only invoke the code referenced by your lambda resource in the template - it will not create any resources per-se.
What you can do though, is launch your template and either invoke the lambda function using the CLI or via sam local invoke (though you will need to perform an STS role assumption to acquire the correct role for the locally invoked function). However, I assume what you are attempting to do is local testing. In that case, you could do the following:
Download localstack [1]
Use local stack to run a local instance of the SQS service
Configure your function to use the local stack endpoint (of which there is are some good examples available [2][3])
Personally I prefer to avoid local testing, and will tend to launch the template and locally invoke via assumed credentials for debugging purposes. I tend to have an integration test suite that I will execute against sandboxed AWS environment for testing purposes. That said, it takes a fair amount of effort to set that up.
[1] https://github.com/localstack/localstack
[2] https://github.com/billthelizard/aws-sam-localstack-example
[3] http://www.piotrnowicki.com/python/aws/2018/11/16/aws-local-lambda-invocation/
Related
I am using SAM to develop a lambda that is triggered by SNS.
When I am using SAM, it seems I get API GATEWAY associated to the Lambda when I deploy.
It is nice in dev, but not in prod.
How would I develop a Lambda without deploying it attached to an API GATEWAY?
"I was not aware of Lambda runtime API"
Using the Lambda runtime API and the AWS SDKs, as a developer, you can build more flexible Lambda functions with the exact functionality that you want.
For example, you can build Lambda functions that can invoke multiple AWS Services to perform a given use case. For example, from a Lambda function, I want to detect PPE information in images within an Amazon S3 bucket, store the results in an Amazon DynamoDB table, and use the SES service to email the results.
Here are a few end to end tutorials.
For example - here are two JavaScript examples:
Creating and using Lambda functions
Creating scheduled events to execute AWS Lambda functions
Here are some Java runtime examples:
Creating an AWS Lambda function that detects images with Personal Protective Equipment
Creating an Amazon Web Services Lambda function that tags digital assets located in Amazon S3 buckets
Creating an ETL workflow by using AWS Step Functions and the AWS SDK for Java
UPDATE
Once you test your code in an IDE, deploy it and test it from the Lambda console. Once it is successfully invoked, you will see a message like this:
Another option I found, this time within SAM is define the function with a different Event source.
The Lambda will be exposed if it's event source is defined as Api
You can define it as many other things (S3,Sns,Sqs etc)
Here is an example of a function triggered by and http event via api gateway, and the other one, by Sns. The config for SNS is not complete, Check this link for full details https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md
in template.yaml Notice the Events section of each Lambda definition.
Resources:
HelloWorldFunction:
Type: AWS::Serverless::Function # More info about Function Resource: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#awsserverlessfunction
Properties:
CodeUri: hello-world/
Handler: app.lambdaHandler
Runtime: nodejs14.x
Events:
HelloWorld:
Type: Api # More info about API Event Source: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#api
Properties:
Path: /hello
Method: get
BooBooFunction:
Type: AWS::Serverless::Function # More info about Function Resource: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#awsserverlessfunction
Properties:
CodeUri: booboo/
Handler: app.lambdaHandler
Runtime: nodejs14.x
Events:
BooBoo:
Type: Sns # More info about API Event Source: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#api
I've created AWS Lambda in C# using Visual Studio that returns some JSON from the API endpoint. Now I want to run that lambda locally. All the examples use AWS SAM, but they create a new function using the SAM template.
When I run the command sam local start-lambda I get an error saying that the template isn't found. So what is certain is that I need template.yaml, but I'm not sure is there a way to generate this template for existing Lambda?
Any help is appreciated!
Check out the Template Anatomy resource on the AWS documentation.
You might find this example helpful (it's greatly simplified). I use NodeJS for development, but the differences between programming languages when you're creating a SAM Template are trivial. The example is an outline for a simple Lambda function someFunction being invoked by an API Gateway (HTTP) event.
AWSTemplateFormatVersion: '2010-09-09'
Transform: 'AWS::Serverless-2016-10-31'
Description: My Serverless Application
Parameters:
# Manually define this in AWS IAM for just the services needed.
lambdaExecutionRole:
Description: 'Required. The role used for lambda execution.'
Type: 'String'
Default: 'arn:aws:iam::nnnnnnnnnnnn:role/LambdaExecutionRole'
Globals:
Function:
Runtime: nodejs10.x
# Environment:
# Variables:
# NODE_ENV: test
# DEBUG: myapp:foo
Resources:
performSomeFunction:
Type: 'AWS::Serverless::Function'
Properties:
FunctionName: performSomeFunction
Handler: lambda.someFunction
CodeUri: ./
Description: description of the function being performed
MemorySize: 256
Timeout: 60
Role:
Ref: lambdaExecutionRole
Events:
# API Gateway proxy endpoint.
ProxyApiRoot:
Type: Api
Properties:
Path: '/'
Method: ANY
ProxyApiGreedy:
Type: Api
Properties:
Path: '/{proxy+}'
Method: ANY
As you're getting started with AWS Lambda, one of the big concepts to keep in mind is how your function will be triggered. Functions are triggered by different kinds of events, and there can be many many different types of events. I tend to use API Gateway, Simple Queue Service and CloudWatch Events to trigger mine, but it entirely depends on your use case.
It turned out that you can export Lambda function, and get the generated .yaml template, which was exactly what I needed.
When trying to deploy to AWS using AWS SAM CLI my Lambda functions using the following script:
aws cloudformation deploy --template-file /Users/ndelvalle/Projects/foo/functions/packaged-template.yaml --stack-name foo --region sa-east-1 --capabilities CAPABILITY_IAM --no-fail-on-empty-changeset
I got the following error in the stack events:
API: iam:CreateRole User: arn:aws:iam::user/nico is not authorized to perform: iam:CreateRole on resource
This is because I don't have role creation permissions on my account. That is why I wonder if there is a way to define pre-created roles to my lambdas, so the script does not need to create the role.
There is more information needed to answer this for you. Many different permissions come into play when deploying with SAM. I implemented SAM template for my company to manage our lambda stacks. We needed to give our Java Developers working on the stacks the same permissions that the SAM template implicitly and explicitly creates, beyond just creating roles. For this we created several special groups in IAM that we attached our Serverless Devs too. It is possible to assign specific predefined roles to Lambdas, https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#awsserverlessfunction. If you define a role in the template, it does not automatically create a role. However, if you are declaring other resources like Api Gateway and DynamoDB, you will still run into problems.
So long story short, if you are working with SAM its better you have your Sys admin give you permissions to Create role, and you will need other permissions as well, or have the deployment of the SAM template done by a Job runner like Jenkins (that has the permissions). It it is too permissive for your team/company, maybe SAM is not a good solution... Best to switch to something like pure CloudFormation and abandon a developer oriented workflow. Somethings to think about, hope its helpful.
You can use the Role property as mentioned in the docs for AWS::Serverless::Function
A sample template which creates a new lambda without creating a new role,
Transform: AWS::Serverless-2016-10-31
Description: >
sam-app
Sample SAM Template for sam-app
# More info about Globals: https://github.com/awslabs/serverless-application-model/blob/master/docs/globals.rst
Globals:
Function:
Timeout: 3
Tracing: Active
Api:
TracingEnabled: True
Resources:
HelloWorldFunction:
Type: AWS::Serverless::Function # More info about Function Resource: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#awsserverlessfunction
Properties:
CodeUri: hello-world/
Handler: app.lambdaHandler
Runtime: nodejs18.x
Role: <ARN of ROLE>
Architectures:
- x86_64
Events:
HelloWorld:
Type: Api # More info about API Event Source: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#api
Properties:
Path: /hello
Method: get
Metadata: # Manage esbuild properties
BuildMethod: esbuild
BuildProperties:
Minify: true
Target: "es2020"
Sourcemap: true
EntryPoints:
- app.ts
Outputs:
# ServerlessRestApi is an implicit API created out of Events key under Serverless::Function
# Find out more about other implicit resources you can reference within SAM
# https://github.com/awslabs/serverless-application-model/blob/master/docs/internals/generated_resources.rst#api
HelloWorldApi:
Description: "API Gateway endpoint URL for Prod stage for Hello World function"
Value: !Sub "https://${ServerlessRestApi}.execute-api.${AWS::Region}.amazonaws.com/Prod/hello/"
HelloWorldFunction:
Description: "Hello World Lambda Function ARN"
Value: !GetAtt HelloWorldFunction.Arn
HelloWorldFunctionIamRole:
Description: "Implicit IAM Role created for Hello World function"
Value: !GetAtt HelloWorldFunctionRole.Arn
So I'm trying to convert an existing spring boot application to an AWS lambda and using SAM.
I'm trying to use aws-sam-cli to try my lambda locally, however with my SAM setup I am getting: Template does not have any APIs connected to Lambda functions
When I do: sam local start-api
My template.yml:
AWSTemplateFormatVersion : '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: foo
Resources:
MailFunction:
Type: AWS::Serverless::Function
Properties:
Handler: bar.LambdaHandler::handleRequest
Runtime: java8
CodeUri: target/foo-bar-1.0.jar
Timeout: 300
MemorySize: 1024
Events:
Timer:
Type: Schedule
Properties:
Schedule: rate(1 day)
Any idea what I'm doing wrong? It looks correct as far as I can tell from https://blog.couchbase.com/aws-serverless-lambda-scheduled-events-tweets-couchbase/ + https://docs.aws.amazon.com/lambda/latest/dg/tutorial-scheduled-events-schedule-expressions.html
You didn't add any API Gateway event to your function. And start-api spawn a local API Gateway.
You need to add at least one Api event to your Events section.
Events:
[...]
Api:
Type: Api
Properties:
Path: /myresource
Method: get
If you just have a Schedule event, try to use generate-event to create such an event.
sam local generate-event schedule ...
and invoke function e.g. sam local invoke function-name -e event_file.json (see)
For Googlers:
Check whether you have an Event with Type: Api
ALSO check whether you have run sam build (very important)
Use the --debug flag so you will know what is going on
As of 2020/7/13, Type: HttpApi does not work with sam local start-api. See issue.
This error message also displays if you are trying to test a websocket API locally. Unfortunately, local testing of websockets is not currently supported - see https://github.com/awslabs/aws-sam-cli/issues/896.
I ran into this error too even when I did have an Api event defined in my SAM template. The problem was that I had a previous template in my .aws-sam/build/ directory which didn't have the Api event defined (from a previous run of sam build). Cleaning out the build directory fixed it.
I am getting this error, but I have function that is working with the HttpApi, it appears that current version of sam does not support HttpApi.
CLI Version
SAM CLI, version 0.52.0
Example Function
FeedsFunction:
Type: AWS::Serverless::Function
Properties:
CodeUri:
Description: "Function that handles feeds"
Events:
Handler:
Type: HttpApi
Properties:
ApiId: !Ref FeedsApi
Path: /
Method: get
Handler: api
MemorySize: 1024
Runtime: go1.x
Timeout: 5
Tracing: Active
There is currently an open issue on GitHub for adding support: https://github.com/awslabs/aws-sam-cli/issues/1641
I got this error when I had a whitespace error in my AWS::Serverless::Function definition, specifically Environment needed to be a child of Properties but was on the same level. Correcting the whitespace made this error disappear. Nodejs 10.15.
I'm creating a Nodejs microservice for AWS Lambda. I scaffolded by project using AWS Codestar, and that set me up with a CI/CD pipeline that automatically deploys the lambda function. Nice.
The issue is that every time it deploys the lambda function it must delete and recreate the function, thus deleting any versions or aliases I made.
This means I really can't roll back to other releases. I basically have use git to actually revert the project, push to git, wait for the super-slow AWS Code Pipeline to flow through successfully, and then have it remake the function. To me that sounds like a pretty bad DR strategy, and I would think the right way to rollback should be simple and fast.
Unfortunately, it looks like the CloudFormation section of AWS doesn't offer any help here. When you drill into your stack on the first CloudFormation page it only shows you information about the latest formation that occurred. Dear engineers of AWS CloudFormation: if there was a page for each stack that showed a history of CloudFormation for this stack and an option to rollback to it, that would be really awesome. For now, though, there's not. There's just information about the latest formation that's been clouded. One initially promising option was "Rollback Triggers", but this is actually just something totally different that lets you send a SNS notification if your build doesn't pass.
When I try to change the CodePipeline stage for deploy from CREATE_CHANGE_SET to CREATE_UPDATE I then get this error when it tries to execute:
Action execution failed UpdateStack cannot be used with templates
containing Transforms. (Service: AmazonCloudFormation; Status Code:
400; Error Code: ValidationError; Request ID:
bea5f687-470b-11e8-a616-c791ebf3e8e1)
My template.yml looks like this by the way:
AWSTemplateFormatVersion: 2010-09-09
Transform:
- AWS::Serverless-2016-10-31
- AWS::CodeStar
Parameters:
ProjectId:
Type: String
Description: AWS CodeStar projectID used to associate new resources to team members
Resources:
HelloWorld:
Type: AWS::Serverless::Function
Properties:
Handler: index.handler
Runtime: nodejs8.10
Environment:
Variables:
NODE_ENV: staging
Role:
Fn::ImportValue:
!Join ['-', [!Ref 'ProjectId', !Ref 'AWS::Region', 'LambdaTrustRole']]
Events:
GetEvent:
Type: Api
Properties:
Path: /
Method: get
PostEvent:
Type: Api
Properties:
Path: /
Method: post
The only options in the CodePipeline "Deploy" action are these:
It would be really great if someone could help me to see how in AWS you can make Lambda functions with CodePipeline in a way that they are easy and fast to rollback. Thanks!