Proper way to deploy a node/express app to lamda function - amazon-web-services

I am looking to deploy a standard node / express app to a lamda function and use it as the back end of my code. I have the API Gateway set up, and everything seems to be working fine as I get a "hello world" response when I hit the API end point for the back end.
The issue is that I need to upload new iterations of the back end and I don't know how to push the code from my local repo or github or anywhere else onto the lambda server/function.
This page says that you should zip it and push it, but that isn't real descriptive. It almost looks like it would create a new lambda each time that you use it.
zip function.zip index.js
aws lambda create-function --function-name my-function \
--zip-file fileb://function.zip --handler index.handler --runtime nodejs12.x \
--role arn:aws:iam::123456789012:role/lambda-ex
Then, there's using same to build and deploy the node app. This seems like a lot of work and setup for a simple app like this. I can set up the front end to deploy to an S3 bucket each time I push to master. Can I do something like that with lambda?

I personally would use the SAM cli. This allows you do build, package and deploy your lambda locally with use of a Cloud Formation template - IMHO a better way to go for your application stack.
You define all of your resources -api gateway, resources, routes, methods, integrations and permissions in the template. Then use sam deploy to not only create your infrastructure, but deploy your application code when it changes.
Here is an example SAM Cloud Formation template that you place in the root of your repo (this assumes a lambda function that needs to access a dynamo db table).
AWSTemplateFormatVersion: 2010-09-09
Transform: AWS::Serverless-2016-10-31
Description: Your Stack Name
Resources:
DynamoTable:
Type: AWS::DynamoDB::Table
Properties:
TableName: my-table
KeySchema:
AttributeName: id
KeyType: HASH
BillingMode: PAY_PER_REQUEST
Lambda:
Type: AWS::Serverless::Function
Properties:
FunctionName: your-lambda-name
CodeUri: app/
Handler: app.handler
Runtime: nodejs14.x
Environment:
Variables:
DYNAMO_TABLE: !Ref DynamoTable
Policies:
- AWSLambdaExecute
- Version: 2012-10-17
Statement:
- Effect: Allow
Action:
- dynamodb:GetItem
- dynamodb:Query
- dynamodb:Scan
- dynamodb:PutItem
- dynamodb:UpdateItem
Resource:
- !Sub 'arn:aws:dynamodb:${AWS::Region}:${AWS::AccountId}:table/${DynamoTable}'
Events:
ApiEvent:
Type: Api
Properties:
Path: /path
Method: get
Now deploying your code (or updated stack as you add / change resources) is as simple as:
sam deploy --template-file template.yml --stack-name your-stack-name --resolve-s3 --region your-region
the sam deploy does a lot in one line:
I include the --template-file parameter - but it is really not necessary in this case because the file is called "template.yml" - which the deploy command will look for.
the "--resolve-s3" option will automatically create an s3 bucket to upload your lambda function code to versus you having to define a bucket (and create it) outside of the template. The alternative would be to specify a bucket "--s3-bucket". However you would have to create that BEFORE attempting to create the stack. This is fine - but you should take this into account when you go to delete your stack. The bucket will not be included in that process and you need to ensure the bucket is deleted in addition to the stack.
I typically add in a Makefile and do my build, prune, test, etc as part of my build and deploy.
i.e.
publish:
npm run build && \
npm test && \
npm prune --production && \
sam deploy --stack-name my-stack resolve-s3 --region us-east-1
then
make publish
More than what you asked for I realize - but perhaps someone else will find utility in this.

Instead of using the aws lambda create-function command, you can use aws lambda update-function-code to update an existing lambda:
aws lambda update-function-code \
--function-name <function-name> \
--zip-file fileb://function.zip

Related

AWS Cloudformation package wont update code

I have this issue:
I have deployed a function with CF that is build with the aws cloudformation package command. The first deployment works perfect. But when it try to update the code nothing happens. This is what I do:
Save changes to my local code and and CF and run aws cloudformation package (aws cloudformation package --template-file mycf.yml --s3-bucket mybucket --output-template-file packaged-mycf.yml --profile myprofile)
I see in the packaged file that where is a new path under CodeUri (CodeUri: s3://mybucket/319bd03cb3cc8d50ceb80e52bf51c53c)
I deploy the update (I do it in the console) I see under update that where is updates to the function
The CF events says update to function is complete
I go to code, the same old code nothing has changed
Have anyone else experienced the same?
I have tried to rename the CF file, script file and the packaged CF file. But it still get the same result.
Does anyone here have an ide on what i may try?
This is how the unpackaged function part of the CF looks like:
CRFunction:
Type: AWS::Serverless::Function
Properties:
Description: Convert IAM Policy into SCP
Handler: scpfunction.lambda_handler
Runtime: python3.9
Timeout: 30
MemorySize: 128
FunctionName: !Sub SCP-Function-${SCPName}
CodeUri: src/
Environment:
Variables:
SCPName: !Ref SCPName
Policy: !Ref IAMPolicyToConvertToSCP
OUs: !Ref OUs
Description: !Ref Description
Policies:
Statement:
- Effect: Allow
Action:
- organizations:CreatePolicy
- organizations:AttachPolicy
- organizations:List*
- iam:get*
Resource: '*'
Have anyone else experienced the same?
That's how it works by design. Changes to the source code are not detected. You have to change either Key or Version in your CloudFormation template to deploy the new code.
Update
s3://mybucket/319bd03cb3cc8d50ceb80e52bf51c53c is only your function code, not the template. To update your function using s3://mybucket/319bd03cb3cc8d50ceb80e52bf51c53c you have to use AWS Lambda console's Amazon S3 location:
I got this solved.
It was not an issue related to AWS.
After restarting VisualStudioCode I got the deployment to work. So it had to do with some local caching or something.
Really weird never experienced anything like that before.

Dynamically create resource names in AWS SAM using parameters

I want to dynamically create names for my resources in my Cloud Formation stack when using AWS SAM if this is possible.
E.g. when I package or deploy I want to be able to add soemthing to the command line like this:
sam package --s3-bucket..... --parameters stage=prod
When in my template.yml file somehow to do something like this:
Resources:
OrdersApi:
Type: AWS::Serverless::Function
Properties:
FunctionName: orders-api-${stage}
CodeUri: ./src/api/services/orders/
...
Note for the OrdersApi property of FunctionName I want to dynamically set it to orders-api-prod which is the value I attempted to pass in on the CLI.
I can do this quite easily using the Serverless Framework but I can't quite work out how to do it with SAM.
You can use functions like Sub to construct resource names in CloudFormation. Something along the lines:
Parameters:
stage:
Type: String
Default: dev
AllowedValues:
- dev
- prod
Resources:
OrdersApi:
Type: AWS::Serverless::Function
Properties:
FunctionName: !Sub 'orders-api-${stage}'
The answer posted by lexicore is correct and you can form values in certain parts of the template.yaml file using the !Sub function e.g.
FunctionName: !Sub 'orders-api-${stage}'
The missing part of why this wouldn't work is that you need to pass the parameters through to the sam deploy command in a specific format. From reading the AWS docs, sam deployis shorthand for aws cloudformation deploy.... That command allows you to pass parameters using the following syntax:
aws cloudformation deploy .... --parameter-overrides stage=dev
This syntax can also be used with the sam deploy command e.g.
sam deploy --template-file packaged.yml ..... --parameter-overrides stage=dev
Note that in this example stage=dev applies to the Parameters section of the template.yaml file e.g.
Parameters:
stage:
Type: String
AllowedValues:
- dev
- stage
- prod
This approach allowed me to pass in parameters and dynamically change values as the cloud formation stack was deployed.

Using AWS::CodeBuild::Project Environment variable in cloudformation template of repository

I want to create a continous delivery pipeline for a Lambda function.
As shown in this docs, the custom environment variables of AWS::CodeBuild::Project can be used in buildspec.yaml like:
aws cloudformation package --template-file template.yaml --s3-bucket $MYEVVARKEY --output-template-file outputtemplate.yaml
Wanted to use those CodeBuild Project environment variables in the SAM template of the repository also. As shown below, I tried with dollar signs, but it did not get it as a key but as a text as it is:
# template.yaml
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Resources:
TimeFunction:
Type: AWS::Serverless::Function
Properties:
FunctionName: $MY_FN_NAME_ENV_VAR
Role: $MY_ROLE_ARN_ENV_VAR
Handler: index.handler
Runtime: nodejs8.10
CodeUri: ./
So, is it possible to utilize CodeBuild Project environment variables in SAM template, if so what's the notation required to achieve that?
CloudFormation can't refer to environment variables, doesn't matter SAM or plain. What you can do is to pass environment variables as parameters via shell in CodeBuild buildspec.yaml file (--parameters ParameterKey=name,ParameterValue=${MY_ENV_VAR}).
Remember to add corresponding parameter to your Parameters section.
If you use aws cloudformation deploy then you should use --parameter-overrides, which is a little simpler form:
--parameter-overrides \
YourParam=${YOUR_ENV_VAR} \
Foo=Bar \

Issues Creating Environments For AWS Lambda Service In CodeStar And CodePipeline

I used AWS CodeStar to create a new application with the "Express.js Aws Lambda Webservice" CodeStar template. This was great because it set me up with a simple CI/CD pipeline using AWS CodePipeline. By default the pipeline has 3 steps for grabbing the source code from a git repo, running the build step, and then deploying to "dev" environment.
My issue is that I can't set it up so that my pipeline has multiple environments: dev, staging, and prod.
My current deploy step has 2 actions: GenerateChangeSet and ExecuteChangeSet. Here are the configurations for the actions in original dev environment build step which work great:
I've created a new deploy stage at the end of my pipeline to deploy to staging, but honestly I'm not sure how to change the configurations. I'm thinking ultimately I want to be able to go into the AWS Lambda section of the AWS console and see three independent lambda functions: binance-bot-dev, binance-bot-staging, binance-bot-prod. Then each of these I could set as cloudwatch scheduled events or expose with their own api gateway url.
This is the configuration that I tried to use for a new deployment stage:
I'm really not sure if this configuration is correct and what exactly I should change in order to deploy in the way I want.
For example, should I be changing "Stack name", or should I keep that as "awscodestar-binance-bot-lambda" or change it for each environment as I am here?
Also, I'm pointing to a different template.yml file in the project. The original template.yml looks like this:
AWSTemplateFormatVersion: 2010-09-09
Transform:
- AWS::Serverless-2016-10-31
- AWS::CodeStar
Parameters:
ProjectId:
Type: String
Description: AWS CodeStar projectID used to associate new resources to team members
Resources:
Dev:
Type: AWS::Serverless::Function
Properties:
Handler: index.handler
Runtime: nodejs4.3
Environment:
Variables:
NODE_ENV: dev
Role:
Fn::ImportValue:
!Join ['-', [!Ref 'ProjectId', !Ref 'AWS::Region', 'LambdaTrustRole']]
Events:
GetEvent:
Type: Api
Properties:
Path: /
Method: get
PostEvent:
Type: Api
Properties:
Path: /
Method: post
For template.staging.yml I use the exact same config except I changed "Dev:" to "Staging:" under "Resources", and I also changed the value of the NODE_ENV environment variable. So, I'm basically wondering is this the correct configuration for what I'm trying to achieve?
Assuming that everything in the configuration is correct, I then need to troubleshoot this error. With everything set as described above I can run my pipeline, but when it gets to my staging build step the GenerateChage_Staging action fails with this error message:
Action execution failed User:
arn:aws:sts::954459734159:assumed-role/CodeStarWorker-binance-bot-CodePipeline/1524253307698
is not authorized to perform: cloudformation:DescribeStacks on
resource:
arn:aws:cloudformation:us-east-1:954459734159:stack/awscodestar-binance-bot-lambda-staging/*
(Service: AmazonCloudFormation; Status Code: 403; Error Code:
AccessDenied; Request ID: dd801664-44d2-11e8-a2de-8fa6c42cbf86)
It seem to me from this error message that I need to add the "cloudformation:DescribeStacks" for my "CodeStarWorker-binance-bot-CodePipeline" so I go to IAM -> Roles and click on the CodeStarWorker-binance-bot-CodePipeline role. However, when I click on "CodeStarWorker-binance-bot-CodePipeline" and drill into the policy information for CloudFormation it looks like this role already has permissions for "DescribeStacks"!
If anyone could point out what I'm doing wrong or offer any guidance on understanding and thinking about how to do multiple environments with AWS CodePipeline that would be great. thanks!
UPDATE:
I changed the "Stack name" in my Deploy_To_Staging pipeline stage back to "awscodestar-binance-bot-lambda". However, I then get this error form the GenerateChange_Staging action:
Action execution failed Invalid TemplatePath:
binance-bot-BuildArtifact::template-export.staging.yml. Artifact
binance-bot-BuildArtifact doesn't exist
UPDATE 2:
In the root of my project I have the buildspec.yml file that was generated by CodeStar. It looks like this:
version: 0.2
phases:
install:
commands:
# Install dependencies needed for running tests
- npm install
# Upgrade AWS CLI to the latest version
- pip install --upgrade awscli
pre_build:
commands:
# Discover and run unit tests in the 'tests' directory
- npm test
build:
commands:
# Use AWS SAM to package the application using AWS CloudFormation
- aws cloudformation package --template template.yml --s3-bucket $S3_BUCKET --output-template template-export.yml
- aws cloudformation package --template template.staging.yml --s3-bucket $S3_BUCKET --output-template template-export.staging.yml
- aws cloudformation package --template template.prod.yml --s3-bucket $S3_BUCKET --output-template template-export.prod.yml
artifacts:
type: zip
files:
- template-export.yml
I then added this to the CloudFormation section:
Then I add this to the "build: -> commands:" section:
- aws cloudformation package --template template.staging.yml --s3-bucket $S3_BUCKET --output-template template-export.staging.yml
- aws cloudformation package --template template.prod.yml --s3-bucket $S3_BUCKET --output-template template-export.prod.yml
And I added this to the "files:"
template-export.staging.yml
template-export.prod.yml
HOWEVER, I am still getting an error that "binance-bot-BuildArtifact does not exist".
Here is the full error after making the buildspec.yml change:
Action execution failed Invalid TemplatePath:
binance-bot-BuildArtifact::template-export.staging.yml. Artifact
binance-bot-BuildArtifact doesn't exist
It seems very strange to me that I can access "binance-bot-BuildArtifact" in one stage of the pipeline but not another. Could it be that the build artifact is only available to the one pipeline stage directly after the build stage? Can someone please help me to be able to access this "binance-bot-BuildArtifact"? Thanks!
For example, should I be changing "Stack name", or should I keep that as "awscodestar-binance-bot-lambda" or change it for each environment as I am here?
You should use a unique stack name for each environment. If you didn't, you would be replacing your 'dev' environment with your 'staging' environment, and so forth.
So, I'm basically wondering is this the correct configuration for what I'm trying to achieve?
I don't think so. You should use the exact same template for each environment. In order to change the environment name for each of your deploys, you can use the 'Parameter Overrides' field to choose the correct value for your 'Environment' parameter.
it looks like this role already has permissions for "DescribeStacks"!
Could the issue here be that your IAM role only has DescribeStacks permission for the dev stack? It looks like it does not have permission to describe the staging stack. Maybe you can add a 'wildcard'/asterisk to the policy so that it matches all of your stack names?
Could it be that the build artifact is only available to the one pipeline stage directly after the build stage?
No, that has not been my experience with CodePipeline. Unfortunately I don't know why it's telling you that your artifact can't be found.
robrtsql has already provided some good advice in terms of using the same template in both stages.
You might find this walkthrough useful.
Basically, it describes adding a Cloudformation "template configuration" which allows you to specify parameters to the Cloudformation stack.
This will allow you to deploy the same template in both your dev and prod environments, but also allow you to tell the difference between a dev deployment and a prod deployment, by choosing a different template configuration in each stage.

How to automatically deploy api gateway in AWS codepipeline

Currently I am able to deploy a lambda by pushing to github. I also automatically deploy a lambda but only because the api gateway is an event in the lambda yaml file
AWSTemplateFormatVersion: '2010-09-09'
Transform: 'AWS::Serverless-2016-10-31'
Description: Identifies paragraphs in documents and links to the law
Resources:
LambdaParagraphLinker:
Type: 'AWS::Serverless::Function'
Properties:
Handler: LambdaParagraphLinker.lambda_handler
Runtime: python3.6
CodeUri: ./
Description: Identifies paragraphs in documents and links to the
law
MemorySize: 512
Timeout: 10
Events:
Api:
Type: Api
Properties:
Path: /LambdaParagraphLinker
Method: ANY
How can I deploy an api gateway using a swagger file ?
Hands down the best way to do this in codepipeline is by using https://serverless.com/ framework. This replaces every super complicated hack-job and workaround I've previously used. Way less complicated IMO.
Create a codepipeline, link it to src & a codebuild project, set a few permissions, done.
//serverless.yml
service: my-api
provider:
name: aws
runtime: python2.7
functions:
hello:
handler: handler.hello
events:
- http:
path: api/v1/message
method: post
//buildspec.yml
version: 0.2
phases:
install:
commands:
#BUILD
- sudo apt-get update -y
build:
commands:
- echo $environment
- serverless package --stage $environment --region us-east-1
- serverless deploy --stage $environment --region us-east-1
Or torture yourself by doing one of the options below...
You can do this in cloudformation from within code pipeline. Export the swagger spec from within the gatewayapi console and place in the cloudformation template.
AWSTemplateFormatVersion: '2010-09-09'
Resources:
PlayersAPI:
Type: AWS::ApiGateway::RestApi
Properties:
Name: MyApi
Description: API Description
Body:
"SWAGGER HERE"
Hooking this up to lambda is a little bit cumbersome but I can describe the steps. First create a codepipeline project with source, build, and deploy steps.
src should be standard from github or codecommit
build should output a zip file and use buildspec.yml Something like this...
//buildspec.yml
version: 0.1
phases:
install:
commands:
#BUILD
- zip -r lambda.zip . -x *.git*
artifacts:
files:
- '**/*.zip'
- '**/*.yml'
discard-paths: no
Have the build step export an artifact MyAppBuild (or whatever you want to call it)
The final pipeline step is creating the lambda function in this repo as a standalone function through the console(its reusable):
https://github.com/tkntobfrk/codepipeline-lambda-s3
This lambda function downloads the pipeline artifact/zipped lambda function and updates it using boto.
After these steps you could add another step as a cloudformation deploy step. Connect it to the lambda function that you've just deployed.
If you are dealing with multiple environments you could create lambda functions and gatewayapi cloudformation template for each environment then run them in sequence.
stage 1: src
stage 2: build
stage 3: deploy lambda test, deploy gateway api cloudformation test
stage 4: validate test
stage 5: deploy lambda prod, deploy gateway api cloudformation prod
Using straight AWS serverless like this works too. However, you need to use a standard artifact location for the uri's. The DefinitionUri: for the API can be the exported swagger from the gatewayapi console.
//cloudformation.yml
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Resources:
MySimpleFunction:
Type: AWS::Serverless::Function
Properties:
Handler: app.lambda_handler
Runtime: python2.7
CodeUri: s3://somebucket/somezip.zip
MyAPI:
Type: AWS::Serverless::Api
Properties:
StageName: prod
DefinitionUri: s3://somebucket/somezip.zip
AWS::Serverless::Api
https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#awsserverlessapi
You can find Swagger docs all over the place, and docs on API Gateway extensions are in the developer guide. I would start by going into the API Gateway console and look at the API that Lambda creates for you. You can go to the 'Stages' page and for any stage, you can Export the API as Swagger.