How do I specify the CodeUri in my pipeline? - amazon-web-services

I'm doing a serverless app in lambda using CloudFormation.
In my CodeBuild project, I set it to zip up the output and place it in "myBucket\AWSServerless1.zip" and it does correctly.
Now I'm working on my CodePipeline, I reference the original CodeBuild project. However, now instead, it puts it in codepipeline-us-west-#####. That's fine. The issue is that the .zip file has a RANDOM name. CodePipeline ignores the name I gave it in the CodeBuild project.
In the serverless.template, I have to specify the CodeUri (which seems to be the CodeBuild project output for some odd reason). If I reference the AWSServerless1.zip, it works fine (but its not building to there, so its stale code)... but...
Since CodePipeline calling CodeBuild gives it a random name, how am I supposed to reference the ACTUAL BuildArtifact in the serverless.template?

I know this is very weird, I was stuck with this behavior of CodePipeline and then had to rewrite the buildspec to make CodePipeline work. CodePipeline makes it's own zip file even if you create your own zip through CodeBuild as well and that too with a unique name.
But there is one way out, Codepipeline will create one zip file but it will unzip it while giving the artifact to CodeDeploy. So you need not worry about its name. CodeDeploy will get the unzipped version of your code. CodePipeline keeps track of the name and it will always point to the newest one.
Suppose :
CodePipeline creates artifact : some-random-name.zip
some-random-name
|- deploy/lib/lambda-code
|- some-file.yaml
Whenever CodePipeline gives artifact to CodeDeploy, it will unzip it so you can anytime refer the code under some-random-name.zip
So in your case when you give CodeUri in the SAM template just give the folder name which is deploy where your lambda code is present.
Resources:
Hello:
Type: 'AWS::Serverless::Function'
Properties:
Handler: example.MyHandler
Runtime: java8
CodeUri: deploy
Description: ''
MemorySize: 512
Timeout: 15
Hope this helps.

I was facing the same error and I managed to work around it by doing the following:
1- On the build specification (buildspec.yml) add a sam package command (this generates a package.yml that will be used by cloudformation to deploy the lambda).
build:
commands:
- sam package
--template-file ../template.yaml
--output-template-file ../package.yml
--s3-bucket onnera-ci-cd-bucketcode here
2- Add the package.yml to output artifacts
artifacts:
files:
- DeviceProvisioning/package.yml
3- On the template.yaml that will be deployed reference directly the CodeUri (internally this will be resolved to the bucket with the output artificats from codebuild).
Resources:
DeviceProvisioningFunction:
Type: AWS::Serverless::Function
Properties:
CodeUri: DeviceProvisioningFunction/target/DeviceProvisioningFunction-1.0.jar
4- On the pipeline make the output of the build phase avaialble on deployment phase:
const buildOutput = [new codepipeline.Artifact()];
const buildAction = new codepipeline_actions.CodeBuildAction({
actionName: 'CodeBuild',
project: deviceProvisioning,
input: sourceOutput,
outputs: buildOutput,
});
5- Use the build output to specify the templatePath on the deploy action:
const deployAction = new codepipeline_actions.CloudFormationCreateUpdateStackAction({
extraInputs: buildAction.actionProperties.outputs,
actionName: "UpdateLambda",
stackName: "DeviceProvisioningStack",
adminPermissions: true,
templatePath: buildOutput[0].atPath("package.yml"),
cfnCapabilities: [CfnCapabilities.AUTO_EXPAND, CfnCapabilities.NAMED_IAM]
});
Make sure that the output artifacts from the build phase are available on the deploy phase.

Related

How to manage different Serverless (with AWS Lambda) environments (ie "dev" and "prod")

I want to create a separate 'dev' AWS Lambda with my Serverless service.
I have deployed my production, 'prod', environment and tried to then deploy a development, 'dev', environment so that I can trial features without affecting customer experience.
In order to deploy the 'dev' environment I have:
Created a new serverless-dev.yml file
Updated the stage and profile fields in my .yml file:
provider:
name: aws
runtime: nodejs14.x
stage: dev
region: eu-west-2
profile: dev
memorySize: 128
timeout: 30
Also update the resources.Resources.<Logical Id>.Properties.RoleName value, as if I try to use the same role as my 'prod' Lambda, I get this message: clearbit-lambda-role-prod already exists in stack
resources:
Resources:
<My Logical ID>:
Type: AWS::IAM::Role
Properties:
Path: /my/cust/path/
RoleName: clearbit-lambda-role-dev # Change this name
Run command: sls deploy -c serverless-dev.yml
Is this the conventional method to achieve this? I can't find anything in the documentation.
Serverless Framework has support for stages out of the box. You don't need a separate configuration, you can just specify --stage <name-of-stage> when running .e.g sls deploy and it will automatically use that stage. All resources created by the Framework under the hood are including stage in it's names or identifiers. If you are defining some extra resources in resources section, you need to change them, or make sure they include stage in their names. You can get the current stage in configuration with ${sls:stage} and use that to construct names that are e.g. prefixed with stage.

Issues Creating Environments For AWS Lambda Service In CodeStar And CodePipeline

I used AWS CodeStar to create a new application with the "Express.js Aws Lambda Webservice" CodeStar template. This was great because it set me up with a simple CI/CD pipeline using AWS CodePipeline. By default the pipeline has 3 steps for grabbing the source code from a git repo, running the build step, and then deploying to "dev" environment.
My issue is that I can't set it up so that my pipeline has multiple environments: dev, staging, and prod.
My current deploy step has 2 actions: GenerateChangeSet and ExecuteChangeSet. Here are the configurations for the actions in original dev environment build step which work great:
I've created a new deploy stage at the end of my pipeline to deploy to staging, but honestly I'm not sure how to change the configurations. I'm thinking ultimately I want to be able to go into the AWS Lambda section of the AWS console and see three independent lambda functions: binance-bot-dev, binance-bot-staging, binance-bot-prod. Then each of these I could set as cloudwatch scheduled events or expose with their own api gateway url.
This is the configuration that I tried to use for a new deployment stage:
I'm really not sure if this configuration is correct and what exactly I should change in order to deploy in the way I want.
For example, should I be changing "Stack name", or should I keep that as "awscodestar-binance-bot-lambda" or change it for each environment as I am here?
Also, I'm pointing to a different template.yml file in the project. The original template.yml looks like this:
AWSTemplateFormatVersion: 2010-09-09
Transform:
- AWS::Serverless-2016-10-31
- AWS::CodeStar
Parameters:
ProjectId:
Type: String
Description: AWS CodeStar projectID used to associate new resources to team members
Resources:
Dev:
Type: AWS::Serverless::Function
Properties:
Handler: index.handler
Runtime: nodejs4.3
Environment:
Variables:
NODE_ENV: dev
Role:
Fn::ImportValue:
!Join ['-', [!Ref 'ProjectId', !Ref 'AWS::Region', 'LambdaTrustRole']]
Events:
GetEvent:
Type: Api
Properties:
Path: /
Method: get
PostEvent:
Type: Api
Properties:
Path: /
Method: post
For template.staging.yml I use the exact same config except I changed "Dev:" to "Staging:" under "Resources", and I also changed the value of the NODE_ENV environment variable. So, I'm basically wondering is this the correct configuration for what I'm trying to achieve?
Assuming that everything in the configuration is correct, I then need to troubleshoot this error. With everything set as described above I can run my pipeline, but when it gets to my staging build step the GenerateChage_Staging action fails with this error message:
Action execution failed User:
arn:aws:sts::954459734159:assumed-role/CodeStarWorker-binance-bot-CodePipeline/1524253307698
is not authorized to perform: cloudformation:DescribeStacks on
resource:
arn:aws:cloudformation:us-east-1:954459734159:stack/awscodestar-binance-bot-lambda-staging/*
(Service: AmazonCloudFormation; Status Code: 403; Error Code:
AccessDenied; Request ID: dd801664-44d2-11e8-a2de-8fa6c42cbf86)
It seem to me from this error message that I need to add the "cloudformation:DescribeStacks" for my "CodeStarWorker-binance-bot-CodePipeline" so I go to IAM -> Roles and click on the CodeStarWorker-binance-bot-CodePipeline role. However, when I click on "CodeStarWorker-binance-bot-CodePipeline" and drill into the policy information for CloudFormation it looks like this role already has permissions for "DescribeStacks"!
If anyone could point out what I'm doing wrong or offer any guidance on understanding and thinking about how to do multiple environments with AWS CodePipeline that would be great. thanks!
UPDATE:
I changed the "Stack name" in my Deploy_To_Staging pipeline stage back to "awscodestar-binance-bot-lambda". However, I then get this error form the GenerateChange_Staging action:
Action execution failed Invalid TemplatePath:
binance-bot-BuildArtifact::template-export.staging.yml. Artifact
binance-bot-BuildArtifact doesn't exist
UPDATE 2:
In the root of my project I have the buildspec.yml file that was generated by CodeStar. It looks like this:
version: 0.2
phases:
install:
commands:
# Install dependencies needed for running tests
- npm install
# Upgrade AWS CLI to the latest version
- pip install --upgrade awscli
pre_build:
commands:
# Discover and run unit tests in the 'tests' directory
- npm test
build:
commands:
# Use AWS SAM to package the application using AWS CloudFormation
- aws cloudformation package --template template.yml --s3-bucket $S3_BUCKET --output-template template-export.yml
- aws cloudformation package --template template.staging.yml --s3-bucket $S3_BUCKET --output-template template-export.staging.yml
- aws cloudformation package --template template.prod.yml --s3-bucket $S3_BUCKET --output-template template-export.prod.yml
artifacts:
type: zip
files:
- template-export.yml
I then added this to the CloudFormation section:
Then I add this to the "build: -> commands:" section:
- aws cloudformation package --template template.staging.yml --s3-bucket $S3_BUCKET --output-template template-export.staging.yml
- aws cloudformation package --template template.prod.yml --s3-bucket $S3_BUCKET --output-template template-export.prod.yml
And I added this to the "files:"
template-export.staging.yml
template-export.prod.yml
HOWEVER, I am still getting an error that "binance-bot-BuildArtifact does not exist".
Here is the full error after making the buildspec.yml change:
Action execution failed Invalid TemplatePath:
binance-bot-BuildArtifact::template-export.staging.yml. Artifact
binance-bot-BuildArtifact doesn't exist
It seems very strange to me that I can access "binance-bot-BuildArtifact" in one stage of the pipeline but not another. Could it be that the build artifact is only available to the one pipeline stage directly after the build stage? Can someone please help me to be able to access this "binance-bot-BuildArtifact"? Thanks!
For example, should I be changing "Stack name", or should I keep that as "awscodestar-binance-bot-lambda" or change it for each environment as I am here?
You should use a unique stack name for each environment. If you didn't, you would be replacing your 'dev' environment with your 'staging' environment, and so forth.
So, I'm basically wondering is this the correct configuration for what I'm trying to achieve?
I don't think so. You should use the exact same template for each environment. In order to change the environment name for each of your deploys, you can use the 'Parameter Overrides' field to choose the correct value for your 'Environment' parameter.
it looks like this role already has permissions for "DescribeStacks"!
Could the issue here be that your IAM role only has DescribeStacks permission for the dev stack? It looks like it does not have permission to describe the staging stack. Maybe you can add a 'wildcard'/asterisk to the policy so that it matches all of your stack names?
Could it be that the build artifact is only available to the one pipeline stage directly after the build stage?
No, that has not been my experience with CodePipeline. Unfortunately I don't know why it's telling you that your artifact can't be found.
robrtsql has already provided some good advice in terms of using the same template in both stages.
You might find this walkthrough useful.
Basically, it describes adding a Cloudformation "template configuration" which allows you to specify parameters to the Cloudformation stack.
This will allow you to deploy the same template in both your dev and prod environments, but also allow you to tell the difference between a dev deployment and a prod deployment, by choosing a different template configuration in each stage.

How do I get Cloud Formation to create a changeset to update my Lambda functions?

I have a Lambda function which I've verified to work correctly. I'm able to update the function by hand on the command line using "update-function-code" but I've been trying to get it working with Code Pipeline and Cloud Formation.
Here are the steps I have so far:
Source - fetch the code from github. This works correctly.
Build - test the code in Solano (3rd party CI). This works too and on the last stage it zips up the repo and uploads it to my S3 bucket.
Deploy - This is the "deploy" action category with the action mode "create or replace a change set". This doesn't work if the Lambda function already exists.
Beta - Execute the changeset. This works if the change set was generated correctly.
My samTemplate.yml looks like this:
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: My Lambda function
Resources:
LambdaFunction:
Type: AWS::Serverless::Function
Properties:
FunctionName: MyLambdaExecute
Description: My Lambda function
Handler: myhandler.handler
Runtime: nodejs6.10
CodeUri: s3://mybucket/mydirectory/mylambdacode.zip
AutoPublishAlias: Staging
Timeout: 30
DeploymentPreference:
Type: AllAtOnce
If the lambda function with the name "MyLambdaExecute" doesn't exist and I push up code to github, it works perfectly. But if I modify some code and push again it runs the first two steps, but then generates an empty change set with the status:
FAILED - No updates are to be performed.
I'm not sure what I have to do to get it to publish a new version. How do I get it to realize it needs to create a new changeset?
I believe you are receiving the "No updates" message because technically nothing is changing in your CloudFormation template. When attempting to build the changeset, the contents of the S3 file are not examined. It just sees that none of the CloudFormation resource properties have changed since the previous deployment.
Instead, you may use a local relative path for the CodeUri, and aws cloudformation package can upload the file to a unique S3 path for you. This ensures that the template changes each time and the new Lambda code is recognized/deployed. For example:
aws cloudformation package --template-file samTemplate.yml --output-template-file sam-output-dev.yml --s3-bucket "$CodePipelineS3Bucket" --s3-prefix "$CloudFormationPackageS3Prefix"
This command can be put into the build step before your create/execute changeset steps.
To see an demonstration of this entire flow in practice, you can look at this repository, although I will warn that it's a bit outdated thanks to the new features released at the end of 2017. (For example, I was publishing Lambda aliases manually via extra steps because it was written pre-AutoPublishAlias.)

CodeDeploy can't find my AWS Lambda Function

I have an Issue with CodeDeploy and AWS Lambda when they work inside AWS CodePipeline. This is my setup:
Source GitHub
AWS CodeBuild
AWS CodeDeploy
The Issue
Step 1. and 2. work without a problem, but when it comes to CodeDeploy I get the following error:
Action execution failed BundleType must be either YAML or JSON
If I unzip the Artifact generated by CodeBuild all the files are in place.
If I try to manually deploy to AWS Lambda from CodeDeploy I then get a different message...
Deployment Failed The deployment failed because either the target
Lambda function FUNCTION_NAME does not exist or the specified function
version or alias cannot be found
This is very confusion as to which Error message is valid, or if they are the same but have a different Error message.
The Setup
The ARN of the function is:
arn:aws:lambda:us-east-1:239748505547:function:email_submition
The ARN for the Alias is:
arn:aws:lambda:us-east-1:239748505547:function:email_submition:default
And my appspec.yml file has the following content
version: 0.0
Resources:
- email_submition:
Type: AWS::Lambda::Function
Properties:
Name: "email_submition"
Alias: "default"
CurrentVersion: "1"
TargetVersion: "2"
And the folder structure of the project is:
.gitignore
appspec.yml
buildspec.yml
index.js
README.md
Question
What am I missing in this configuration?
So really this should be a comment not an answer. I do not have 50 rep yet so it's here.
I am having the same issues as you. I'm not sure if you found a solution or not. I was able to successfully execute a deployment with the following appspec.yml:
version: 0.0
Resources:
- mylambdafunction:
Type: AWS::Lambda::Function
Properties:
Name: "mylambdafunction"
Alias: "staging"
CurrentVersion: "2"
TargetVersion: "3"
Both the current version and target version had to exist before CodeDeploy would work. Of course I've tested this by doing a manual deployment.
I think what is needed here is something that actually updates the code and creates a new version. Which is what I would have thought CodeDeploy would do.
Edit: Further research has yielded information about CodePipeline I hadn't realized.
Per here it looks like to run through the Pipeline you need your buildspec, appspec, and a cft. The reason the pipeline fails is because you need to include a CloudFormation Template for the lambda function, this is what deploys the actual code. The appspec.yml is there to migrate traffic from the old version to the new version but the cft is what does the deployment of new code.
Edit2: This example app got me squared away.
Use CodeBuild to build your app but also to generate your CFT for doing actual deployment. This means you build your CFT with the lambda resource.
This removes appspec completely from the resources and instead you use a CFT to define the Lambda function. Here is a link to the SAM docs.
I can not help you with the CodeBuild part as I use a 3rd party CI solution but maybe I can help with the rest.
I think there is a mistake in the AWS documentation as I've never been able to get this to work either. They say to call "aws deploy push" on the command line and give it your appspec.yml file instead of a zip for Lambda, but no matter what you do, you will always get the error:
Action execution failed BundleType must be either YAML or JSON
I think this is because push automatically calls "register-application-revision" after it uploads. If you split this into separate parts, this will work.
Your appspec.yml should look like the
version: 0.0
Resources:
- YourFunctionName:
Type: "AWS::Lambda::Function"
Properties:
Name: "YourFunctionName"
Alias: "YourFunctionNameAlias"
CurrentVersion: "CurrentAliasVersionGoesHere"
TargetVersion: "NewlyPublishedVersionGoesHere"
The version you use should be the version the current alias is attached to. The target version should be the new version you just published (see below) This part still confusing me a bit. I don't understand why it can't figure out what the current version the alias is pointing to by itself.
Also, note that you can always just upload new code for your Lambda code with update-function-code and it will overwrite the latest version. Or you can publish which will create a new version and always just call the latest version. CodeDeploy is only necessary if you want to do some fancy gradually deployment or have different versions for test and live code.
I'd try the following:
Publish your lambda function:
aws lambda update-function-code --function-name YourFunction --zip-file fileb://~/your-code.zip --publish
Take note of the version number it created
Upload your appspec.yml file to S3
aws s3 cp appspec.yml s3://your-deploy-bucket/your-deploy-dir/appspec.yml
Register your application revision:
aws deploy register-application-revision --application-name YourApplcationName --s3-location bucket=your-deploy-bucket,key=your-deploy-dir/appspec.yml,bundleType=YAML
From the CLI this won't appear to do anything, but it did.
Get the application revision to make sure it worked
aws deploy get-application-revision --application-name YourApplcationName --s3-location bucket=your-deploy-bucket,key=your-deploy-dir/appspec.yml,bundleType=YAML
Create a deployment to deploy your code
aws deploy create-deployment --s3-location bucket=your-deploy-bucket,key=your-deploy-dir/appspec.yml,bundleType=YAML

How to automatically deploy api gateway in AWS codepipeline

Currently I am able to deploy a lambda by pushing to github. I also automatically deploy a lambda but only because the api gateway is an event in the lambda yaml file
AWSTemplateFormatVersion: '2010-09-09'
Transform: 'AWS::Serverless-2016-10-31'
Description: Identifies paragraphs in documents and links to the law
Resources:
LambdaParagraphLinker:
Type: 'AWS::Serverless::Function'
Properties:
Handler: LambdaParagraphLinker.lambda_handler
Runtime: python3.6
CodeUri: ./
Description: Identifies paragraphs in documents and links to the
law
MemorySize: 512
Timeout: 10
Events:
Api:
Type: Api
Properties:
Path: /LambdaParagraphLinker
Method: ANY
How can I deploy an api gateway using a swagger file ?
Hands down the best way to do this in codepipeline is by using https://serverless.com/ framework. This replaces every super complicated hack-job and workaround I've previously used. Way less complicated IMO.
Create a codepipeline, link it to src & a codebuild project, set a few permissions, done.
//serverless.yml
service: my-api
provider:
name: aws
runtime: python2.7
functions:
hello:
handler: handler.hello
events:
- http:
path: api/v1/message
method: post
//buildspec.yml
version: 0.2
phases:
install:
commands:
#BUILD
- sudo apt-get update -y
build:
commands:
- echo $environment
- serverless package --stage $environment --region us-east-1
- serverless deploy --stage $environment --region us-east-1
Or torture yourself by doing one of the options below...
You can do this in cloudformation from within code pipeline. Export the swagger spec from within the gatewayapi console and place in the cloudformation template.
AWSTemplateFormatVersion: '2010-09-09'
Resources:
PlayersAPI:
Type: AWS::ApiGateway::RestApi
Properties:
Name: MyApi
Description: API Description
Body:
"SWAGGER HERE"
Hooking this up to lambda is a little bit cumbersome but I can describe the steps. First create a codepipeline project with source, build, and deploy steps.
src should be standard from github or codecommit
build should output a zip file and use buildspec.yml Something like this...
//buildspec.yml
version: 0.1
phases:
install:
commands:
#BUILD
- zip -r lambda.zip . -x *.git*
artifacts:
files:
- '**/*.zip'
- '**/*.yml'
discard-paths: no
Have the build step export an artifact MyAppBuild (or whatever you want to call it)
The final pipeline step is creating the lambda function in this repo as a standalone function through the console(its reusable):
https://github.com/tkntobfrk/codepipeline-lambda-s3
This lambda function downloads the pipeline artifact/zipped lambda function and updates it using boto.
After these steps you could add another step as a cloudformation deploy step. Connect it to the lambda function that you've just deployed.
If you are dealing with multiple environments you could create lambda functions and gatewayapi cloudformation template for each environment then run them in sequence.
stage 1: src
stage 2: build
stage 3: deploy lambda test, deploy gateway api cloudformation test
stage 4: validate test
stage 5: deploy lambda prod, deploy gateway api cloudformation prod
Using straight AWS serverless like this works too. However, you need to use a standard artifact location for the uri's. The DefinitionUri: for the API can be the exported swagger from the gatewayapi console.
//cloudformation.yml
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Resources:
MySimpleFunction:
Type: AWS::Serverless::Function
Properties:
Handler: app.lambda_handler
Runtime: python2.7
CodeUri: s3://somebucket/somezip.zip
MyAPI:
Type: AWS::Serverless::Api
Properties:
StageName: prod
DefinitionUri: s3://somebucket/somezip.zip
AWS::Serverless::Api
https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#awsserverlessapi
You can find Swagger docs all over the place, and docs on API Gateway extensions are in the developer guide. I would start by going into the API Gateway console and look at the API that Lambda creates for you. You can go to the 'Stages' page and for any stage, you can Export the API as Swagger.