I have a pipeline in AWS with Codestar, CodeBuild, CloudFormation, etc.
I am trying to figure out how to get information from the CloudFormation step returned to the CodeBuild step. Let me break it down:
I have a buildspec.yml for CodeBuild
# buildspec.yml
...
phases:
...
build:
commands:
- aws cloudformation package --region $REGION --template template.yml --s3-bucket $S3_BUCKET --output-template $OUTPUT_TEMPLATE
The above kicks off a CloudFormation build using our template.yml
# template.yml
...
S3BucketAssets:
Type: AWS::S3::Bucket
...
At this point, it creates a unique name for an S3 bucket. Awesome. Now, for step 2 in my buildspec.yml for CodeBuild, I want to push items to the S3 bucket just created in the CloudFormation template. BUT, I don't know how to get the dynamically created name of the S3 bucket from the CloudFormation template. I want something similar to:
# buildspec.yml
...
phases:
...
build:
commands:
# this is the same step as shown above
- aws cloudformation package --region $REGION --template template.yml --s3-bucket $S3_BUCKET --output-template $OUTPUT_TEMPLATE
# this is the new step
- aws s3 sync dist_files/ s3://{NAME_OF_THE_NEW_S3_BUCKET}
How can I accomplish getting the dynamically named S3 bucket so that I can push to it?
I am aware that within a CloudFormation template, you can reference the S3 bucket name with something like !GetAtt [ClientWebAssets, WebsiteURL]. But, I do not know how to get that information out of the cloudformation template and back into the codebuild template.
You could move to using CodePipeline. Stage one would be deploying the application via cloudformation with an output artifact being the stack creation output
http://docs.aws.amazon.com/codepipeline/latest/userguide/reference-pipeline-structure.html#action-requirements
Related
I have a multi-account setup at my current company. I am using CodeBuild in Account B, I am running a buildspec that uploads a transformed sam template to an s3 bucket in Account A. So the Cloudformation package uploads the sam build of the code to the "account-a-bucket". The job successfully uploads the transformed template as defined in the artifacts section, to a bucket in Account B. The problem comes when trying to deploy the template in account C. Because codebuild creates the lambda code artifact in the build step and writes the object to the bucket outside of the account. When you go look at the actual lambda artifact depositied in the bucket, so s3:://account-a-bucket/e309uofijokasjdfokajsllsk , you will see in the object permission that it does not belong to any account. Therefore no one can access it. How do I make codebuild create the object in the other account bucket, so it is owned by an account?
To note as well, I already configured the Bucket Policy to grant access to account-a-bucket to all accounts in my organization, additionally the canonical ids of the accounts for permissions. So I know for a fact is the lambda artifact is created with no canonical account owner, however the artifact that is uploaded (in the artifact section of buildspec) is created under the account canonical id.
I know you can use, if for example I was uploading in the build phase using an
aws s3api copy-object --bucket destination_awsexammplebucket --key source_awsexamplebucket/myobject --acl bucket-owner-full-control
I could use --acl bucket-owner-full-control, but that is not a supported flag with aws cloudformation package.
env:
variables:
RELEASE_NUMBER: ""
MINOR_NUMBER: "value"
phases:
install:
runtime-versions:
docker: 18
build:
commands:
- echo Building Release Version $RELEASE_NUMBER
- pip install --user aws-sam-cli
- USER_BASE_PATH=$(python -m site --user-base)
- export PATH=$PATH:$USER_BASE_PATH/bin
- sam build -t template.yaml
- aws cloudformation package --template-file template.yaml --s3-bucket account-a-bucket --output-template-file TransformedTemplate.yaml
artifacts:
files:
- TransformedTemplate.yaml
discard-paths: yes
The 'aws cloudformation package' command does not have an "--acl" option which is the cause of the issue you are facing. This is an open issue [1] but one that has not got any traction.
For now, I am thinking you can parse out the S3 object key from 'TransformedTemplate.yaml', and then run the following command in your buildspec to put an ACL on the S3 object:
$ aws s3api put-object-acl --bucket account-a-bucket --key keyname --acl bucket-owner-full-control
For parsing a json file, 'jq' is probably the best utility. Since you are using Yaml, yq [2] seems to be an option though I have never tested it myself.
Ref:
[1] https://github.com/aws/aws-cli/issues/2681
[2] https://yq.readthedocs.io/en/latest/
Created a CloudFormation stack template named foo.yaml. I validate, package, and deploy a stack from foo.yaml:
aws cloudformation package `
--template-file .\foo.yaml`
--s3-bucket abc123 `
--output-template-file .\foo.pkg.yaml
$parameters = ...
aws cloudformation deploy `
--template-file .\foo.pkg.yaml `
--stack-name foo `
--capabilities CAPABILITY_IAM `
--s3-bucket abc123 `
--parameter-overrides $parameters
I then delete the S3 bucket used in that deployment. Is there a way I can regenerate that bucket from CloudFormation?
P.S. I realize I can create the bucket and populate it manually, but I'm asking if there is anything automatic.
No, there isn't a button or option to re-populate an S3 bucket with objects created when a stack's template was packaged.
You can ONLY do this manually:
Find the CF stack instance's template
Locate the bucket name from the template
Create a new S3 bucket with the same name
Re-package your CF stack template adding the bucket name from step-3 to the aws cloudformation package command:
aws cloudformation package --s3-bucket
Bucket recreated w/artifacts!
If you're asking if the bucket can be created automatically from a CloudFormation command, then it's not really possible. You can create a S3 Bucket Resource ofcourse as a part of a Stack Resource.
I believe it is simpler to create the Bucket manually.
I'm trying to create and then zip a Docker container to upload to S3 to be run by an AWS Lambda function. I was trying to work off an article but the instructions are sparse (https://github.com/abhisuri97/auto-alt-text-lambda-api).
I've installed Docker and the Amazon Linux image but I don't know how to create a Docker container that contains the github repo, and then zip it so that it can be accessed by Lambda.
This is what I've tried to piece together from other tutorials:
git clone https://github.com/abhisuri97/auto-alt-text-lambda-api.git
cd auto-alt-text-lambda-api
docker run -v -it amazonlinux:2017.12
zip -r -9 -q ~/main.zip
Any help would be greatly appreciated.
The instructions aren't clear but I suspect the reference to Docker is just for testing. You don't need Docker to run an AWS Lambda function. You will need an AWS API Gateway API though to execute the Lambda function over HTTPS.
I'd recommend starting with a CloudFormation stack using the AWS Serverless Application Mode (https://docs.aws.amazon.com/lambda/latest/dg/serverless_app.html).
Create an S3 bucket for the zip file and create a CloudFormation template similar to:
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Resources:
LambdaFunction:
Type: AWS::Serverless::Function
Properties:
Handler: application.predict
Runtime: python2.7
Events:
HttpGet:
Type: Api
Properties:
Path: 'auto-alt-text-api'
Method: get
Package the Lambda function with:
aws cloudformation package --template-file template.yaml --output-template-file template-out.yaml --s3-bucket <your-bucket> --s3-prefix <your-prefix>
Then deploy it with:
aws cloudformation deploy --template-file template-out.yaml --stack-name auto-alt-text-lambda-api-stack --capabilities CAPABILITY_IAM
You will probably have to add IAM roles and Lambda permissions to the template for the application to work properly.
Follow the read for an example it says:
# AWS SAM Hello World Example #
A simple AWS SAM template that specifies a single Lambda function.
## Usage ##
To create and deploy the SAM Hello World example, first ensure that you've met the requirements described in the [root README](../../README.md). Then
follow the steps below.
### Test your application locally ###
Use [SAM Local](https://github.com/awslabs/aws-sam-local) to run your Lambda function locally:
sam local invoke "HelloWorldFunction" -e event.json
### Package artifacts ###
Run the following command, replacing `BUCKET-NAME` with the name of your bucket:
sam package --template-file template.yaml --s3-bucket BUCKET-NAME --output-template-file packaged-template.yaml
This creates a new template file, packaged-template.yaml, that you will use to deploy your serverless application.
### Deploy to AWS CloudFormation ###
Run the following command, replacing `MY-NEW-STACK` with a name for your CloudFormation stack.
sam deploy --template-file packaged-template.yaml --stack-name MY-NEW-STACK --capabilities CAPABILITY_IAM
This uploads your template to an S3 bucket and deploys the specified resources using AWS CloudFormation.
Now what is the sam local command to delete the whole stack including s3 bucket and CF stack?
Edit for 2022:
Sam cli now has a delete command can see official docs here. Should be able run like this now:
sam delete --stack-name MY-NEW-STACK
Thanks #jesusnoseq for pointer to update 🍻.
Legacy 2018 Answer
There is currently no sam command to delete all of these resources. You would just use the relevant aws cli commands which sam-cli is just a wrapper around anyways. Example to delete you CFN stack.
aws cloudformation delete-stack --stack-name MY-NEW-STACK
I'm attempting to setup AWS Codepipeline with a Cloudformation SAM stack that deploys to a Lambda and am doing this via the Pipeline console. The pipeline passes the clone from Codecommit process and the build process with Codebuild, however fails on the deploy with Cloudformation.
I get the following error message:
Action execution failed
Invalid TemplatePath: MyAppBuild::samTemplate.yml
I've been following the documentation (http://docs.aws.amazon.com/lambda/latest/dg/automating-deployment.html) pretty closely and in field 16 of the codepipeline setup form (Template File) I have been putting samTemplate.yml.
I also have samTemplate.yml in the root of my repo (which is also the root of the project).
I've matched the Codebuild output artifact name with the Cloudformation input artifact name and they match exactly.
Am I missing something here? How do I get the Cloudformation deploy process to recognize the sam template?
EDIT 1 I've switched to using Codestar instead of directly using CodePipeline. Nothing special in my buildspec.yml, but it's below if anyone's interested.
version: 0.2
phases:
install:
commands:
- echo "install"
pre_build:
commands:
- echo "pre_build"
build:
commands:
- aws cloudformation package --template template.yml --s3-bucket $S3_BUCKET --output-template template-export.json
artifacts:
type: zip
files:
- template-export.json
One thing to keep in mind for anyone new to CodeBuild, the zip file that gets created as a result is of the root level directory of your build operations and IS the zip file that gets transferred to final deployment (in my case, Lambda).
You need to add aws cloudformation package command in you buildspec.yml.
aws cloudformation package --debug --template-file <YourSamTemplate.yml> --s3-bucket <YourbucketName> --output-template-file <YourOutputSamTemplate.yml>
where YourSamTemplate.yml is the name of samTemplate.yml in your project root.
and YourOutputSamTemplate.yml is the new name which you eant to give your output file after cloud formation package command is done.
Then in the next stage you need to define your input-artifact as output of your aws codebuild stage and then use this input-artifact to map your template.
build-output::YourOutputSamTemplate.yml
I believe the output of the build stage is what needs to be passed into the CloudFormation action.
In the tutorial the build output is called NewSamTemplate.yaml
So try updating your TemplatePath to MyAppBuild::NewSamTemplate.yml
In my case, the error was in the filename. Instead of template.yml the actual file was template.yaml