I have AWS Glue jobs setup to upload test data to our database. Uploading takes place only 1 time and no additional runs are required unless additions or changes are required on the test data. However, we have multiple environments where the upload needs to happen. One way is to deploy the jobs using CDK and manually run the jobs in each environment. Looking for pointers to automatically trigger a run when the jobs is either updated or created.
Use a CustomResource to invoke it via command line or the Glue SDK.
It is important to remember that CDK is not a deployment solution - it is an infrastructure as code solution. CDK does not actually do any deployment - the cdk deploy command is just a shortcut for sending the template to CloudFormation.
CDK is just a way to lay code over top the creation of the CloudFormation templates and give developers far more options. All it really does is generate a CloudFormation Template - everything else is window dressing.
As such, anything that has to happen after the CloudFormation template is synthed and deployed is not possible for CDK to interact with. You need to make a custom resource that can watch for Stack Updates and when the stack is done deploying, trigger whatever else you want.
Alternatively, this is a perfect use of CodePipeline - run your cdk in the the pipeline (either with a Synth and CodeDeploy stages or a single codeBuild that just runs cdk deploy) and then in a stage after it have a lambda that triggers your jobs.
I have an AWS Lambda "Application" which was created from an AWS Lambda app template. This in turn created two stacks. The serverlessrepo-??-toolchain stack and the Lambda Application stack that has the actual application...
I've done the development and added lambdas, permission, and such. Really evolved the template.yml and the buildspec.yml.
It all works and properly rebuilds the stack.
But in an AWS Lambda Application that is using CodeDeploy/CodePipeline, what is the best strategy for deploying additional environments? Let's assume the first one - the one made by the serverlessrepo-??-toolchain stack -- is Dev. How do I create a QA and Prod from my template.yml?
They need to be new stacks, yes? As in each environment is its own stack.
Thank you.
I'm learning SAM, and I created two projects.
The first one, example1, I created it from the AWS web console, by going to Lambda, Applications, and choosing this template:
After the wizard finishes creating the app, it looks like this:
I'm interested in the yellow-highlighted area because I don't understand it yet.
I tried to replicate this more or less manually by using sam init and created example2. It's easy to look at the template.yml it creates and see how the stuff in Resources are created, but how is the stuff in Infrastructure created.
When I deploy example2 with sam deploy --guided, indeed there's nothing in Infrastructure:
Given example2, how should I go about creating the same infrastructure as example1 had out of the box (and then changing it, for example, I want several environments, prod, staging, etc). Is this point and click in the AWS console or can it be done with CloudFormation?
I tried adding a permission boundary to example2, on of the things example1 has in Infrastructure, I created the policy in IAM (manually, in the console), added it to the template.yml, and deployed it but it didn't show up in "Infrastructure".
Part 1: In which I answer your question
Where are these infrastructure entries coming from in AWS SAM?
I replicated your steps in the Lambda console to create a "Serverless API Backend" called super-app. When you press create, AWS creates
two CloudFormation Stacks, each with a YAML template. You can view the stack resources and the YAML templates in the CloudFormation console under Stacks > Templates Tab.
super-app: the "Resources" stack with the lambda and dynamo resources you managed to replicate.
serverlessrepo-super-app-toolchain: the mystery stack with the "Infrastructure" CI/CD resources1.
Is this point and click in the AWS console or can it be done with CloudFormation?
Yes and Yes. You can use sam deploy (or aws cloudformation deploy) to update the stacks. Or point and click.
Example: update the serverlessrepo-super-app-toolchain template with the SAM CLI:
# compile
sam build -t cicd_template.yaml --region us-east-1 --profile sandbox
# send changes to the cloud
sam deploy --stack-name serverlessrepo-super-app-toolchain --capabilities CAPABILITY_NAMED_IAM --region us-east-1 --profile sandbox
You must pass in values for the template parameters at deploy-time. The current values for the parameters are in the console under CloudFormation > Stack > Parameters Tab. You can pass them using the --parameter-overrides param in the deploy command. If the
parameters are static, I find it easier to pass SAM parameter values in samconfig.toml, which sam deploy will use by default:
# samconfig.toml
version = 0.1
[default]
[default.deploy]
[default.deploy.parameters]
# template default parameters - fill in the template blanks
# Where do the values come from? the CloudFormation console, Parameters tab
AppId = "super-app"
AppResourceArns = "arn:aws:lambda:us-east-1:1xxxxxx:function..."
ConnectionArn = "arn:aws:codestar-connections:us-east-1:xxxxxx:connection/xxxx3c5c-f0fe-4eb9-8164-d3c2xxxxx6e2"
GitHubRepositoryOwner = "mygithuborg"
RepositoryName = "super-app"
SourceCodeBucketKey = "sample-apps/nodejs/14.x/javascript/sam/web-backend.zip"
SourceCodeBucketName = "prodiadstack-subsystemsn-apptemplatesbucket03axxx-96eem3xxxxxx"
UseCodeCommit = false
If there were changes made in the template, they will deploy. Success!
Part 2: In which I try to convince you to use the CDK instead
SAM and YAML templates are far from dead, but I think it's safe to say that for proficient developers starting out with AWS, the newer AWS Cloud Development Kit is a natural first choice for ambitious applications that need CI/CD and testing. For most of us, editing a 800-line YAML file is not a fun experience.
AWS Infrastructure-As-Code
There are lots AWS and 3rd Party IaaC tools to deploy infra on AWS. Each abstraction is best for somebody sometime. The important thing to remember is that no matter what higher-level IaaC toolset you use, it ends up being deployed as a CloudFormation template. Here are the AWS approaches, oldest to newest:
CloudFormation YAML2 templates
The OG, all-powerful, lowest-level approach is to hand-code YAML templates. The Cfn template reference
docs are indespensible no
matter what tool you use, because that's what gets deployed.
SAM YAML templates
With AWS SAM, you
still handcode YAML, but less3. A SAM template is a superset of CloudFormation with some higher-level abstractions for the main serverless components like Lambdas, DynamoDB tables and Queues. The SAM CLI compiles the SAM template to Cfn. It has nifty features like local testing and deploy conveniences.
Cloud Development Kit
The newest, shiniest IaaC approach is the CDK, now on V2. With the CDK, we write Typescript/Python/Java/etc. instead of YAML. The CDK CLI compiles your language code to Cfn and deploys with cdk deploy. It has a bigger set of high-level infra abstractions that goes beyond serverless, and escape hatches to expose low-level Cfn constructs for advanced use cases. It natively supports testing and CI/CD.
AWS CDK workshop including testing and pipelines. Lots of AWS CDK example apps.
Note that CloudFormation is the ultimate soure of this info. The lambda console makes a cloudformation.DescribeStack API call to fetch it.
YAML or JSON
SAM also has a marketplace-like repository with reusable AWS and 3rd party components
Edit :
If I understand correctly, you want to reproduce the deployment on the SAM app. If that's the case, there is an AWS sample that covers the same approach.
It seems you are using either CodeStar/CodeCommit/CodePipeline/CodeDeploy/Code... etc. from AWS to deploy your SAM application on example1.
At deploy time, these resources under infrastructure are created by the "Code" services family in order to authorize, instantiate, build, validate, store, and deploy your application to CloudFormation.
On the other hand, on example2, whenever you build your project in your local machine, both instantiation, build, validation, storage (of the upload-able built artifacts) are leveraged by your own device, hence not needed be provisioned by AWS.
To shortly answer your question: No. Your can't recreate these infrastructure resources on your own. But again, you wouldn't need to do so while deploying outside of AWS' code services.
Source code in my organization is managed in a GitHub repository. For now, our CI process uses AWS CodePipeline as follows:
Webhooks detect code changes in a specific git branch
The updated branch is then used the input for AWS CodeBuild
The finished build is deployed onto one of our staging environments using Elastic Beanstalk
Tests are run on the Elastic Beanstalk environment.
We want to add detection of new pull requests in our git repository. Whenever a new PR is created in our repo, we'd like to automatically trigger a build to an EB environment, through CodePipeline as above.
Our roadblocks:
Looking at the available settings for GitHub Webhooks in CodePipeline, we cannot find a way to specify that the pipeline's trigger should be a new PR.
In any case, the GitHub source for a CodePipeline must be a specific branch. We'd like PRs to be detected in any branch.
What would be the best approach here? I've seen some methods being discussed, but most of them appear to be on the cumbersome/high-maintenance side. If there's anything new in the AWS toolchain that makes this easy, it'd be cool to know.
Thanks!
The best approach to solving this problem seems to be creating a CodePipeline for each PR using a parameterized CloudFormation stack.
Essentially the steps are:
Define your CodePipeline using CloudFormation and have a parameter that identifies the environment - Prod, QA, PR_xyz etc.
Set up CodeBuild to trigger on any changes to your GitHub repository. When a new PR is created, have CodeBuild construct a new CodePipeline based on your CloudFormation template. Supply the name of the PR as the environment name when creating the CloudFormation stack.
Detailed steps are described here: https://moduscreate.com/blog/track-git-branches-aws-codepipeline/
Is there any way to automate cloudformation templates deployment? I mean it would be awesome if I just push the changes in the code and somebody looking for those changes in the code and once they appear - deploy the updated template.
Yes, with AWS Code Pipeline !
http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/continuous-delivery-codepipeline.html
With AWS CloudFormation and AWS CodePipeline, you can use continuous delivery to automatically build and test changes to your AWS CloudFormation templates [...]
AWS CodePipeline has built-in integration with AWS CloudFormation, so you can specify AWS CloudFormation-specific actions, such as creating, updating, or deleting a stack, within a pipeline.
CodePipeline is a great way to do what you're looking for.
At Giftbit we do this is by having a Github Repo that has our CloudFormation template in it.
When we want to make a change, we make the changes on a branch in the repo, and create a pull request into the staging branch. CodePipeline monitors the staging branch then automates a CodeBuild to validate the templates, package any SubStacks, then creates a Change Set and Executes it.
Below are have some examples to help Quick Start anyone interested:
Continuous Integration CloudFormation Template Example
Serverless Application Model (SAM) that gets deployed