I have AWS Glue jobs setup to upload test data to our database. Uploading takes place only 1 time and no additional runs are required unless additions or changes are required on the test data. However, we have multiple environments where the upload needs to happen. One way is to deploy the jobs using CDK and manually run the jobs in each environment. Looking for pointers to automatically trigger a run when the jobs is either updated or created.
Use a CustomResource to invoke it via command line or the Glue SDK.
It is important to remember that CDK is not a deployment solution - it is an infrastructure as code solution. CDK does not actually do any deployment - the cdk deploy command is just a shortcut for sending the template to CloudFormation.
CDK is just a way to lay code over top the creation of the CloudFormation templates and give developers far more options. All it really does is generate a CloudFormation Template - everything else is window dressing.
As such, anything that has to happen after the CloudFormation template is synthed and deployed is not possible for CDK to interact with. You need to make a custom resource that can watch for Stack Updates and when the stack is done deploying, trigger whatever else you want.
Alternatively, this is a perfect use of CodePipeline - run your cdk in the the pipeline (either with a Synth and CodeDeploy stages or a single codeBuild that just runs cdk deploy) and then in a stage after it have a lambda that triggers your jobs.
I'm learning SAM, and I created two projects.
The first one, example1, I created it from the AWS web console, by going to Lambda, Applications, and choosing this template:
After the wizard finishes creating the app, it looks like this:
I'm interested in the yellow-highlighted area because I don't understand it yet.
I tried to replicate this more or less manually by using sam init and created example2. It's easy to look at the template.yml it creates and see how the stuff in Resources are created, but how is the stuff in Infrastructure created.
When I deploy example2 with sam deploy --guided, indeed there's nothing in Infrastructure:
Given example2, how should I go about creating the same infrastructure as example1 had out of the box (and then changing it, for example, I want several environments, prod, staging, etc). Is this point and click in the AWS console or can it be done with CloudFormation?
I tried adding a permission boundary to example2, on of the things example1 has in Infrastructure, I created the policy in IAM (manually, in the console), added it to the template.yml, and deployed it but it didn't show up in "Infrastructure".
Part 1: In which I answer your question
Where are these infrastructure entries coming from in AWS SAM?
I replicated your steps in the Lambda console to create a "Serverless API Backend" called super-app. When you press create, AWS creates
two CloudFormation Stacks, each with a YAML template. You can view the stack resources and the YAML templates in the CloudFormation console under Stacks > Templates Tab.
super-app: the "Resources" stack with the lambda and dynamo resources you managed to replicate.
serverlessrepo-super-app-toolchain: the mystery stack with the "Infrastructure" CI/CD resources1.
Is this point and click in the AWS console or can it be done with CloudFormation?
Yes and Yes. You can use sam deploy (or aws cloudformation deploy) to update the stacks. Or point and click.
Example: update the serverlessrepo-super-app-toolchain template with the SAM CLI:
# compile
sam build -t cicd_template.yaml --region us-east-1 --profile sandbox
# send changes to the cloud
sam deploy --stack-name serverlessrepo-super-app-toolchain --capabilities CAPABILITY_NAMED_IAM --region us-east-1 --profile sandbox
You must pass in values for the template parameters at deploy-time. The current values for the parameters are in the console under CloudFormation > Stack > Parameters Tab. You can pass them using the --parameter-overrides param in the deploy command. If the
parameters are static, I find it easier to pass SAM parameter values in samconfig.toml, which sam deploy will use by default:
# samconfig.toml
version = 0.1
[default]
[default.deploy]
[default.deploy.parameters]
# template default parameters - fill in the template blanks
# Where do the values come from? the CloudFormation console, Parameters tab
AppId = "super-app"
AppResourceArns = "arn:aws:lambda:us-east-1:1xxxxxx:function..."
ConnectionArn = "arn:aws:codestar-connections:us-east-1:xxxxxx:connection/xxxx3c5c-f0fe-4eb9-8164-d3c2xxxxx6e2"
GitHubRepositoryOwner = "mygithuborg"
RepositoryName = "super-app"
SourceCodeBucketKey = "sample-apps/nodejs/14.x/javascript/sam/web-backend.zip"
SourceCodeBucketName = "prodiadstack-subsystemsn-apptemplatesbucket03axxx-96eem3xxxxxx"
UseCodeCommit = false
If there were changes made in the template, they will deploy. Success!
Part 2: In which I try to convince you to use the CDK instead
SAM and YAML templates are far from dead, but I think it's safe to say that for proficient developers starting out with AWS, the newer AWS Cloud Development Kit is a natural first choice for ambitious applications that need CI/CD and testing. For most of us, editing a 800-line YAML file is not a fun experience.
AWS Infrastructure-As-Code
There are lots AWS and 3rd Party IaaC tools to deploy infra on AWS. Each abstraction is best for somebody sometime. The important thing to remember is that no matter what higher-level IaaC toolset you use, it ends up being deployed as a CloudFormation template. Here are the AWS approaches, oldest to newest:
CloudFormation YAML2 templates
The OG, all-powerful, lowest-level approach is to hand-code YAML templates. The Cfn template reference
docs are indespensible no
matter what tool you use, because that's what gets deployed.
SAM YAML templates
With AWS SAM, you
still handcode YAML, but less3. A SAM template is a superset of CloudFormation with some higher-level abstractions for the main serverless components like Lambdas, DynamoDB tables and Queues. The SAM CLI compiles the SAM template to Cfn. It has nifty features like local testing and deploy conveniences.
Cloud Development Kit
The newest, shiniest IaaC approach is the CDK, now on V2. With the CDK, we write Typescript/Python/Java/etc. instead of YAML. The CDK CLI compiles your language code to Cfn and deploys with cdk deploy. It has a bigger set of high-level infra abstractions that goes beyond serverless, and escape hatches to expose low-level Cfn constructs for advanced use cases. It natively supports testing and CI/CD.
AWS CDK workshop including testing and pipelines. Lots of AWS CDK example apps.
Note that CloudFormation is the ultimate soure of this info. The lambda console makes a cloudformation.DescribeStack API call to fetch it.
YAML or JSON
SAM also has a marketplace-like repository with reusable AWS and 3rd party components
Edit :
If I understand correctly, you want to reproduce the deployment on the SAM app. If that's the case, there is an AWS sample that covers the same approach.
It seems you are using either CodeStar/CodeCommit/CodePipeline/CodeDeploy/Code... etc. from AWS to deploy your SAM application on example1.
At deploy time, these resources under infrastructure are created by the "Code" services family in order to authorize, instantiate, build, validate, store, and deploy your application to CloudFormation.
On the other hand, on example2, whenever you build your project in your local machine, both instantiation, build, validation, storage (of the upload-able built artifacts) are leveraged by your own device, hence not needed be provisioned by AWS.
To shortly answer your question: No. Your can't recreate these infrastructure resources on your own. But again, you wouldn't need to do so while deploying outside of AWS' code services.
I'm trying to figure out a way to come up with a CI/CD pipeline for CloudFormation. We use Cloudformation Console directly to deploy our infrastructure and app to the cloud.
Does anyone have any examples of how they have created a CI/CD pipeline using Jenkins or other types of CI tools to do some type of linting, CI, version control, and artifact deployment to Artifactory (or similar toolset)? I'd like to execute a pipeline once a new version of the cloud formation templates is uploaded to Artifactory.
You can always use CodePipeline.
see docs:
CodePipeline
CI:
I am using GitHub, so before i can merge a pull request, my code must pass 3 tests.
Those tests are 3 Codebuilds containers that run tests.
CD:
After my code merged it invoke a CodePipeline that use mainly CodeDeploy and CodeBuild.
About your goal:
I'd like to execute a pipeline once a new version of the cloud formation templates is uploaded to Artifactory.
I don't really think you need a pipeline for this.
Let assume your artifacts uploaded to s3 bucket called artifact-bucket.
You can create a CloudWatch rule that will execute StepFunctions state machine when file added to
artifact-bucket.
see docs:
Trigger StepFunctions from S3
You can easily deploy stack with StepFunctions.
Source code in my organization is managed in a GitHub repository. For now, our CI process uses AWS CodePipeline as follows:
Webhooks detect code changes in a specific git branch
The updated branch is then used the input for AWS CodeBuild
The finished build is deployed onto one of our staging environments using Elastic Beanstalk
Tests are run on the Elastic Beanstalk environment.
We want to add detection of new pull requests in our git repository. Whenever a new PR is created in our repo, we'd like to automatically trigger a build to an EB environment, through CodePipeline as above.
Our roadblocks:
Looking at the available settings for GitHub Webhooks in CodePipeline, we cannot find a way to specify that the pipeline's trigger should be a new PR.
In any case, the GitHub source for a CodePipeline must be a specific branch. We'd like PRs to be detected in any branch.
What would be the best approach here? I've seen some methods being discussed, but most of them appear to be on the cumbersome/high-maintenance side. If there's anything new in the AWS toolchain that makes this easy, it'd be cool to know.
Thanks!
The best approach to solving this problem seems to be creating a CodePipeline for each PR using a parameterized CloudFormation stack.
Essentially the steps are:
Define your CodePipeline using CloudFormation and have a parameter that identifies the environment - Prod, QA, PR_xyz etc.
Set up CodeBuild to trigger on any changes to your GitHub repository. When a new PR is created, have CodeBuild construct a new CodePipeline based on your CloudFormation template. Supply the name of the PR as the environment name when creating the CloudFormation stack.
Detailed steps are described here: https://moduscreate.com/blog/track-git-branches-aws-codepipeline/
I want to achieve a Continuous delivery for provisioning AWS resources using Bitbucket & AWS. My use case is to create a kinesis Firehose Delivery stream with destination as Elastic Search. I want this to be achieved by using the AWS cloudformation templates (keeping in mind the different stages for dev, uat, prod). Whenever I update my bitbucket repo the build should get created and the stack will get updated in AWS. Any help will be highly appreciated.
I have searched a lot over the internet but could not find any relevant examples which clearly describes my use case.
Cloudformatiom template committed in bitbucket will provision AWS resources in the cloud
You can use AWS CodePipeline for this purpose. The only issue is that Code Pipeline does not work directly with Bitbucket, only works with AWS CodeCommit or GitHub as the triggering repo.
But there is a workaround for that. You can set up syncing from Bitbucket to GitHub and then set the GitHub repository as the source repository for the pipeline you will create on AWS CodePipeline. You can find many guides for syncing so I won't explain here. The pipeline itself can be defined as a template.
I have already explained Setting up AWS CodePipeline in another answer here that you can follow for this purpose. Hope this helps!