I'm trying to figure out a way to come up with a CI/CD pipeline for CloudFormation. We use Cloudformation Console directly to deploy our infrastructure and app to the cloud.
Does anyone have any examples of how they have created a CI/CD pipeline using Jenkins or other types of CI tools to do some type of linting, CI, version control, and artifact deployment to Artifactory (or similar toolset)? I'd like to execute a pipeline once a new version of the cloud formation templates is uploaded to Artifactory.
You can always use CodePipeline.
see docs:
CodePipeline
CI:
I am using GitHub, so before i can merge a pull request, my code must pass 3 tests.
Those tests are 3 Codebuilds containers that run tests.
CD:
After my code merged it invoke a CodePipeline that use mainly CodeDeploy and CodeBuild.
About your goal:
I'd like to execute a pipeline once a new version of the cloud formation templates is uploaded to Artifactory.
I don't really think you need a pipeline for this.
Let assume your artifacts uploaded to s3 bucket called artifact-bucket.
You can create a CloudWatch rule that will execute StepFunctions state machine when file added to
artifact-bucket.
see docs:
Trigger StepFunctions from S3
You can easily deploy stack with StepFunctions.
Related
I have AWS Glue jobs setup to upload test data to our database. Uploading takes place only 1 time and no additional runs are required unless additions or changes are required on the test data. However, we have multiple environments where the upload needs to happen. One way is to deploy the jobs using CDK and manually run the jobs in each environment. Looking for pointers to automatically trigger a run when the jobs is either updated or created.
Use a CustomResource to invoke it via command line or the Glue SDK.
It is important to remember that CDK is not a deployment solution - it is an infrastructure as code solution. CDK does not actually do any deployment - the cdk deploy command is just a shortcut for sending the template to CloudFormation.
CDK is just a way to lay code over top the creation of the CloudFormation templates and give developers far more options. All it really does is generate a CloudFormation Template - everything else is window dressing.
As such, anything that has to happen after the CloudFormation template is synthed and deployed is not possible for CDK to interact with. You need to make a custom resource that can watch for Stack Updates and when the stack is done deploying, trigger whatever else you want.
Alternatively, this is a perfect use of CodePipeline - run your cdk in the the pipeline (either with a Synth and CodeDeploy stages or a single codeBuild that just runs cdk deploy) and then in a stage after it have a lambda that triggers your jobs.
Is it possible to create a new REST api gateway using codepipeline? I already have a terraform script to create the pipeline, but I want to know if there is a way to create a pipeline that will take my script and propogate it from a dev environment api gateway to a test environment? I am trying to automate the pipeline to possibly run the script for me once the code is updated in a code commit stash
Any suggestions would be greatly appreciated.
To run a script from AWS CodePipeline you can use AWS CodeBuild action in one of your CodePipeline stages.
With CodeBuild you can specify the list of commands you want to run, like installing and running terraform.
I want to implement CI/CD in AWS CodeCommit.
I know its possible manually to kickstart the process once the code reached CodeCommit. But I am using Azure DevOps Repo as my source code repo and want to automate the process.
The deployement is done using AWS SAM. I am looking for a method like; when I push a code to Azure Repo , it should reach the AWS CodeCommit and do the CI/CD without any further manual intervention.
Is there any way to do that?
Azure repos and CodeCommit are compliant with the git standard. The git standard allows you to specify multiple remotes. This is useful for if you were maintaining a mirror or, as in your use case, you need to do something in different environments.
You can read about setting multiple remotes here (provided by github; even though you’re not using github the process and commands should be the same).
Once you have your multiple remotes setup, you can configure your CI/CD pipeline to kick off its process to deploy your SAM template based on your push; when you push your code changes they will be sent to both your Azure repo, and your CodeCommit repo, and your CI/CD pipeline that is monitoring your CodeCommit repo will see the change and kickoff its execution.
Its worth pointing out that you’ll need to properly setup and configure your CI/CD pipeline. AWS provides a number of services to support this including AWS CodePipeline, AWS CodeBuild, and AWS CodeDeploy.
I have jenkins installed on an AWS EC2 Instance. My end state is whenever I commit cloudformation templates to my bitbucket repo, jenkins will automatically create/update/delete cf stack.
My thoughts on it was via aws cf cli commands in the jenkinsfile after installing aws cli on the server. Is there a better way of approaching this? I am new to devops
You could try AWS Cloudformation Plugin, but it's up for adoption and wasn't updated in 3 years.
I would say your approach with using the AWS cf cli commands looks safer.
I would say using CLI commands in your Jenkins pipelines is a good practice.
I am a fan of setting up Jenkins pipelines using the S3 artifact manager so your pipeline artifacts like CF templates are automatically available from S3. From there just execute the CloudFormation stack in a Jenkins task.
If your hosting Jenkins in AWS it's also nice to just add an IAM role to the instance to control what API actions Jenkins is allowed to run and use a plugin like CloudBees AWS CLI for your pipeline tasks.
Is there any way to automate cloudformation templates deployment? I mean it would be awesome if I just push the changes in the code and somebody looking for those changes in the code and once they appear - deploy the updated template.
Yes, with AWS Code Pipeline !
http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/continuous-delivery-codepipeline.html
With AWS CloudFormation and AWS CodePipeline, you can use continuous delivery to automatically build and test changes to your AWS CloudFormation templates [...]
AWS CodePipeline has built-in integration with AWS CloudFormation, so you can specify AWS CloudFormation-specific actions, such as creating, updating, or deleting a stack, within a pipeline.
CodePipeline is a great way to do what you're looking for.
At Giftbit we do this is by having a Github Repo that has our CloudFormation template in it.
When we want to make a change, we make the changes on a branch in the repo, and create a pull request into the staging branch. CodePipeline monitors the staging branch then automates a CodeBuild to validate the templates, package any SubStacks, then creates a Change Set and Executes it.
Below are have some examples to help Quick Start anyone interested:
Continuous Integration CloudFormation Template Example
Serverless Application Model (SAM) that gets deployed