Source code in my organization is managed in a GitHub repository. For now, our CI process uses AWS CodePipeline as follows:
Webhooks detect code changes in a specific git branch
The updated branch is then used the input for AWS CodeBuild
The finished build is deployed onto one of our staging environments using Elastic Beanstalk
Tests are run on the Elastic Beanstalk environment.
We want to add detection of new pull requests in our git repository. Whenever a new PR is created in our repo, we'd like to automatically trigger a build to an EB environment, through CodePipeline as above.
Our roadblocks:
Looking at the available settings for GitHub Webhooks in CodePipeline, we cannot find a way to specify that the pipeline's trigger should be a new PR.
In any case, the GitHub source for a CodePipeline must be a specific branch. We'd like PRs to be detected in any branch.
What would be the best approach here? I've seen some methods being discussed, but most of them appear to be on the cumbersome/high-maintenance side. If there's anything new in the AWS toolchain that makes this easy, it'd be cool to know.
Thanks!
The best approach to solving this problem seems to be creating a CodePipeline for each PR using a parameterized CloudFormation stack.
Essentially the steps are:
Define your CodePipeline using CloudFormation and have a parameter that identifies the environment - Prod, QA, PR_xyz etc.
Set up CodeBuild to trigger on any changes to your GitHub repository. When a new PR is created, have CodeBuild construct a new CodePipeline based on your CloudFormation template. Supply the name of the PR as the environment name when creating the CloudFormation stack.
Detailed steps are described here: https://moduscreate.com/blog/track-git-branches-aws-codepipeline/
Related
Background: I am using github actions for CI and aws codepipeline for CD. Github actions pushes a set of versioned artifacts to S3 and ECR. I setup my AWS codepipeline using CDK.
Question: how do I get the codepipeline to pick up those artifacts and deploy them?
opt 1: Just tag your images and everything else with "latest"
answer: no, having a pipeline that always deploys the latest is not the same as a pipeline that deploys version X.
opt 2: Just send the version number (version X) to codepipeline so that codepipeline knows which artifacts to fetch
answer: no, codepipeline seems to support passing variables between actions (actions generate output variables, other action can pick them up), but I have found no documentation stating that a codepipeline can be triggered with input parameters.
opt 3: tag your commit in github and use a webhook to pass that information along to codepipeline.
answer: no, codepipeline can filter webhooks so that you can trigger the pipeline for certain events, but it does not support parsing the webhook body, picking out stuff you want to use.
opt 4: resolve the version number in cdk synth before that pesky critter tries to update itself.
answer: yeah, that kinda works, I can query an ecr repo, find that actual version number of the release and regenerate the pipeline so that it points to the resolved version. Its not the same as passing a version number from github to codepipeline, but at least my pipeline is versioned and all my deployment units (like ECS services, batch jobs, etc) are pointing to an explicit version after deployment. Unfortunately, this has several drawbacks, like making the deployment pipeline (even) slow(er) and if the pipeline fails I will have update the pipeline by running cdk deploy from my machine.
opt 5: you come in to save the day :-)
I am new to AWS and trying to create a pipeline for CICD. Stages involved in my pipeline are:
Source -> Codecommit
Build -> Codebuild project
Deploy using Cloudformation
I am able to complete the pipeline and deployment is successful. But I am struggling to implement a rollback procedure with this. How to deploy previous version without making a code revert in the repository? Any help regarding this?
I changed the pipeline configuration a bit and now I am able to deploy any version from history. Below is the solution:
Source -> Codecommit
Build -> Codebuild project
Deploy using CodeDeploy instead of Cloudformation
Now, deployment history can be triggered at any time. Pick the version from the history in the deployments under code deploy and retry deployment.
Unfortunately there is currently no rollback step for CodePipeline, traditionally people would rollback by reverting the change from their master branch (which is meant to represent the state of live).
If you're unable to do this revert, then you will need to manage the rollback either from a different service or different pipeline.
As you're using CloudFormation you could take a look at implementing Rollback Triggers which would monitor the status of an alarm. If the alarm fails then it could rollback and fail the pipeline.
Is there any way to automate cloudformation templates deployment? I mean it would be awesome if I just push the changes in the code and somebody looking for those changes in the code and once they appear - deploy the updated template.
Yes, with AWS Code Pipeline !
http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/continuous-delivery-codepipeline.html
With AWS CloudFormation and AWS CodePipeline, you can use continuous delivery to automatically build and test changes to your AWS CloudFormation templates [...]
AWS CodePipeline has built-in integration with AWS CloudFormation, so you can specify AWS CloudFormation-specific actions, such as creating, updating, or deleting a stack, within a pipeline.
CodePipeline is a great way to do what you're looking for.
At Giftbit we do this is by having a Github Repo that has our CloudFormation template in it.
When we want to make a change, we make the changes on a branch in the repo, and create a pull request into the staging branch. CodePipeline monitors the staging branch then automates a CodeBuild to validate the templates, package any SubStacks, then creates a Change Set and Executes it.
Below are have some examples to help Quick Start anyone interested:
Continuous Integration CloudFormation Template Example
Serverless Application Model (SAM) that gets deployed
Can give me some input on configuring AWS CodePipeline to trigger a ZipDownload or a Git pull if the developer commits a code into the Git branch ? I'm new to AWS CodePipeline and AWS Lamba.
For CodePipeline to subscribe to a GitHub repo (using the console):
Create a Source Action of "GitHub", and choose your Repo. Choose an Output artifact name that you will remember for the next step. Under advanced, select "Run pipeline when changes in source content are detected".
Create a Build Action using the build provider of your choice. Choose the Input artifact name you chose in step #1.
When a change occurs in the GitHub repo, CodePipeline will execute the Source action, which will build a zip of the repo as is, and put it in an S3 bucket as an Output artifact. It will then pass this file's S3 name into the Build action (using the variable name given in the steps above), so that the zip file can be downloaded and built.
If you have a working buildspec.yml in the root of the repo, you can use the AWS CodeBuild provider, and the artifacts will be copied to the output bucket when done.
CodePipeline is supposed to manage automation from source code to final deployment, which embrace the concept of continuous integration. Though how to use CodePipeline depends on use case, CodePipeline does source code download for you by detecting source change. Which means what you should thinking about is what to do next after pushing the code, such as run a build action or test action.
It's worth to follow the tutorial to build a pipeline and learn how it works: Tutorial: Create a Simple Pipeline (AWS CodeCommit Repository). It may only take half an hour.
Though it's using codecommit, but it works similar as github. For integration with GitHub, there is a blog you may found useful: Integrating Git with AWS CodePipeline
I have followed this guide:
https://blogs.aws.amazon.com/application-management/post/Tx33XKAKURCCW83/Automatically-Deploy-from-GitHub-Using-AWS-CodeDeploy
It mentions that it will push the default branch from GitHub.
What about all the other branches one might have in the same repo?
Can I somehow specify which branch to deploy?
Here is how you can accomplish branch-specific deploy scenarios using AWS Code Deploy and AWS CodePipeline:
Assuming you've already set up an application and deploy group with Code Deploy, create one group for your "Dev" branch, and another deploy group for "qa" or "stage".
Enable CodePipeline in your AWS Console.
Create a new pipeline by authorizing your Github account and providing access to the repository and branches you desire.
In the BETA section of your new pipeline, edit it, authorize github again, and choose the specific branch you wish to deploy when changes are made.
Now your system will automatically deploy based on a specific branch.
After scratching and cursing and researching and out of the box thinking...I managed to do it like this.
As long as CodeDeploy plays nicely only with the default branch, let's manipulate that one from the GitHUb API [you can do it also from the settings of the GH UI].
This is the code to change/update the default branch from your repo.
I have confirmed that CodeDeploy had no problem deploying the new branch! :]