We have a CodePipeline set up to do a build, deploy to a QA ECS environment, then a manual approval step to deploy to Prod.
What gets confusing though, is when there are several builds running one after another. Several builds get deployed to QA in sequence, but then the Approval button seems to approve them one at a time, and it's not clear which build you're approving when you click on it.
What I would like to be able to do is to approve the latest build, in case
the earlier builds had issues that were fixed by the later builds. What would be the best way to accomplish that?
I had the same problem. Manual approvals are confusing since several pipeline executions can get queued and it's easy to lose track of things. I think we can blame this on CodePipeline's bad UX.
The workaround I settled with is to have two identical pipelines for the same project. They have the same source stage (same repo/branch) but different deploy stages (one deploys to QA, one deploys to prod). No more manual approval stages. The QA pipeline is set to auto-execute when changes in the source (repo/branch) are detected while the Prod pipeline needs to be manually released.
Basically, we replaced the Manual Approval with Manual Release. Manual release always releases the latest from source unlike manual approvals.
You should place the deploy and approval action in the same stage. This lets you approve exactly what you tested. Why? Because exactly one pipeline execution can be in a pipeline stage at any given time.
...approve the latest build, in case the earlier builds had issues
that were fixed by the later builds.
If you want to let later builds catch up, reject the earlier build that is waiting for approval.
One option if you don't want to have multiple pipelines is to by default disable stage transitions into your environments that required controlled releases.
When you are ready to deploy into an environment, you enable the stage transition to allow the most recent release from the previous stage to be processed and then disable the transitions again.
It's still a bit clunky, but reasonably effective once you get used to it. Having to reject each change that comes through becomes very slow and cumbersome to manage, so by disabling transitions you choose when to promote a release.
IMO, CodePipeline should have an option to automatically supersede executions if they are paused at the manual approval stage.
In the CodePipeline UI, you can see the history of Manual approvals in your pipelines' History. Click on History to see what's in progress (Manual Approvals that haven't timed out will always be in progress) and the source (git) short-sha that triggered it (if you need to narrow down to the relevant commit).
To know which Manual approval you're approving, in Pipeline view, click on View current revisions next to the Manual step (to get the Execution ID), then find the matching Execution ID in History (should be the oldest one).
Only way I found to get to the latest Approval is to hit reject n-1 times in the pipeline (where n is how many manual approvals are still in progress) until I only have 1 approval left (or until I find matching Execution ID).
Well, we can solve this problem as how you describe it with development, but it might also be a process glitch.
For example: If we have a development branch, a release branch (staging) and a master branch ( production ) we could easily solve this issue.
Development branch
Things we develop will be going through the development branch stage where we don't need the manual approval, as we don't want to check every changes. We have setup automated unit tests for that.
Release branch
This will deploy to the staging environment where we extensively test the software quality, also based on the regression tests on an acceptance chain with acceptance systems. This should prevent all the big issues before merging towards master branch. Next to that, we could also manually test the release branch on the staging environment. If this works, be happy and easily migrate towards master
Master branch
This will deploy to the production environment with a manual approval before the actual deployment is taking place, knowing for sure you will only push 1 change, being the merge from release to master, preventing the issues you've summarized in the ticket.
Another way is to develop a new AWS feature where you can uncheck or check a checkbox saying: always take the latest release, but that will not help adding value to the pipeline integration as things will be pushed without testing well enough.
Related
We have a CodePipeline that runs on every GitHub commit/merge to the main branch, building the application and releasing it to a staging environment where we can manually test the application. Every now and then, ad-hoc, weekly, etc, depending on the project, we'd release to production manually. To implement this I added a ManualApprovalStep to my CodePipeline between staging and production but that means that my pipeline is never green. It's always stuck in blue:
This makes me think that I'm using the wrong tool here.
My mental model is coming from Heroku (ignore the review apps, I'm not tackling that challenge yet):
In Heroku there's a Tests tab that's green if the tests pass and there's a pipeline that's green if it gets deployed to staging. Lack of promotion to production in Heroku is not a non-green state in Heroku but it would be in ManualApprovalStep.
Is there another tool that AWS gives me to model this way of working that I'm missing?
Update: another big difference. The ManualApprovalStep seems to pile each change and releasing each change, one by one, not releasing whatever was the last release to staging, so clearly it's not analogous to the release to production that Heroku has.
You are right that the ManualApprovalStep is not a natural "promotion" mechanism. They are for yes-no approvals and will result in execution failure if rejected or after 7 days. Disabled Stage Transitions also sit awkwardly with your use case.
pipeline.CodePipline executions are (a) triggered on a change to a source and (b) meant to run all stages from start-to-finish. Executions are hard to interrupt. A consequence of a requirement to deploy environments independently is that environments are best modelled as independent pipelines, not stages within a single pipeline.
Simple Option: 2 github branches, 2 Pipelines
Clone your pipeline setup. A staging pipeline is tied to a staging branch source. A prod pipeline is triggered on changes to the main branch. This setup is easy to reason about and has the advantage that deploys always match your source. But it does not replicate the Heroku "promotion" concept.
Complex Option: 1 github branch, 2 Pipeline?
You could probably get something closer to the "promotion" pattern by having a pipelines.CodePipeline deployment for staging (tied to github) and a separate codepipeline.Pipeline pipeline for prod. The latter can be triggered by EventBridge events. Asset handling would be complex in this scenario.
[Edit:] Amplify CI/CD for the Front-end, CodePipeline for Back-end
AWS Amplify CI/CD gives you automatic feature branch deploys, PR review approvals etc. for front-end apps. Manual deploys require a workaround, but are possible. See this related SO question. The CDK supports Amplify build configurations. The catch is that these CI/CD goodies work for front-end apps, but not for arbitrary infrastructure stacks. To get the best of both worlds, split the app in two. Use Amplify for the high-velocity front-end and stick with CodePipeline for the back-end deploys.
I am not able to find a way to stop the auto triggering of the pipeline whenever I push code to bitbucket.
My assumption is that you want more control over when your pipeline does certain things.
Rather than achieving this through stopping the pipeline from getting triggered, I'd recommend using either stage transitions or manual approvals to achieve this control inside the pipeline.
Stage transitions are better when you want to "turn off" a pipeline and have the latest thing run through when you turn it back on.
Manual approvals are better when you want the version to be locked while waiting for approval so you can run tests without worrying that the version will change.
You mentioned in your comment that you wanted to only run your pipeline at certain times, so a way you could do that is to enable and disable the stage transition after source on a schedule.
https://docs.aws.amazon.com/codepipeline/latest/userguide/transitions.html
https://docs.aws.amazon.com/codepipeline/latest/userguide/approvals.html
You can disable DetectChanges parameter on your Source action as explained here. Extract with the relevant context:
DetectChanges: Controls automatically starting your pipeline when a new commit is made on the configured repository and branch. If unspecified, the default value is true, and the field does not display by default.
This works on Bitbucket, GitHub, and GitHub Enterprise Server actions. I have a CloudFormation template configured with this option and works. Not sure about the same option on AWS console, because I saw that some configurations are only available from CloudFormation or aws cli. As you can read "this field does not display by default".
My team has been running into issues with our CodePipeline where features were pushed out into production when they shouldn't have been due to our Docker image patching. A little background on our architecture: Our pipeline has two sources, one for the source code and one for the Docker image builder. Docker builds via CodeBuild and is deployed to dev, test, and then prod environments with manual approval steps in between.
Our Docker image receives monthly patching which triggers the pipeline to execute and is what caused the features to be pushed out. We redesigned our git branching strategy so that our master branch will only contain stable releases, but I could still see this issue potentially occurring again if a specific release date is specified. Is there a way to push out the image patching without pushing out the latest commit?
Can CodePipeline Use a Specific Commit
This is an often requested feature but unfortunately CodePipeline will always bring the latest commit from the selected branch in the Source action.
CodePipeline tied to a single git branch is more of a feature of CodePipeline as the design is more inclined towards Trunk based development [0]. Also, as per the designers of this service, CodePipeline is designed for post-merge/release validation. That is, once your change is ready to be released to production and is merged into your master/main branch, CodePipeline takes over and automatically tests and releases the final merged set of changes. CodePipeline has a lot of features like stage locking, superseding versions, etc. which don't work well for the case where you want to test a change in isolation before it's merged (e.g. feature branch testing or pull request testing.) Therefore there currently isn't a recommended way to do this in CodePipeline.
[0] https://trunkbaseddevelopment.com/
Having said that, there is a way to hack this with S3 Source action in pipeline instead of GitHub/CodeCommit source action. Essentially your pipeline's S3 source action is tied to S3 bucket/key. You can then upload a zip of any specific commit to this S3 bucket/key and trigger the pipeline.
I have been using AWS for more than a year now. Lately, I have been focusing on building a CI/CD Pipeline.
My pipeline has 4 stages:
Source (Github)
Testing (using CodeBuild)
Staging (deploys to Staging)
Manual Approval
Prod (deploys to Staging)
According to this AWS Doc, If no response is submitted within seven days, the action is marked as "Failed."
The Pipeline is relatively active (several deploys to staging per day) and what I found out is, that the Approvals "queue" and you have to Approve many times before the most recent changes get to Production.
Is there a way to set the expiration time of an Approval to less than 7 days?
There actually isn't a queue, but while an approval action is in-progress it holds the stage "lock" for that stage so that the change in that stage does not change underneath you while you run manual testing.
While that stage lock is held, there is a "slot" for the change waiting to be promoted into that stage when the stage lock is released. As newer changes pass the previous stage they will replace the change in the slot. Therefore, when you approve or reject a manual approval action, only the most recent pending change is promoted.
Rather than a manual approval, you might want to just disable the transition between staging and prod. Disabling the transition won't hold the lock in either stage, so when you enable it again the most recent change will be promoted.
Transitions are better for when you want to simply control when you deploy to prod, and manual approvals are better when you want to run some manual testing against a consistent version.
See this documentation on transitions: https://docs.aws.amazon.com/codepipeline/latest/userguide/transitions.html
Development teams are often plagued by builds in version control being transiently broken. The entire team's productivity can come to a halt while trying recover from a build broken by one person.
Is there software that would allow hosting Git in a way that prevents breaking builds in version control by not accepting commits that fail to pass tests in the first place? The usage scenario could for example look as follows:
The software runs on a server that continuously pulls revisions from Git repositories that developers have published.
For each pulled revision, the software builds the revision and tests if it passes the unit tests.
If it passes the tests, the revision is merged into the "stable" branch.
If it doesn't pass the tests, it is rejected and the revision is not merged into the "stable" branch. The developer is forced to correct the revision and resubmit it.
Developers by default pull from the "stable" branch that should never be broken—in the sense that tests do not fail—and are more productive as they spend less time being blocked by broken builds. And the usefulness of such a system grows with the team size.
A few notes:
Git's pre-commit hooks and similar are not satisfactory in this case. The solution should be automatic and enforced on the server side for each commit.
Looking for a solution that has been implemented and thought out as far as possible, instead of writing a system like this myself from scratch.
I think this is more a build server feature that ties into a VCS such as Git. TeamCity does have support for this, but I've not tried it so I can't comment on how good it actually is.
http://www.jetbrains.com/teamcity/features/delayed_commit.html
The Hudson guys have been discussing it for a while, but I've yet to see it in a release.
http://wiki.hudson-ci.org/display/HUDSON/Designing+pre-tested+commit
We use gerrit and hudson. It is what android and Cyanogenmod use as well (along with many others).
Gerrit allows for code review and automatic building of every commit with automatic rejection of those that fail tests.
Hudson runs the tests.
Hudson: http://wiki.hudson-ci.org/display/HUDSON/Designing+pre-tested+commit
Gerrit: http://gerrit.googlecode.com
This system works well with the repo tool to have a large number of small repositories, this will reduce merge conflicts which have to be handled manually via a rebase.
Note: it is quite a bit of work to get up and running if you have a large existing code base, but totally worth it.
It's a really good idea. Bamboo supports this quite naturally. Several Bamboo customers as well as teams at Atlassian have this exact methodology setup and working to great effect. Bamboo has event listeners which can tell (without polling) when a commit is pushed to a 'test verfication' repo and then verify it by running the tests before pushing to the stable branch. www.atlassian.com/bamboo