I am using AWS CodePipeline in order to automatically check out code, build an application with CodeBuild and deploy the application to an ECS cluster for development. After that I inserted a manual step to approve deployment to the staging environment. This works well. However, when I run the pipeline again, there seems to be no way to approve the actions in one of previous executions. As far as I can see, I can only push the latest build artifact to staging (and later to production). This is surely not, what I would like to do. I could use more than one pipeline - one for each stage - for this, but than, what is the manual approval good for?
Currently updating a pipeline will end all in-flight executions at the end of their current action. This includes cancelling in-flight approvals.
After updating your pipeline you can click "Release Change" to have a fresh execution run through your pipeline and after that the changes will continue to be released as usual.
Unlike creating a pipeline, editing a pipeline does not rerun the most
recent revision through the pipeline. If you want to run the most
recent revision through a pipeline you've just edited, you must
manually rerun it. Otherwise, the edited pipeline will run the next
time you make a change to a source location configured in the source
stage of the pipeline. For information, see Start a Pipeline Manually
in AWS CodePipeline.
From the documentation here: https://docs.aws.amazon.com/codepipeline/latest/userguide/pipelines-edit.html
Related
I am not able to find a way to stop the auto triggering of the pipeline whenever I push code to bitbucket.
My assumption is that you want more control over when your pipeline does certain things.
Rather than achieving this through stopping the pipeline from getting triggered, I'd recommend using either stage transitions or manual approvals to achieve this control inside the pipeline.
Stage transitions are better when you want to "turn off" a pipeline and have the latest thing run through when you turn it back on.
Manual approvals are better when you want the version to be locked while waiting for approval so you can run tests without worrying that the version will change.
You mentioned in your comment that you wanted to only run your pipeline at certain times, so a way you could do that is to enable and disable the stage transition after source on a schedule.
https://docs.aws.amazon.com/codepipeline/latest/userguide/transitions.html
https://docs.aws.amazon.com/codepipeline/latest/userguide/approvals.html
You can disable DetectChanges parameter on your Source action as explained here. Extract with the relevant context:
DetectChanges: Controls automatically starting your pipeline when a new commit is made on the configured repository and branch. If unspecified, the default value is true, and the field does not display by default.
This works on Bitbucket, GitHub, and GitHub Enterprise Server actions. I have a CloudFormation template configured with this option and works. Not sure about the same option on AWS console, because I saw that some configurations are only available from CloudFormation or aws cli. As you can read "this field does not display by default".
My team has been running into issues with our CodePipeline where features were pushed out into production when they shouldn't have been due to our Docker image patching. A little background on our architecture: Our pipeline has two sources, one for the source code and one for the Docker image builder. Docker builds via CodeBuild and is deployed to dev, test, and then prod environments with manual approval steps in between.
Our Docker image receives monthly patching which triggers the pipeline to execute and is what caused the features to be pushed out. We redesigned our git branching strategy so that our master branch will only contain stable releases, but I could still see this issue potentially occurring again if a specific release date is specified. Is there a way to push out the image patching without pushing out the latest commit?
Can CodePipeline Use a Specific Commit
This is an often requested feature but unfortunately CodePipeline will always bring the latest commit from the selected branch in the Source action.
CodePipeline tied to a single git branch is more of a feature of CodePipeline as the design is more inclined towards Trunk based development [0]. Also, as per the designers of this service, CodePipeline is designed for post-merge/release validation. That is, once your change is ready to be released to production and is merged into your master/main branch, CodePipeline takes over and automatically tests and releases the final merged set of changes. CodePipeline has a lot of features like stage locking, superseding versions, etc. which don't work well for the case where you want to test a change in isolation before it's merged (e.g. feature branch testing or pull request testing.) Therefore there currently isn't a recommended way to do this in CodePipeline.
[0] https://trunkbaseddevelopment.com/
Having said that, there is a way to hack this with S3 Source action in pipeline instead of GitHub/CodeCommit source action. Essentially your pipeline's S3 source action is tied to S3 bucket/key. You can then upload a zip of any specific commit to this S3 bucket/key and trigger the pipeline.
In a pipeline we have 3 projects bind deployed, in the first stage we retrieve all the projects, And in subsequent stages we deploy and run test on each, that will total 4 stages, 1 for getting the sources and 1 each for deployment, test and other actions. Our change release are triggered by any commit done to any of the projects in the pipeline.
Normally this works ok but apparently AWS pipeline doesn't queue the change release and can trigger one after the other if a commit is done while a change release is running, so it will run in parallel in the same instance (ec2), and subsequently generate errors. Is there a way to configure a queue for the AWS pipeline release change? This discarding the option of manual approvals.
Thanks for the help in advance.
Based on your description it sounds like you have three projects in one pipeline with a stage for each project and one EC2 instance.
Why not create an independent pipeline for each project? Otherwise it sounds like you need mutual exclusion across the project stages. You could combine the three stages and let CodePipeline enforce one pipeline execution at a time occupying a stage.
I should probably mention based on your question that CodePipeline is intended for continuous delivery and it's desirable to have multiple changes moving through the pipeline at the same time. This is more obvious with deep pipelines (i.e. if it takes 3 days to fully release a change, you probably don't want to wait 3 days before a new change can start traversing the pipeline).
I have been using AWS for more than a year now. Lately, I have been focusing on building a CI/CD Pipeline.
My pipeline has 4 stages:
Source (Github)
Testing (using CodeBuild)
Staging (deploys to Staging)
Manual Approval
Prod (deploys to Staging)
According to this AWS Doc, If no response is submitted within seven days, the action is marked as "Failed."
The Pipeline is relatively active (several deploys to staging per day) and what I found out is, that the Approvals "queue" and you have to Approve many times before the most recent changes get to Production.
Is there a way to set the expiration time of an Approval to less than 7 days?
There actually isn't a queue, but while an approval action is in-progress it holds the stage "lock" for that stage so that the change in that stage does not change underneath you while you run manual testing.
While that stage lock is held, there is a "slot" for the change waiting to be promoted into that stage when the stage lock is released. As newer changes pass the previous stage they will replace the change in the slot. Therefore, when you approve or reject a manual approval action, only the most recent pending change is promoted.
Rather than a manual approval, you might want to just disable the transition between staging and prod. Disabling the transition won't hold the lock in either stage, so when you enable it again the most recent change will be promoted.
Transitions are better for when you want to simply control when you deploy to prod, and manual approvals are better when you want to run some manual testing against a consistent version.
See this documentation on transitions: https://docs.aws.amazon.com/codepipeline/latest/userguide/transitions.html
We have a CodePipeline set up to do a build, deploy to a QA ECS environment, then a manual approval step to deploy to Prod.
What gets confusing though, is when there are several builds running one after another. Several builds get deployed to QA in sequence, but then the Approval button seems to approve them one at a time, and it's not clear which build you're approving when you click on it.
What I would like to be able to do is to approve the latest build, in case
the earlier builds had issues that were fixed by the later builds. What would be the best way to accomplish that?
I had the same problem. Manual approvals are confusing since several pipeline executions can get queued and it's easy to lose track of things. I think we can blame this on CodePipeline's bad UX.
The workaround I settled with is to have two identical pipelines for the same project. They have the same source stage (same repo/branch) but different deploy stages (one deploys to QA, one deploys to prod). No more manual approval stages. The QA pipeline is set to auto-execute when changes in the source (repo/branch) are detected while the Prod pipeline needs to be manually released.
Basically, we replaced the Manual Approval with Manual Release. Manual release always releases the latest from source unlike manual approvals.
You should place the deploy and approval action in the same stage. This lets you approve exactly what you tested. Why? Because exactly one pipeline execution can be in a pipeline stage at any given time.
...approve the latest build, in case the earlier builds had issues
that were fixed by the later builds.
If you want to let later builds catch up, reject the earlier build that is waiting for approval.
One option if you don't want to have multiple pipelines is to by default disable stage transitions into your environments that required controlled releases.
When you are ready to deploy into an environment, you enable the stage transition to allow the most recent release from the previous stage to be processed and then disable the transitions again.
It's still a bit clunky, but reasonably effective once you get used to it. Having to reject each change that comes through becomes very slow and cumbersome to manage, so by disabling transitions you choose when to promote a release.
IMO, CodePipeline should have an option to automatically supersede executions if they are paused at the manual approval stage.
In the CodePipeline UI, you can see the history of Manual approvals in your pipelines' History. Click on History to see what's in progress (Manual Approvals that haven't timed out will always be in progress) and the source (git) short-sha that triggered it (if you need to narrow down to the relevant commit).
To know which Manual approval you're approving, in Pipeline view, click on View current revisions next to the Manual step (to get the Execution ID), then find the matching Execution ID in History (should be the oldest one).
Only way I found to get to the latest Approval is to hit reject n-1 times in the pipeline (where n is how many manual approvals are still in progress) until I only have 1 approval left (or until I find matching Execution ID).
Well, we can solve this problem as how you describe it with development, but it might also be a process glitch.
For example: If we have a development branch, a release branch (staging) and a master branch ( production ) we could easily solve this issue.
Development branch
Things we develop will be going through the development branch stage where we don't need the manual approval, as we don't want to check every changes. We have setup automated unit tests for that.
Release branch
This will deploy to the staging environment where we extensively test the software quality, also based on the regression tests on an acceptance chain with acceptance systems. This should prevent all the big issues before merging towards master branch. Next to that, we could also manually test the release branch on the staging environment. If this works, be happy and easily migrate towards master
Master branch
This will deploy to the production environment with a manual approval before the actual deployment is taking place, knowing for sure you will only push 1 change, being the merge from release to master, preventing the issues you've summarized in the ticket.
Another way is to develop a new AWS feature where you can uncheck or check a checkbox saying: always take the latest release, but that will not help adding value to the pipeline integration as things will be pushed without testing well enough.