In a pipeline we have 3 projects bind deployed, in the first stage we retrieve all the projects, And in subsequent stages we deploy and run test on each, that will total 4 stages, 1 for getting the sources and 1 each for deployment, test and other actions. Our change release are triggered by any commit done to any of the projects in the pipeline.
Normally this works ok but apparently AWS pipeline doesn't queue the change release and can trigger one after the other if a commit is done while a change release is running, so it will run in parallel in the same instance (ec2), and subsequently generate errors. Is there a way to configure a queue for the AWS pipeline release change? This discarding the option of manual approvals.
Thanks for the help in advance.
Based on your description it sounds like you have three projects in one pipeline with a stage for each project and one EC2 instance.
Why not create an independent pipeline for each project? Otherwise it sounds like you need mutual exclusion across the project stages. You could combine the three stages and let CodePipeline enforce one pipeline execution at a time occupying a stage.
I should probably mention based on your question that CodePipeline is intended for continuous delivery and it's desirable to have multiple changes moving through the pipeline at the same time. This is more obvious with deep pipelines (i.e. if it takes 3 days to fully release a change, you probably don't want to wait 3 days before a new change can start traversing the pipeline).
Related
I am trying to make a code pipeline which will build my branch when I make a pull request to the master branch in AWS. I have many developers working in my organisation and all the developers work on their own branch. I am not very familiar with ccreating lambda function. Hoping for a solution
You can dynamically create pipelines everytime a new pull-request has been created. Look for the CodeCommit Triggers (in the old CodePipeline UI), you need lambda for this.
Basically it works like this: Copy existing pipeline and update the the source branch.
It is not the best, but afaik the only way to do what you want.
I was there and would not recommend it for the following reasons:
I hit this limit of 20 in my region: "Maximum number of pipelines with change detection set to periodically checking for source changes" - but, you definitely want this feature ( https://docs.aws.amazon.com/codepipeline/latest/userguide/limits.html )
The branch-deleted trigger does not work correctly, so you can not delete the created pipeline, when the branch has been merged into master.
I would recommend you to use Github.com if you need a workflow as you described. Sorry for this.
I have recently implemented an approach that uses CodeBuild GitHub webhook support to run initial unit tests and build, and then publish the source repository and built artefacts as a zipped archive to S3.
You can then use the S3 archive as a source in CodePipeline, where you can then transition your PR artefacts and code through Integration testing, Staging deployments etc...
This is quite a powerful pattern, although one trap here is that if you have a lot of pull requests being created at a single time, you can get CodePipeline executions being superseded given only one execution can proceed through a given stage at a time (this is actually a really important property, especially if your integration tests run against shared resources and you don't want multiple instances of your application running data setup/teardown tasks at the same time). To overcome this, I publish an S3 notification to an SQS FIFO queue when CodeBuild publishes the S3 artifact, and then poll the queue, copying each artifact to a different S3 location that triggers CodePipeline, but only if there are are currently no executions waiting to execute after the first CodePipeline source stage.
We can very well have dynamic branching support with the following approach.
One of the limitations in AWS code-pipeline is that we have to specify branch names while creating the pipeline. We can however overcome this issue using the architecture shown below.
flow diagram
Create a Lambda function which takes the GitHub web-hook data as input, using boto3 integrate it with AWS pipeline(pull the pipeline and update), have an API gateway to make the call to the Lambda function as a rest call and at last create a web-hook to the GitHub repository.
External links:
https://aws.amazon.com/quickstart/architecture/git-to-s3-using-webhooks/
https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/codepipeline.html
Related thread: Dynamically change branches on AWS CodePipeline
I have been using AWS for more than a year now. Lately, I have been focusing on building a CI/CD Pipeline.
My pipeline has 4 stages:
Source (Github)
Testing (using CodeBuild)
Staging (deploys to Staging)
Manual Approval
Prod (deploys to Staging)
According to this AWS Doc, If no response is submitted within seven days, the action is marked as "Failed."
The Pipeline is relatively active (several deploys to staging per day) and what I found out is, that the Approvals "queue" and you have to Approve many times before the most recent changes get to Production.
Is there a way to set the expiration time of an Approval to less than 7 days?
There actually isn't a queue, but while an approval action is in-progress it holds the stage "lock" for that stage so that the change in that stage does not change underneath you while you run manual testing.
While that stage lock is held, there is a "slot" for the change waiting to be promoted into that stage when the stage lock is released. As newer changes pass the previous stage they will replace the change in the slot. Therefore, when you approve or reject a manual approval action, only the most recent pending change is promoted.
Rather than a manual approval, you might want to just disable the transition between staging and prod. Disabling the transition won't hold the lock in either stage, so when you enable it again the most recent change will be promoted.
Transitions are better for when you want to simply control when you deploy to prod, and manual approvals are better when you want to run some manual testing against a consistent version.
See this documentation on transitions: https://docs.aws.amazon.com/codepipeline/latest/userguide/transitions.html
I am using AWS CodePipeline in order to automatically check out code, build an application with CodeBuild and deploy the application to an ECS cluster for development. After that I inserted a manual step to approve deployment to the staging environment. This works well. However, when I run the pipeline again, there seems to be no way to approve the actions in one of previous executions. As far as I can see, I can only push the latest build artifact to staging (and later to production). This is surely not, what I would like to do. I could use more than one pipeline - one for each stage - for this, but than, what is the manual approval good for?
Currently updating a pipeline will end all in-flight executions at the end of their current action. This includes cancelling in-flight approvals.
After updating your pipeline you can click "Release Change" to have a fresh execution run through your pipeline and after that the changes will continue to be released as usual.
Unlike creating a pipeline, editing a pipeline does not rerun the most
recent revision through the pipeline. If you want to run the most
recent revision through a pipeline you've just edited, you must
manually rerun it. Otherwise, the edited pipeline will run the next
time you make a change to a source location configured in the source
stage of the pipeline. For information, see Start a Pipeline Manually
in AWS CodePipeline.
From the documentation here: https://docs.aws.amazon.com/codepipeline/latest/userguide/pipelines-edit.html
We have a CodePipeline set up to do a build, deploy to a QA ECS environment, then a manual approval step to deploy to Prod.
What gets confusing though, is when there are several builds running one after another. Several builds get deployed to QA in sequence, but then the Approval button seems to approve them one at a time, and it's not clear which build you're approving when you click on it.
What I would like to be able to do is to approve the latest build, in case
the earlier builds had issues that were fixed by the later builds. What would be the best way to accomplish that?
I had the same problem. Manual approvals are confusing since several pipeline executions can get queued and it's easy to lose track of things. I think we can blame this on CodePipeline's bad UX.
The workaround I settled with is to have two identical pipelines for the same project. They have the same source stage (same repo/branch) but different deploy stages (one deploys to QA, one deploys to prod). No more manual approval stages. The QA pipeline is set to auto-execute when changes in the source (repo/branch) are detected while the Prod pipeline needs to be manually released.
Basically, we replaced the Manual Approval with Manual Release. Manual release always releases the latest from source unlike manual approvals.
You should place the deploy and approval action in the same stage. This lets you approve exactly what you tested. Why? Because exactly one pipeline execution can be in a pipeline stage at any given time.
...approve the latest build, in case the earlier builds had issues
that were fixed by the later builds.
If you want to let later builds catch up, reject the earlier build that is waiting for approval.
One option if you don't want to have multiple pipelines is to by default disable stage transitions into your environments that required controlled releases.
When you are ready to deploy into an environment, you enable the stage transition to allow the most recent release from the previous stage to be processed and then disable the transitions again.
It's still a bit clunky, but reasonably effective once you get used to it. Having to reject each change that comes through becomes very slow and cumbersome to manage, so by disabling transitions you choose when to promote a release.
IMO, CodePipeline should have an option to automatically supersede executions if they are paused at the manual approval stage.
In the CodePipeline UI, you can see the history of Manual approvals in your pipelines' History. Click on History to see what's in progress (Manual Approvals that haven't timed out will always be in progress) and the source (git) short-sha that triggered it (if you need to narrow down to the relevant commit).
To know which Manual approval you're approving, in Pipeline view, click on View current revisions next to the Manual step (to get the Execution ID), then find the matching Execution ID in History (should be the oldest one).
Only way I found to get to the latest Approval is to hit reject n-1 times in the pipeline (where n is how many manual approvals are still in progress) until I only have 1 approval left (or until I find matching Execution ID).
Well, we can solve this problem as how you describe it with development, but it might also be a process glitch.
For example: If we have a development branch, a release branch (staging) and a master branch ( production ) we could easily solve this issue.
Development branch
Things we develop will be going through the development branch stage where we don't need the manual approval, as we don't want to check every changes. We have setup automated unit tests for that.
Release branch
This will deploy to the staging environment where we extensively test the software quality, also based on the regression tests on an acceptance chain with acceptance systems. This should prevent all the big issues before merging towards master branch. Next to that, we could also manually test the release branch on the staging environment. If this works, be happy and easily migrate towards master
Master branch
This will deploy to the production environment with a manual approval before the actual deployment is taking place, knowing for sure you will only push 1 change, being the merge from release to master, preventing the issues you've summarized in the ticket.
Another way is to develop a new AWS feature where you can uncheck or check a checkbox saying: always take the latest release, but that will not help adding value to the pipeline integration as things will be pushed without testing well enough.
Is there are way to make a CodeBuild project execute build tasks one at a time (max concurrency = 1)?
I know one of the selling points of CodeBuild is that you can run builds concurrently and I like that feature.
However, for this one specific project, I NEED to make sure only one CodeBuild build task for this project executes at a time. If there is an incoming "startBuild" request while a previous request is still running, I want it to be queued and wait until the previous build task if finished.
As additional info on the project, this project runs integration tests across our various APIs (serverless APIs and legacy APIs on EC2) and running those tests concurrently may cause the tests to fail due to their setup and teardown procedures.
I am from the AWS CodeBuild team. Thanks for your feedback. At this point the feature you requested isn't supported. We'll pass along the feedback to our product management team so they may consider adding it to our future roadmap.
However, you maybe able to implement something at your end by using CodeBuild's Build notifications feature. At a high level, you could listen to the CloudWatch Events sent by CodeBuild to find when a build completes, and at that time "release" a new build from a queue that you maintain at your end.
Update (2021-02-17)
AWS CodeBuild just added the ability to define a maximum number of concurrent builds per project.
Reference: https://docs.aws.amazon.com/codebuild/latest/userguide/create-project-console.html#create-project-console-project-config