Is there anyway to execute action in a pipeline stage based on a condition.
I mean for example, when the pipeline releases, it sources the source files at the source stage, and in the build, deploy or other stages are executed if a file exists in the artifact or is not empty...
In fact I would to execute the whole pipeline stages only when my condition is met...
Till I didn't see how to well achieve this.
What I think now to play on approval feature, I don't what you think and if one here has already had the same requirement.
The simplest way to achieve this is likely to add a Test stage to your pipeline. You test can simply check for the desired file in the build artifact, and simply fail the check if the file isn't there, which would halt the pipeline.
Usings tests with AWS CodePipeline
Related
I have a number of services in a single GitHub repository, each service has its own CodePipeline on AWS managed through Terraform. Instead of triggering all of the pipelines on commit, I'd like to know how I can trigger each service's pipeline if its directory had any changes on commit, without having to split the services each into its own repository.
I don't think that there's a conditional source stage support per folder at code pipeline as we speak. Just finished checking this documentation about sources in CodePipeline. It does not seem to contain a folder-level filtering.
You could try this CDK-based template solution which showcases a mono-repository, which is composed of multiple services, have different CI/CD pipelines for each service. The solution detects which top level directory the modification happened and triggers the AWS CodePipeline configured to that directory.
This is sad but they might add it in the future. I've also wanted Quality gates, images from readme files in code-commit but these features seem too hard to implement haha.
It ended up being simpler than I had anticipated, there are github actions that do exactly what I needed.
This action checks whether a path had a change, and this action triggers a specific pipeline.
I am not able to find a way to stop the auto triggering of the pipeline whenever I push code to bitbucket.
My assumption is that you want more control over when your pipeline does certain things.
Rather than achieving this through stopping the pipeline from getting triggered, I'd recommend using either stage transitions or manual approvals to achieve this control inside the pipeline.
Stage transitions are better when you want to "turn off" a pipeline and have the latest thing run through when you turn it back on.
Manual approvals are better when you want the version to be locked while waiting for approval so you can run tests without worrying that the version will change.
You mentioned in your comment that you wanted to only run your pipeline at certain times, so a way you could do that is to enable and disable the stage transition after source on a schedule.
https://docs.aws.amazon.com/codepipeline/latest/userguide/transitions.html
https://docs.aws.amazon.com/codepipeline/latest/userguide/approvals.html
You can disable DetectChanges parameter on your Source action as explained here. Extract with the relevant context:
DetectChanges: Controls automatically starting your pipeline when a new commit is made on the configured repository and branch. If unspecified, the default value is true, and the field does not display by default.
This works on Bitbucket, GitHub, and GitHub Enterprise Server actions. I have a CloudFormation template configured with this option and works. Not sure about the same option on AWS console, because I saw that some configurations are only available from CloudFormation or aws cli. As you can read "this field does not display by default".
I am trying to make a code pipeline which will build my branch when I make a pull request to the master branch in AWS. I have many developers working in my organisation and all the developers work on their own branch. I am not very familiar with ccreating lambda function. Hoping for a solution
You can dynamically create pipelines everytime a new pull-request has been created. Look for the CodeCommit Triggers (in the old CodePipeline UI), you need lambda for this.
Basically it works like this: Copy existing pipeline and update the the source branch.
It is not the best, but afaik the only way to do what you want.
I was there and would not recommend it for the following reasons:
I hit this limit of 20 in my region: "Maximum number of pipelines with change detection set to periodically checking for source changes" - but, you definitely want this feature ( https://docs.aws.amazon.com/codepipeline/latest/userguide/limits.html )
The branch-deleted trigger does not work correctly, so you can not delete the created pipeline, when the branch has been merged into master.
I would recommend you to use Github.com if you need a workflow as you described. Sorry for this.
I have recently implemented an approach that uses CodeBuild GitHub webhook support to run initial unit tests and build, and then publish the source repository and built artefacts as a zipped archive to S3.
You can then use the S3 archive as a source in CodePipeline, where you can then transition your PR artefacts and code through Integration testing, Staging deployments etc...
This is quite a powerful pattern, although one trap here is that if you have a lot of pull requests being created at a single time, you can get CodePipeline executions being superseded given only one execution can proceed through a given stage at a time (this is actually a really important property, especially if your integration tests run against shared resources and you don't want multiple instances of your application running data setup/teardown tasks at the same time). To overcome this, I publish an S3 notification to an SQS FIFO queue when CodeBuild publishes the S3 artifact, and then poll the queue, copying each artifact to a different S3 location that triggers CodePipeline, but only if there are are currently no executions waiting to execute after the first CodePipeline source stage.
We can very well have dynamic branching support with the following approach.
One of the limitations in AWS code-pipeline is that we have to specify branch names while creating the pipeline. We can however overcome this issue using the architecture shown below.
flow diagram
Create a Lambda function which takes the GitHub web-hook data as input, using boto3 integrate it with AWS pipeline(pull the pipeline and update), have an API gateway to make the call to the Lambda function as a rest call and at last create a web-hook to the GitHub repository.
External links:
https://aws.amazon.com/quickstart/architecture/git-to-s3-using-webhooks/
https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/codepipeline.html
Related thread: Dynamically change branches on AWS CodePipeline
In a pipeline we have 3 projects bind deployed, in the first stage we retrieve all the projects, And in subsequent stages we deploy and run test on each, that will total 4 stages, 1 for getting the sources and 1 each for deployment, test and other actions. Our change release are triggered by any commit done to any of the projects in the pipeline.
Normally this works ok but apparently AWS pipeline doesn't queue the change release and can trigger one after the other if a commit is done while a change release is running, so it will run in parallel in the same instance (ec2), and subsequently generate errors. Is there a way to configure a queue for the AWS pipeline release change? This discarding the option of manual approvals.
Thanks for the help in advance.
Based on your description it sounds like you have three projects in one pipeline with a stage for each project and one EC2 instance.
Why not create an independent pipeline for each project? Otherwise it sounds like you need mutual exclusion across the project stages. You could combine the three stages and let CodePipeline enforce one pipeline execution at a time occupying a stage.
I should probably mention based on your question that CodePipeline is intended for continuous delivery and it's desirable to have multiple changes moving through the pipeline at the same time. This is more obvious with deep pipelines (i.e. if it takes 3 days to fully release a change, you probably don't want to wait 3 days before a new change can start traversing the pipeline).
I have a specific case which I'm not sure if it's possible with AWS CodePipeline, and I didn't find any information about it in the documentation and event by googling....
So I would like to know if I can set two sources in a pipeline (it could be in the same stage or different stages).
Here is my use case :
I would like my pipeline to start when a file (a specific object) is modified in my s3 bucket
When this file changes and the pipeline is triggered, I would like to clone a codecommit repository and then process the build and other stages...
In the other hand when there is a commit on the master branch of my codecommit repository, I would like the pipeline to start and build my sources.
So The pipeline should be triggered either when the change comes from s3 or codecommit
I don't want to version the s3 file in my codecommit repository because it should be encrypted and used by others teams than dev team working with the git repository
And any time my pipeline starts either if it's from the s3 bucket change or the codecommit push, I should source the commit from the repository for build purposes...
I don't know if my objectives specifications are clear, if yes is it possible to use two source actions in a pipeline as described above and how to achieve this?
Thank you in advance.
Cheers,
Eugène NG
Yes. It is possible to have two sources for an AWS CodePipeline. Or many for that matter. The two sources have to be in your first stage.
Then in your build phase properties, you need to tell it that you are expecting two sources.
Then tell the build project which is your primary source. This is going to be the one that you want your build project to execute the codebuild.
From your buildspec or from any scripts you call, you can then access the source directories by referencing:
$CODEBUILD_SRC_DIR_SourceOutput1
$CODEBUILD_SRC_DIR_SourceOutput2
Just replace SourceOutputX above with what you call your output from the source stage.
I found the following link with more information:
https://docs.aws.amazon.com/codebuild/latest/userguide/sample-multi-in-out.html
Yes, CodePipeline allows multiple source actions in a single pipeline. A change in either source will trigger a pipeline execution. The thing to know is that every pipeline execution will pull the latest source for both actions (not just the one with a change that triggered the pipeline execution).