I would like to conditionally execute certain stage in AWS Codepipeline depending on that if I put certain file on repo location. So, if I put "some_file.txt" on certain location in repo, I want for Codepipeline to check existence of this file and if it's there continue further to deploy code to production, otherwise stop on that stage.
With this I would like to avoid manual approval action and control release process with committing a file. Is this possible and what would be best practice?
I think you could create a lambda action for that:
Invoke an AWS Lambda function in a pipeline in CodePipeline
The lambda function can access the input artifact, and check if your file of interest is there or not.
Depending on the outcome of the check, the function with either put_job_success_result or put_job_failure_resul to continue or stop the pipeline.
you can use the spec file to check if there's the needed file present. If not, then you can execute a "stop-pipeline-execution" https://docs.aws.amazon.com/cli/latest/reference/codepipeline/stop-pipeline-execution.html
command. The required args can be fetched from the env vars and one more thing to note is to give that stage of yours adequate permission(s) to be able to execute the command.
Related
I'm exploring how to do CI/CD using cdk-pipelines, but while the setup part works, I don't understand the control part. All the examples are really simple inline code Lambda Functions.
How do I "release a change" of a codepipeline.Pipeline inside a cdk_pipelines?
const cdk_pipeline = new pipelines.CodePipeline(...)
cdk_pipeline.addStage(new BuildImageTestStage(...))
// build-image-test-stage.ts
new BuildImageTestStack(...)
// build-image-test-stack.ts
const pipeline = new codepipeline.Pipeline(...)
pipeline.addStage(...CodeStarConnectionsSourceAction...)
pipeline.addStage(...CodeBuildAction...)
It sets up the pipeline just fine, but it doesn't fire off the actual codepipeline itself.
Options that I see:
codepipeline.Pipelines(triggerOnPush: true) - doesn't work because I want the cloud formation to run first THEN build+test. Trigger on push effectively runs parallel.
Completely separate setup from code deployment by using codepipeline_actions.CloudFormationCreateUpdateStackAction
Unsure option:
cdk_pipeline.addStage(..., { post: [] }) - CDK Pipeline post to trigger a code deploy, but I'm unsure on how to "wait" for it to finish.
Separately, I wish cdk-pipelines were named differently than codepipeline. It's just so hard to search.
[Edit after exchanges in the comments]: pipeline.CodePipeline is a wrapper for a codepipeline.Pipeline. It creates a new codepipeline.Pipeline under the hood in its construtor. The const pipeline = new codepipeline.Pipeline(...) line is adding a *second* codepipeline.Pipeline. You almost certainly don't want two. You have a few options:
Option 1: Pass codePipeline: pipeline in the cdk_pipeline constructor. cdk_pipeline will use your codepipeline.Pipeline instead of creating its own.
Option 2: Get rid of your pipeline instance. Refactor your build/test actions as pre or post Steps in the options for cdk_pipline.addStage or addWave. A CodeBuildStep will add a build step, for example. You can add arbitrary CodePipeline actions as steps, too.
AWS pipeline executions always run from beginning to end, each step "waiting" for the next to finish (except if stopped by an error). A CDK pipeline.CodePipeline always starts with the source action defined in its synth property.
The addStage method adds a serial stage to the pipeline. To add parallel stages, use addWave.
A CDK pipeline.CodePipeline makes it easy to create a pipeline that deploys CDK apps. It abstracts away the details of the more general-purpose codepipeline.Pipeline. If your use case is something other than building/testing/deploying CDK apps, though, you may be better off working directly with the latter, as the docs say.
I want to run a CFN stack from code pipeline, but the paramaters are scatered in different outputs from previous action. The only way I can think of is to run a Lambda action which load all of them and create a new one output out of it.
Is there something more straight forward?
I'm new to AWS and transitioning from Azure. I would like to create a pipeline in CodePipeline which asks the user for input (for example: the user needs to input a value for the variable "hello"), and uses that input to run a CodeBuild project. In Azure DevOps this was quite easy to define in the pipeline YML specification, but I can't seem to find a way to easily do this in AWS, or am I missing something?
AWS CodePipeline not supporting this feature currently. What you can do is, pass this parameter in your commit message (if pipeline trigger on commits to branches) or in your Git tag (if pipeline trigger on git tag push).
example:
commit message: my commit message [my_var]
git tag: my_var-1.0.0
Then in your buildspec.yml file collect the commit message or tag and check whether it contains your required parameters. If so execute the next commands otherwise exit the script.
I'm currently struggling to configure automated input to my AWS StepFunctions state machine. Basically, I am trying to set up a state machine that is notified whenever an object create event takes place in a certain S3 bucket. When that happens, the input is passed to a choice state which checks the file size. If the file is small enough, it invokes a Lambda function to process the file contents. If the file is too large, it invokes a Lambda to split up the file into files of manageable size, and then invokes the other Lambda to process the contents of those files. The problem with this is that I cannot figure out a way to pass the file size in as input to the state machine.
I am generally aware of how input is passed to StepFunctions, and I know that S3 Lambda triggers contain file size as a parameter, but I still haven't been able to figure out a practical way of passing file size as an input parameter to a StepFunctions state machine.
I would greatly appreciate any help on this issue and am happy to clarify or answer any questions that you have to the best of my ability. Thank you!
Currently S3 events can't triggers Step Function directly, so one option would be to create a S3 event that triggers a lambda. The lambda works as a proxy and passes the file info to the step function and kicks it off, also you can select data you want and only pass selective data to Step Functions.
The other option is to configure a state machine as a target for a CloudWatch Events rule. This will start an execution when files are added to an Amazon S3 bucket.
The first option is more flexible.
I'm using CDK to set up a CI/CD Pipeline. I have currently a code build from a git into the pipeline. There are then two builds - one that pulls out code for a lambda and builds an artifact for it, and a second that issues the cdk synth to construct the lambda framework (including a nested bucket and dynamo).
Then it heads to a deploy stage, but fails because it can't find the parameters for the location of the lambda code
ive been using this example: https://docs.aws.amazon.com/cdk/latest/guide/codepipeline_example.html
the only differences from this example are that I'm using python for all of it and due to known future needs, the lamdba's are are in a parallel directory from the stack code
|-Lambdas
|--Lambda1
|---Lambda1Code
|--Lambda2
|---Lambda2Code
|-CDKStacks
|--LambdaCreationStack
|--PipelineCreationStack
|--app.py
Everything runs up until deploy where it fails with the error "The following CloudFormation Parameters are missing a value:" and then lists the BucketName and ObjectKey
I assigned those as overrides as per the above link:
admin_permissions=True,
parameter_overrides=dict(
lambda_code.assign(
bucket_name=lambda_location.bucket_name,
object_key=lambda_location.object_key,
object_version=lambda_location.object_version
)
),
as part of the pipeline actions CloudFormationCreateUpdateStackAction, and passed the code just like in the example from lambda stack to the pipeline stack. But every time the lambda stack is attempted to deploy the parameters for the location of the code 'do not exist'
I've tried overriding the parameters, but being in the pipeline and dynamically created I am hesitant to follow further (and my attempts didnt work anyways). I've tried a bunch of different stack/nested stack/single stack configurations but haven't had a Successs yet.
thoughts?
This basically boils down to CodeUri in the Cloudformation template will automatically append the s3 bucket if your CodeUri starts with ./
So you have 2 options.
In your pipeline output your artifact as normal, just do the whole repo from the codebuild into the code deploy. Your code deoploy can pick up the artifact naturally and will automatically append the S3 url to that
if you're using Python however, you MUST be aware that starting from a lambda directory deeper in the tree will mean that the python Imports expect that directory to be a root directory - meaning if you were in Lambdas/Lambda1 and wanted to import a file that existed in the Lambda1 directory, in order for it to work on AWS Lambda you would need to have the import be just the file name, ignoring the rest of the path.
This means that coding can be difficult, and running unit tests can be difficult as well. You'll want to add all the individual lambda folders (and their paths) from root to the PYTHONPATH env variable of your codebuild instance so the unit tests know where to do so (and add a .env file to your IDE as well to handle this in your local)
You use CDK and you cdk synth the stack you want to deploy. This creates a cdk.out folder with a bunch of asset zips in it plus the stack template (a json). you adjust your artifact output in the codebuild to output the cdk.out folder, and the asset zips are automatically (thanks to cdk) subbed into the codeUri locations in the also automatically synthed template. Once you know what the templates name is its easy to set the CodeDeploy to look for that template name and it will find the asset zips individually for each lambda.