If you create a CodePipeline via CloudFormation. It starts it automatically, that can be a problem because the pipeline can rewrite the same stack...
Is there any way to disable this behaviour?
Thanks.
Had same issue, I don't want a pipeline launch on pipeline creation (which is the default beahviour).
Best solution I fount is :
Create an EventBridge rule which catch the pipelineExecution on
pipeline creation
Stop the pipeline execution from the lambda triggered
Rule looks like this :
{
"source": ["aws.codepipeline"],
"detail-type": ["CodePipeline Pipeline Execution State Change"],
"detail": {
"state": ["STARTED"],
"execution-trigger": {
"trigger-type": ["CreatePipeline"]
}
}
}
It works fine
Sadly, there seem to be no way of this this. Docs clearly states that a newly created pipeline immediately starts running:
Now that you've created your pipeline, you can view it in the console. The pipeline starts to run after you create it.
The initial run will always happen. Subsequent runs depend on your source action. For example, if you use CodeCommit as your source, you can disable CloudWatch Event that triggers the pipeline.
Thus if you want to use CodePipeline in your project, you have to design it so that it does not causes any issues due to immediate start.
You can disable the Event rule from automatically starting your pipeline.
Go to Amazon EventBridge -> Rules and disable the rule that notifies the CodePipeline.
Further to Marcin's comment, it would seem there are 2 approaches you can take which would limit the run of the pipeline.
Create a disabled StageTransition or Manual Approval stage directly after the Source stage. This would prevent the pipeline executing any other action aside from getting the source which would have no impact or capability to re-write anything.
Alternatively if your source stage is from a repository, you can opt to handle the pipeline triggers yourself by disabling the PollForSourceChanges parameter in your cloudformation template.
Pipeline:
Type: AWS::CodePipeline::Pipeline
Properties:
Name: *NAME*
RoleArn: *IAMROLE*
Stages:
- Name: Source
Actions:
- Name: CodeCommitSourceAction
RunOrder: 1
ActionTypeId:
Category: Source
Provider: CodeCommit
Owner: AWS
Version: '1'
OutputArtifacts:
- Name: Source
Configuration:
RepositoryName: *REPOSITORYNAME*
BranchName: *BRANCH*
PollForSourceChanges: "false" #prevents codepipeline polling repository for changes.
So the correct answer is...
Commit your code before you deploy for the first time
Deploy only the pipeline
Let Code Pipeline do its thing
99% of cases it will finish sooner than your machine.
I am trying to create a bucket using Deployment manager but when I want to create the deployment, I get the following error:
ERROR: (gcloud.deployment-manager.deployments.create) Error in Operation [operation-1525606425901-56b87ed1537c9-70ca4aca-72406eee]: errors:
- code: RESOURCE_ERROR
location: /deployments/posts/resources/posts
message: '{"ResourceType":"storage.v1.bucket","ResourceErrorCode":"403","ResourceErrorMessage":{"code":403,"errors":[{"domain":"global","message":"myprojectid#cloudservices.gserviceaccount.com
does not have storage.buckets.get access to posts.","reason":"forbidden"}],"message":"myprojectid#cloudservices.gserviceaccount.com
does not have storage.buckets.get access to posts.","statusMessage":"Forbidden","requestPath":"https://www.googleapis.com/storage/v1/b/posts","httpMethod":"GET","suggestion":"Consider
granting permissions to myprojectid#cloudservices.gserviceaccount.com"}}'
If I understand it correctly, the deployment manager uses a service account (as described in the message) to actually create all my resources. I've checked IAM and made sure that the service role (myprojectid#cloudservices.gserviceaccount.com) does have access as "Editor" and even added "Storage Admin" (which includes storage.buckets.get) to be extra sure. However, I still get the same error message.
Am I assigning the permissions to the wrong IAM user / what am I doing wrong?
command used:
gcloud deployment-manager deployments create posts --config posts.yml
my deployment template:
bucket.jinja
resources:
- name: {{ properties['name'] }}
type: storage.v1.bucket
properties:
name: {{ properties['name'] }}
location: europe-west1
lifecycle:
rule:
- action:
type: Delete
condition:
age: 30
isLive: true
labels:
datatype: {{ properties['datatype'] }}
storageClass: REGIONAL
posts.yml
imports:
- path: bucket.jinja
resources:
- name: posts
type: bucket.jinja
properties:
name: posts
datatype: posts
I tested your code with success and I believe that the issue was that you were trying to create/update a bucket own by a different user belonging to a different project upon which your service account has no power.
Therefore please try to redeploy changing the name that likely is a unique one and let me know if this solves the issue. This can be an issue in some scenario because either you choose name pretty long or there is the risk that is already taken.
Notice that you have to change the name of the bucket since it has to be unique across all the project of all the users.
This could seem an excessive requirement, but it makes possible to create static website or to refer to file with the standard URL:
https://storage.googleapis.com/nomebucket/folder/nomefile
From the trace error I believe that this is the issue, you are trying to create a bucket that does not exist and you do not own.
Notice that if you remove the permissions from the service account you do not receive the message telling you that the service account does not have any power on the bucket:
xxx#cloudservices.gserviceaccount.com does not have storage.buckets.get access to posts.
But instead a message pointing you that the service account has no power on the project:
Service account xxx#cloudservices.gserviceaccount.com is not authorized
to take actions for project xxx. Please add xxx#cloudservices.gserviceaccount.com
as an editor under project xxx using Google Developers Console
Notice that if you try to create a bucket you already own there is no issue.
$ gcloud deployment-manager deployments create posts22 --config posts.yml
The fingerprint of the deployment is xxx==
Waiting for create [operation-xxx-xxx-xxx-xxx]...done.
Create operation operation-xxx-xxx-xxx-xxx completed successfully.
NAME TYPE STATE ERRORS INTENT
nomebuckettest4536 storage.v1.bucket COMPLETED []
I am using CodePipeline with CodeCommit. Builds are triggered automatically with push to master branch. In CodePipeline console it is clearly visible that i am receiving commit id but i need to get it in the build environment so i can add them as a tag to the ECS image when i build it. Is there a way to get in in build environment.
You can use the CODEBUILD_RESOLVED_SOURCE_VERSION environment variable to retrieve the commit hash displayed in CodePipeline at build time.
Adding an answer that explains how to achieve this in CloudFormation, as it took me a while to figure it out. You need to define your stage as:
Name: MyStageName
Actions:
-
Name: StageName
InputArtifacts:
- Name: InputArtifact
ActionTypeId:
Category: Build
Owner: AWS
Version: '1'
Provider: CodeBuild
OutputArtifacts:
- Name: OutputArtifact
Configuration:
ProjectName: !Ref MyBuildProject
EnvironmentVariables:
'[{"name":"COMMIT_ID","value":"#{SourceVariables.CommitId}","type":"PLAINTEXT"}]'
In your actions you need to have this kind of syntax. Note that the EnvironmentVariables property of a CodePipeline stage is different from a AWS::CodeBuild::Project property. If you were to add #{SourceVariables.CommitId} as an env variable there, it wouldn't be resolved properly.
CodePipeline now also allows you to configure your pipeline with variables that are generated at execution time. In this example your CodeCommit action will produce a variable called CommitId that you can pass into a CodeBuild environment variable via the CodeBuild action configuration.
Here is a conceptual overview of the feature: https://docs.aws.amazon.com/codepipeline/latest/userguide/reference-variables.html
For an example walk through of passing the commit id into your build action you can go here:
https://docs.aws.amazon.com/codepipeline/latest/userguide/actions-variables.html
It would also be worth considering tagging the image with the CodePipeline execution id instead of the commit id, that way it prevents future builds with the same commit from overwriting the image. Using the CodePipeline execution id is also shown in the example above.
Is this what you're looking for?
http://docs.aws.amazon.com/codepipeline/latest/userguide/monitoring-source-revisions-view.html#monitoring-source-revisions-view-cli
Most (if not all) of the language SDKs have this API built in also.
Additionally to #Bar's answer: just adding EnvironmentVariables is not enough, you need to set Namespace also.
For example:
pipeBackEnd:
Type: AWS::CodePipeline::Pipeline
Properties:
...
Stages:
- Name: GitSource
Actions:
- Name: CodeSource
ActionTypeId:
Category: Source
...
Configuration: (...)
Namespace: SourceVariables # <<< === HERE, in Source
- Name: Deploy
Actions:
- Name: BackEnd-Deploy
ActionTypeId:
Category: Build
Provider: CodeBuild (...)
Configuration:
ProjectName: !Ref CodeBuildBackEnd
EnvironmentVariables: '[{"name":"BranchName","value":"#{SourceVariables.BranchName}","type":"PLAINTEXT"},{"name":"CommitMessage","value":"#{SourceVariables.CommitMessage}","type":"PLAINTEXT"}]'
Also, it may be useful: list of CodePipeline variables
When I run CloudFormation deploy using a template with API Gateway resources, the first time I run it, it creates and deploys to stages. The subsequent times I run it, it updates the resources but doesn't deploy to stages.
Is that behaviour as intended? If yes, how'd I get it to deploy to stages whenever it updates?
(Terraform mentions a similar issue: https://github.com/hashicorp/terraform/issues/6613)
Seems like there is no way to easily create a new Deployment whenever one of your Cloudformation Resources changes.
One way to work around that would be to use a Lambda-backed Custom Resource (see http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/template-custom-resources.html).
The Lambda should create the new Deployment, only if one of your Resources has been updated. To determine if one of your Resources has been updated,
you will probably have to implement custom logic around this API call: http://docs.aws.amazon.com/AWSCloudFormation/latest/APIReference/API_DescribeStackEvents.html
In order to trigger updates on your Custom Resource, I suggest you supply a Cloudformation Parameter that will be used to force an update of your Custom Resource (e.g. the current time, or a version number).
Note that you will have to add a DependsOn clause to your Custom Resource that will include all Resources relevant to your API. Otherwise, your deployment might be created before all your API Resources are updated.
Hope this helps.
When your template specifies a deployment, CloudFormation will create that deployment only if it doesn't already exist. When you attempt to run it again, it observes that the deployment still exists so it won't recreate it, thus no deployment. You need a new resource id for the deployment so that it will create a new deployment. Read this for more information: https://currentlyunnamed-theclassic.blogspot.com/2018/12/mastering-cloudformation-for-api.html
CloudFormation in Amazon's words is:
AWS CloudFormation takes care of provisioning and configuring those resources for you
http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/Welcome.html
Redeployment of APIs is not a provisioning task... It is a promotion activity which is part of a stage in your software release process.
AWS CodePipeline is a continuous delivery service you can use to model, visualize, and automate the steps required to release your software.
http://docs.aws.amazon.com/codepipeline/latest/userguide/welcome.html
CodePipeline also supports execution of Lambda functions from Actions in the pipeline. So, as advised before, create a Lambda function to deploy your API but call it from Codepipeline instead of CloudFormation.
Consult this page for details:
http://docs.aws.amazon.com/codepipeline/latest/userguide/actions-invoke-lambda-function.html
I was using above approach but it looks to complicated to me just to deploy API gateway. If we are changing name of the resources then it takes time to delete and recreate the resources which increases down time for you application.
I'm following below approach to deploy API gateway to the stage using AWS CLI and it is not affecting the deployment with Cloudformation stack.
What I'm doing is, running below AWS CLI command after deployment is completed for API Gateway. It will update the existing stage with latest updates.
aws apigateway create-deployment --rest-api-id tztstixfwj --stage-name stg --description 'Deployed from CLI'
The answer here is to use the AutoDeploy property of the Stage:
Stage:
Type: AWS::ApiGatewayV2::Stage
Properties:
StageName: v1
Description: 'API Version 1'
ApiId: !Ref: myApi
AutoDeploy: true
Note that the 'DeploymentId' property must be unspecified when using 'AutoDeploy'.
See documentation, here: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-apigatewayv2-stage.html
From the blogspot post linked by TheClassic (best answer so far!), you have to keep in mind that if you aren't generating your templates with something that can insert a valid timestamp in place of $TIMESTAMP$, you must update that manually with a time stamp or otherwise unique ID. Here is my functional example, it successfully deletes the existing deployment and creates a new one, but I will have to update those unique values manually when I want to create another change set:
rDeployment05012019355:
Type: AWS::ApiGateway::Deployment
DependsOn: rApiGetMethod
Properties:
RestApiId:
Fn::ImportValue:
!Sub '${pApiCoreStackName}-RestApi'
StageName: !Ref pStageName
rCustomDomainPath:
Type: AWS::ApiGateway::BasePathMapping
DependsOn: [rDeployment05012019355]
Properties:
BasePath: !Ref pPathPart
Stage: !Ref pStageName
DomainName:
Fn::ImportValue:
!Sub '${pApiCoreStackName}-CustomDomainName'
RestApiId:
Fn::ImportValue:
!Sub '${pApiCoreStackName}-RestApi'
I may be late, but here are the options which which you do a redeployment if a API resources changes, may be helpful to people who still looking for options -
Try AutoDeploy to true. If you are using V2 version of deployment. Note that you need to have APIGW created through V2. V1 and V2 are not compatible to each other. https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-apigatewayv2-stage.html#cfn-apigatewayv2-stage-autodeploy
Lambda backed custom resource, Lambda inturn call createDeployment API - https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/template-custom-resources.html
CodePipeline that has an action that calls a Lambda Function much like the Custom Resource would - https://docs.aws.amazon.com/codepipeline/latest/userguide/actions-invoke-lambda-function.html
SAM(Serverless Application Model) follows a similar syntax to CloudFormation which simplifies the resource creation into abstractions and uses those to build and deploy a normal CloudFormation template. https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/what-is-sam.html
If you are using any abstraction layer to cloudformation like Sceptre, you can have a hook to call createDeployment after any-update to the resource https://sceptre.cloudreach.com/2.3.0/docs/hooks.html
I gone with third option since I kept using Sceptre for Cloudformation deployment. Implementing hooks in sceptre is easy as well.
Reading through this article, I did not come to a conclusion right away, as the information here is stretched through multiple sources. I try to sum up all the findings from here (and linked source) as my personal testing to help others avoid the hunt.
Important to know is that each API always has a dedicated URL. The associated stages only get a separate suffix. Updating the deployment does not change the URL, recreating the API does.
API
├─ RestAPI (incl. Resource, Methods etc)
├─ Deployment
├─ Stage - v1 https://6s...com/v1
├─ Stage - v2 https://6s...com/v2
Relation stage and deployment:
To deploy AWS API Gateway through CloudFormation (Cfn) you need a RestApi-Cfn-Resource and a Deployment-Cfn-Resource. If you give the Deployment-Resource a stage name, the deployment automatically creates a deployment on top of the "normal" creation. If you leave this out, the API is created without any stage. Either way, if you have a deployment, you can add n-stages to a deployment by linking the two, but a stage and its API always has only one deployment.
Updating simple API:
Now if you want to update this "simple API" just consisting of a RestAPI plus a deployment you face the issue, that if the deployment has a stage name - it can not be updated as it already "exists". To detect that the deployment has to be updated in the first place, you have to either add a timestamp or hash to the deployment resource name in CloudFormation else there is even no update triggered.
Solving the deployment update:
To now enable updating the deployment, you have to split deployment and stage up into separate Cfn-Resources. Meaning, you remove the stage name from the Deployment-Cfn-Resource and create a new Stage-Cfn-Resource which references the deployment resource. This way you can update the deployment. Still, the stage - the part you reference via URL - is not automatically updated.
Propagating the update from the deployment to your stages:
Now that we can update the deployment - aka the blueprint of the API - we can propagate the change to its respective stage. This step AS OF MY KNOWLEDGE is not possible using CloudFormation. Therefore, to trigger the update you either need to add a "custom resource" our you do it manually. Other "none" CloudFormation ways are summed up by #Athi's answer above, but no solution for me as I want to limit the used tooling.
If anybody has an example for the Lambda update, please feel free to ping me - then I would add it here. The links I found so far only reference a plain template.
I hope this helped others understanding the context a bit better.
Sources:
Problem description with Cfn-template, 2
Adding timestamp to deployment resource, 2
Using CodePipeline as a solution
Related question and CLI update answer
Related terraform issue
Related AWS forum thread
This worked for me :
cfn.yml
APIGatewayStage:
Type: 'AWS::ApiGateway::Stage'
Properties:
StageName: !Ref Environment
DeploymentId: !Ref APIGatewayDeployment$TIMESTAMP$
RestApiId: !Ref APIGatewayRestAPI
Variables:
lambdaAlias: !Ref Environment
MethodSettings:
- ResourcePath: '/*'
DataTraceEnabled: true
HttpMethod: "*"
LoggingLevel: INFO
MetricsEnabled: true
DependsOn:
- liveLocationsAPIGatewayMethod
- testJTAPIGatewayMethod
APIGatewayDeployment$TIMESTAMP$:
Type: 'AWS::ApiGateway::Deployment'
Properties:
RestApiId: !Ref APIGatewayRestAPI
DependsOn:
- liveLocationsAPIGatewayMethod
- testJTAPIGatewayMethod
bitbucket-pipelines.yml
script:
- python3 deploy_api.py
deploy_api.py
import time
file_name = 'infra/cfn.yml'
ts = str(time.time()).split(".")[0]
print(ts)
with open(file_name, 'r') as file :
filedata = file.read()
filedata = filedata.replace('$TIMESTAMP$', ts)
with open(file_name, 'w') as file:
file.write(filedata)
========================================================================
Read this for more information: https://currentlyunnamed-theclassic.blogspot.com/2018/12/mastering-cloudformation-for-api.html
If you have something to do the $TIMESTAMP$ replacement, I'd probably go with that as it's cleaner and you don't have to do any manual API Gateway management.
I have found that the other solutions posted here mostly do the job with one major caveat - you can't manage your Stage and Deployment separately in CloudFormation because whenever you deploy your API Gateway, you have some sort of downtime between when you deploy the API and when the secondary process (custom resource / lambda, code pipeline, what have you) creates your new deployment. This downtime is because CloudFormation only ever has the initial deployment tied to the Stage. So when you make a change to the Stage and deploy, it reverts back to the initial deployment until your secondary process creates your new deployment.
*** Note that if you are specifying a StageName on your Deployment resource, and not explicitly managing a Stage resource, the other solutions will work.
In my case, I don't have that $TIMESTAMP$ replacement piece, and I needed to manage my Stage separately so I could do things like enable caching, so I had to find another way. So the workflow and relevant CF pieces are as follows
Before triggering the CF update, see if the stack you're about to update already exists. Set stack_exists: true|false
Pass that stack_exists variable in to your CF template(s), all the way down to the stack that creates the Deployment and Stage
The following condition:
Conditions:
StackExists: !Equals [!Ref StackAlreadyExists, "True"]
The following Deployment and Stage:
# Only used for initial creation, secondary process re-creates this
Deployment:
DeletionPolicy: Retain
Type: AWS::ApiGateway::Deployment
Properties:
Description: "Initial deployment"
RestApiId: ...
Stage:
Type: AWS::ApiGateway::Stage
Properties:
DeploymentId: !If
- StackExists
- !Ref AWS::NoValue
- !Ref Deployment
RestApiId: ...
StageName: ...
Secondary process that does the following:
# looks up `apiId` and `stageName` and sets variables
CURRENT_DEPLOYMENT_ID=$(aws apigateway get-stage --rest-api-id <apiId> --stage-name <stageName> --query 'deploymentId' --output text)
aws apigateway create-deployment --rest-api-id <apiId> --stage-name <stageName>
aws apigateway delete-deployment --rest-api-id <apiId> --deployment-id ${CURRENT_DEPLOYMENT_ID}
Use SAM
AWS::Serverless::Api
This does the deployment for you when it does the Transformation
I wish to create the following AWS CodePipeline process
Developers push code to GitHub
CodeDeploy deploys code to Test environment EC2
Test engineer tests the web app on EC2
Test engineer manually approves this revision
CodeDeploy deploys code to Live environment EC2
My problem is in step 4 and 5, how can I make the codepipeline wait for manual approval (step 4) and then if approved, automatically proceeds to deploying next stage (step 5)
Thanks
To address your problem with Step 4 and 5 this can be accomplished 2 ways:
1) AWS has added the ability to add a manual approval step via the console:
https://aws.amazon.com/about-aws/whats-new/2016/07/aws-codepipeline-adds-manual-approval-actions/
Open existing CodePipeline
Edit CodePipeline
Select the pencil icon on the Stage you want the Manual approval in
Click then add this type of action:
2) ManualApproval can also be added to a CodePipeline CloudFormation Template Action like this:
- InputArtifacts: []
Name: !Join ["",[!Ref GitHubRepository, "-prd-approval"]]
ActionTypeId:
Category: Approval
Owner: AWS
Version: '1'
Provider: Manual
OutputArtifacts: []
Configuration:
NotificationArn: !Ref ManualApprovalNotification
ExternalEntityLink: OutputTestUrl
RunOrder: 3