I have a CodePipeline set up where changes to code builds and pushes an image to ECR. I am looking to automate updating ECS with the new image as it is built. I have configured the ECS Blue/Green action but when it runs it fails almost immediately with a message about an "Internal Error". There is no failed deployment created in CodeDeploy.
I have configured CodePipeline with two inputs:
the source code used to build the image
a zip in S3 containing the appspec.yaml and the taskdef.json
When either input changes I rebuild the container and push to ECR tagged 'latest'. The next step should be a Blue/Green deployment to ECS. I have configured CodeDeploy and the job works if triggered manually.
When it is triggered via CodePipeline it will fail and I receive a message "Action execution failed
InternalError. Error reference code: <some id>". I suspect that there may be some underlying issue with IAM but I can't find where to start looking at this stage. There is no failed deployment shown in CodeDeploy so I don't see a way to get more information about what has failed.
My policy for CodePipeline is copied from the one documented here: https://docs.aws.amazon.com/codepipeline/latest/userguide/how-to-custom-role.html#view-default-service-role-policy
I have read through the troubleshooting docs here: https://docs.aws.amazon.com/codepipeline/latest/userguide/troubleshooting.html
I believe my issue is similar to the one described here: https://forums.aws.amazon.com/thread.jspa?messageID=897822
After a bit more reading of similar posts here, on serverfault.com and the AWS forums I have been able to resolve this.
In my case the issue was that my taskdef.json was not valid. It took me several hours going through each step to realise that while it was valid JSON it only included the container definitions section. On fixing that it appears to now be working correctly.
In the end I have two, related, CodePipelines. One for deploying updated ECR images to ECS (described above) and the other which updates infrastructure and generates a zip containing taskdef.json and appspec.yaml. If that zip changes then my container pipeline runs; likewise if the container image source changes. It needs more testing but right now this appears to be working very smoothly.
Related
I am using AWS CodeCommit for managing my repository and have an EC2 instance where the code is deployed.
Tried setting up CodePipeline - skipped Build Stage and it seems to work fine but it takes over 10-12 mins to complete. I see the majority of time is taken in the Source stage where I I believe it zips the code, uploads to S3 and then deploy downloads from S3 (which is another concern as this would mean higher S3 and network cost).
Would there be a more optimal way to setup a pipeline that would quickly deploy changes to a branch onto EC2 instance?
Thank you.
Tried to lookup documentation on CodePipeline but there is no much information
In this case to avoid the time taken process at source stage. We can use the same configuration for the code pipeline by skipping build stage, but here we can configure Step 4 : beta page as shown in below figure:
Once we start our pipeline, it will start to run automatically. First, it detects the sample code in source location, bundles up the files, and move to second stage that you defined. During this stage, it passes the code to Elastic Beanstalk, which contains the EC2 instance that will host our code. Elastic Beanstalk handles deploying the code to the EC2 instance.
I am trying to create bitbucket pipeline for performing an EC2 deployment using AWS CodeDeploy but it is showing warning at the time of creation which is:
Expected a section but found sequence.
What is this warning about even if I am doing exactly the same way given in documentation.
Below is screenshot:
Someone let me know how can I correct it.
I know that there are already countless questions in this direction, but unfortunately I was not able to find the right answer yet. If a post already exists, please just share the link here.
I have several gitlab CI / CD pipelines. The first pipeline uses Terraform to build the complete infrastructure for an ECS cluster based on Fargate. The second / third pipeline creates nightly builds of the frontend and the backend and pushes the Docker Image with the tag "latest" into the ECR of the (staging) AWS account.
What I now want to achieve is that the corresponding ECS tasks are redeloyed so that the latest Docker images are used. I actually thought that there is a way to do this via CloudWatch Events or whatsoever, but I don't find a really good starting point here. A workaround would be to install the AWS CLI in the CI / CD pipeline and then do a service update with "force new deployment". But that doesn't seem very elegant to me. Is there any better way here?
Conditions:
The solution must be fully automated (either in AWS or in gitlab CI / CD)
Switching to AWS CodePipeline is out of discussion
Ideally as close as possible to AWS standards. I would like to avoid extensive lambda functions that perform numerous actions due to their maintainability.
Thanks a lot!
Ok, for everybody who is interested in an answer. I solved it that way:
I execute the following AWS CLI command in the CICD pipeline
aws ecs update-service --cluster <<cluster-name>> --service <<service-name>> --force-new-deployment --region <<region>>
Not the solution I was looking for but it works.
As a general comment it is not recommended to always push the same container tag because then rolling back to a previous version in case of failure becomes really difficult.
One suitable option would be to use git tags.
Let's say you are deploying version v0.0.1
You can create a file app-version.tf which will contain the variable backend-version = v0.0.1 that you can reference on the task definition of the ecs service.
Same thing can be done for the container creation using git describe.
So, you get a new task definition for every git tag and the possibility of rolling back just by changing a value in the terraform configuration.
It is beneficial to refer to images using either digests or unique immutable tags. After the pipeline pushes the image, it could:
Grab the image's digest/unique tag
Create a new revision of the task definition
Trigger an ECS deployment with the new task definition.
As sgramo93 mentions, the big benefit is that rolling back your application can be done by deploying an older revision of the task definition.
I want to trigger a blue/green ECS deploy on ECR image update. The deployment stage requires three input artifacts: imageDetail.json, appspec.json and taskdef.json.
While creating a pipeline I pick ECR repository as a source, which creates an imageDetail.json SourceArtifact, this is clear. Later in a build stage I can put it in output artifacts.
What I miss completely is how to provide the two remaining files? Should I inline them in the build stage definition buildspec.yaml (they are quite big and inlining seems tedious) or fetch them somehow from CodeCommit (so far I thought I can do it without having to set up CodeCommit only for that purpose)?
What is the usual practice for providing those files?
Yes, you will need to source the other files from a Code repository (github/CodeCommit). This tutorial is a nice guide on the ECS/CodeDeploy deployment pipeline:
https://docs.aws.amazon.com/codepipeline/latest/userguide/tutorials-ecs-ecr-codedeploy.html
I created a repo in Code Commit for a static s3 website
Then I created a CodePipeline and configured the code build part.
There I set the Build Spec file with the some basic commands:
build and then copy the files in the s3 bucket.
The third step the Code Deploy I'm not sure why it's needed.
When I run it it gets stuck for an hour.
I did disable it and the site was deployed just fine.
Am I missing something?
You can disable the CodeDeploy part if it is working fine for you. Or you can skip the CodeBuild step and use appspec.yml to deploy the static website onto S3.
You have to use either of the steps to make it work, you can't skip both the steps.
CodeDeploy part is present in the CodePipeline in case you need to deploy it on your EC2 fleet or Autoscaling Group after you have built the artifacts. If not needed, just skip it.
Codepipeline has three stages source->codeBuild->codeDeploy. According to Amazon you must use atleast two stages of the Codepipeline, You cannot skip the first stage (i.e source) but you choose any one or both from the remaining. For your use case source and CodeBuild stages are enough you don't need codedeploy. Just remove the codeDeploy stage.