I have a CD work flow using CodePipeline + CodeCommit + CodeDeploy to EC2 instances.
I have everything else working except for this --file-exists-behavior option. If I create a deployment from the Aws CLI or the console. The deployment works fine.
Is there anyway I can use this option with CodePipeline?
I am still getting an error without this option.
The deployment failed because a specified file already exists at this location
So, when you deploy, CodeDeploy will look for a file (on the instance) with information on previously deployed files for the application/deployment-group in question, if it finds the file it then it uses it to cleanup (delete) the existing files, preparing for copying of the new revision files.
Refer This AWS Documentation
But if the cleanup file doesn't exist, either because the existing code was initially deployed in some other way (eg cloudformation instance userdata/metadata sources) or the CodeDeploy deployment-group has been recreated for some reason then the cleanup file will either not exist or not match the deployment-group id. Therefore it wont be run. Therefore the files will not be cleaned up, and the deployment will fail since CodeDeploy does not overwrite files.
What a Solution could be in this scenario is to have a hook (eg BeforeInstall) that calls a script to delete the applications files (if they exist) on the instance.
Hope this helps
Related
I have an Autoscaling Group that gets scaled/deployed on using CodeDeploy, I had created a working instance and got it all up and running and took an image of it for a launch template, this instance was attached to the Autoscaling group and when I deployed it worked perfectly fine.
However, over the weekend, AWS decided to tear that instance down and create a whole new one, and when it did, the deployment failed because of the error File already exists. I understand what the error is, but I don't understand why it occurred? That deployment had already occurred on that image, so it shouldn't run into that issue....
Anyways, I know when you manually kick off a deployment, you can opt to Overwrite existing files to avoid this error. But I haven't been able to find this setting for when Autoscaling automatically kicks off deployments. How can you allow duplicate files to be overwritten by CodeDeploy when the deployment is kicked off automatically by your Auto Scaling group?
Make sure you have specified overwrite in your appspec.yml file
file_exists_behavior: DISALLOW|OVERWRITE|RETAIN
AWS Docs
I have a Wordpress repository that I'm deploying to a Beanstalk application through the CodePipeline + Github.
When the CodePipeline releases any changes on the master branch, the deployment process starts, but it replaces the whole project and I'd like to keep some directories in the instance such as uploads/ and plugins/.
Is there a way to keep them when the Deployment stage starts?
I solved the issue using the S3 service instead.
Setup a S3 bucket
Install this plugin https://github.com/humanmade/S3-Uploads
The plugin contains instructions to perform the migration
Done, all files will be uploaded to S3 instead and the Pipeline will never replace it.
I have used Cloudformation template to create my EC2 instance. In this instance there is a file
in home directory that I copy from S3 while creating stack.
I have that file stored locally as well. Now, I modify that file locally and want to copy it to S3
and from S3 to EC2 instance.
I want to automate this process through Cloudformation. So that, whenever I modify this file locally,
after doing update stack, it uploads the modified file to S3 and from S3 to my EC2 instance.
Can anyone please help how this can be achieved?
One thing that comes to mind (bearing in mind the application specific nature of what you are trying to do) is using ECS instead of just EC2.
Note: This may be overkill, but it would work. Also if updates were extremely frequent this would be a major pain so just uploading the file to S3 with a script alongside update-stack (if the update-stack is even necessary) and then polling for changes to that S3 file in your EC2 application would be fine.
Anyway, this is a pattern we use when we are doing something like training a model with new data, which we then wish to deploy to AWS, replacing an application with an older version of the model.
You build the Docker image locally and your special file gets included inside the container. You push the Docker image to DockerHub or AWS ECS Registry or wherever. You update the cloudformation template ECS configuration to use the tag of this new Docker image and update the stack. Then ECS pulls this new image and the new Docker container(s) take the place of the old one(s) and it will have your special file inside it.
I know of at least one way: set up your EC2 in an AutoScaling Group (ASG). Then, on creation, use cfn-init on your UserData and have it fetch the file under sources. Use both Creation and Update policies. On update, have the WillReplace attribute set to true under AutoScalingReplacingUpdate. When you update, CloudFormation will create a new ASG with your fresh new file copy. If you signal success on your update, it will remove the previous instance and ASG, giving you an immutable infrastructure setup. Put everything behind a load balancer, and you got yourself a highly available blue/green deployment too.
I have a site in a S3 bucket, configured for web access, for which I run an aws s3 sync command every time I push on a specific git repository (I'm using Gitlab at the moment).
So if I push to stable branch, a Gitlab runner performs the npm start build command for building the site, and then aws s3 sync to synchronize to a specific bucket.
I want to migrate to CodeCommit and use pure AWS tools to do the same.
So far I was able to successfully setup the repository, create a CodeBuild for building the artifact, and the artifact is being stored (not deployed) to a S3 bucket. Difference is that I can't get it to deploy to the root folder of the bucket instead of a subfolder, seems like the process is not made for that. I need it to be on a root folder because of how the web access is configured.
For the deployment process, I was taking a look at CodeDeploy but it doesn't actually let me deploy to S3 bucket, it only uses the bucket as an intermediary for deployment to a EC2 instance. So far I get the feeling CodeDeploy is useful only for deployments involving EC2.
This tutorial with a similar requirement to mine, uses CodePipeline and CodeBuild, but the deployment step is actually a aws s3 sync command (same as I was doing on Gitlab), and the actual deployment step on CodePipeline is disabled.
I was looking into a solution which involves using AWS features made for this specific purpose, but I can't find any.
I'm also aware of LambCI, but to me looks like what CodePipeline / CodeBuild is doing, storing artifacts (not deploying to the root folder of the bucket). Plus, I'm looking for an option which doesn't require me to learn or deploy new configuration files (outside AWS config files).
Is this possible with the current state of AWS features?
Today AWS has announced as a new feature the ability to target S3 in the deployment stage of CodePipeline. The announcement is here, and the documentation contains a tutorial available here.
Using your CodeBuild/CodePipeline approach, you should now be able to choose S3 as the deployment provider in the deployment stage rather than performing the sync in your build script. To configure the phase, you provide an S3 bucket name, specify whether to extract the contents of the artifact zip, and if so provide an optional path for the extraction. This should allow you to deploy your content directly to the root of a bucket by omitting the path.
I was dealing with similar issue and as far as I was able to find out, there is no service which is suitable for deploying app to S3.
AWS CodeDeploy is indeed for deploying code running as server.
My solution was to use CodePipeline with three stages:
Source which takes source code from AWS CodeCommit
Build with AWS CodeBuild
Custom lambda function which after successful build takes artifact from S3 artifact storage, unzip it and copies files to my S3 website host.
I used this AWS lambda function from SeamusJ https://github.com/SeamusJ/deploy-build-to-s3
Several changes had to be made, I used node-unzip-2 instead of unzip-stream for unziping artifict from s3.
Also I had to change ACLs in website.ts file
Uploading from CodeBuild is currently the best solution available.
There's some suggestions on how to orchestrate this deployment via CodePipeline in this answer.
I am trying to set AWS Code deploy with Github to automate my deloyment. the problem i am having is the ec2 instance is already active and has the web app code. the problem is there is also other content on the instance that I don't want under source control.
Code deploy is failing with the following error:
The deployment failed because a specified file already exists at this location
this is because the source code is on the instance and was not added as a revision.
So my question is there any way to make the code deploy recognise the current files on the instance as an inital revision?
This is possible! This is an option during a deployment: "Content options:
Choose the action for AWS CodeDeploy to take during a deployment when a file on a target instance has the same name as a file in the application revision, for the same target location."
You can choose fail, overwrite and retain. Retain is probably the best choice in your case.
You can find more information
docs.aws.amazon.com (strangely under 'rollback')
cli docs.aws.amazon.com (see --file-exists-behavior (string))
This is my sample appspec.yml
version: 0.0
os: linux
files:
- source: /
destination: /var/www/wordpress/
overwrite: true
file_exists_behavior: OVERWRITE
And this is Updated AWS User Guide for AppSpec 'files' section (EC2/On-Premises deployments only). In there you will find three options, DISALLOW,OVERWRITE and RETAIN.
You can simply delete the entire contents of your web root and allow CodeDeploy to update it. Any type of deployment will replace the entire contents.
If you're feeling unsure then copy the webroot to another folder, and in the very unlikely event of a failure you can copy it all back.
Deployments often fail when a file has been modified on the disk, I assume CodeDeploy is checking the modified time of the file.
If you just want the error to go away, you can use one of the earlier Lifecycle Hook scripts (BeforeInstall maybe) to delete that folder before the source gets re-added in a later Hook. If the source doesn't get re-added in a later Hook, add it yourself.
I was getting this error even on new instances, so CodeDeploy must be caching that folder somehow, then getting upset that it did so.
The issue is that CodeDeploy maintains a cleanup file that it updates after every deployment. These are the files it will delete before installing the new ones. You could of course create this yourself, but that's hard. The point of CodeDeploy is to make things easy. This is what I did (Note: My app has a lot of files and I was about to find something other than CodeDeploy that actually made transitioning to it easy instead of painful and stupid, and then I formulated this approach; if you have a small number of files, it might be easier to create a BeforeInstall script or delete them manually):
create a free-tier EC2 instance (t2.micro). If your app is giant, you may need more disk space, so adjust as necessary. THIS WON'T BE RUNNING YOUR APP!
Create a new deployment group pointing to that new instance.
Create a new YAML that doesn't actually do any BeforeInstall or AfterInstall tasks. You are not trying to run your app. You are literally moving files over. This is so that you can grab that cleanup file.
After deployment, ssh into the ec2 instance (remember to allow SSH in the Security Group!) and go to /opt/codedeploy-agent/deployment-root/deployment-instructions/
You will see {deployment-group-identifier}-cleanup. scp this to the ec2 instance on which you actually want to run your app. Or, FileZilla it out onto your main machine and then onto the main ec2 instance. That was the easiest choice for me, because my local machine already had all the right keys.
SSH into the main ec2 instance and move the cleanup file into the /opt/codedeploy-agent/deployment-root/deployment-instructions/ if it isn't already there. Note the failed {deployment-group-identifier} and replace the cleanup file's {deployment-group-identifier} with the failed identifier.
Rerun the deployment.
8.???
Profit.
I will provide more in-depth instructions for each of these steps (except for 8 and 9... which will go away) when I have time.
I had same issue when working in CodeDeploy with EC2 Instances.
I removed EC2 Instances from AWS CodeDeploy Deployment Group. I added new EC2 instances to the deployment group.
Otherwise, Replace new deployment group with an existing deployment group.