Is there any possibility to use CodeDeploy environment variables in section files of AppSpec file - amazon-web-services

I have website, which stored on AWS EC2 servers.
We have 2 servers, one for production environment and another one for development and staging environments.
Development and staging environments located in different folders. For example development env stored in /var/www/development, while staging stored in /var/www/staging.
I'd like to use AWS CodeDeploy to upload files directly from bitbucket. I put AppSpec file, which copy source code to /var/www/html folder and install all dependencies and configurations. But I want my AppSpec file to copy source code to /var/www/development or to /var/www/staging depending on Development group, that was selected.
Is there is any way to do it or, maybe, there are some better approach in my situation?

The appspec.yml is a bit inflexible so use the following to deploy code in to different folders on the same instance.
version: 0.0
os: linux
files:
- source: /
destination: /var/www/my-temp-dir
permissions:
- object: /var/www/my-temp-dir
owner: ec2-user
group: ec2-user
hooks:
BeforeInstall:
- location: ci/integrations-deploy-pre.sh
runas: root
AfterInstall:
- location: ci/integrations-deploy-post.sh
runas: root
Inside of my integrations-deploy-post.sh file, I then use the CodeDeploy environment variables to move the files in to the place I need them to be;
#!/bin/bash
if [ "$DEPLOYMENT_GROUP_NAME" == "Staging" ]
then
cp -R /var/www/my-temp-dir /var/www/my-staging-dir
chown -R ec2-user:ec2-user /var/www/my-staging-dir
# Insert other commands that need to run...
fi
if [ "$DEPLOYMENT_GROUP_NAME" == "UAT" ]
then
cp -R /var/www/my-temp-dir /var/www/my-uat-dir
chown -R ec2-user:ec2-user /var/www/my-uat-dir
# Insert other commands that need to run...
fi
NOTE: In my integrations-deploy-post.sh You'll also need the commands you want to run on production. Removed for simplicity.

The recommended way to change AppSpec or custom scripts behavior is to utilize environment variables provided by the CodeDeploy agent. You have access to the deployment group name and the application name.
if [ "$DEPLOYMENT_GROUP_NAME" == "Staging" ]; then
# Copy to /var/www/staging
elif [ "$DEPLOYMENT_GROUP_NAME" == "Development" ]; then
# Copy to /var/www/development
elif [ "$DEPLOYMENT_GROUP_NAME" == "Production" ]; then
# Copy to /var/www/html
else
# Fail the deployment
fi

I had the same problem, but I used source control as a solution to this.
My workflow is using Gitlab CI > AWS Code Pipeline (S3 Source and CodeDeploy).
So in my development branch, my AppSpec file would look like this:-
version: 0.0
os: linux
files:
- source: /
destination: /var/www/html/my-project-dev
hooks:
AfterInstall:
- location: scripts/after_install.sh
timeout: 400
runas: root
in my staging branch:-
version: 0.0
os: linux
files:
- source: /
destination: /var/www/html/my-project-staging
hooks:
AfterInstall:
- location: scripts/after_install.sh
timeout: 400
runas: root
My Gitlab-CI just uses a shell executor to my EC2 instance and it basically compresses my project folder and uploads to S3.
.gitlab-ci.yml
stages:
- deploy
setup dependencies:
stage: .pre
script:
- echo "Setup Dependencies"
- pip install awscli
deploy to s3:
stage: deploy
script:
- tar -cvzf /tmp/artifact_$CI_COMMIT_REF_NAME.tar ./*
- echo "Copy artifact to S3"
- aws s3 cp /tmp/artifact_$CI_COMMIT_REF_NAME.tar s3://project-artifacts/
clean up:
stage: .post
script:
- echo "Removing generated artifact"
- rm /tmp/artifact_$CI_COMMIT_REF_NAME.tar
Note that $CI_COMMIT_REF_NAME is used to differentiate the artifact file being generated. In development branch it would be artifact_development.tar, in staging branch artifact_staging.tar.
Then, I have 2 pipelines listening to the two respective artifacts which deploys to 2 different CodeDeploy Application.
Not sure if this is the best way, surely welcome any suggestions that is better

Related

How to deploy 2 CodeCommit repositories using CodeDeploy and CodePipeline with the same appspec.yml?

I have 3 CodeCommit repositories:
Repo 1: App A files
Repo 2: App B files
Repo 3: Some config files for apps A and B + appspec.yml
I would like to create 2 CodePipelines deploying my apps on an EC2. The first one taking Repo1 and Repo3 and then the second one taking Repo2 and Repo3. I want to use the same appspec.yml (because Repo 1 and 2 have the same tree structure and I don't want to duplicate appspec.yml and common config files from Repo3)
It seems like it's not possible to have 2 sources in CodePipeline if the next stage is CodeDeploy. So I decided to put Repo 3 as source and use a BeforeInstall script to git clone Repo 1 or Repo2 depending on the deployment group.
So my appspec.yml looks like that:
version: 0.0
os: linux
files:
- source: config
destination: /var/lib/app
hooks:
BeforeInstall:
- location: scripts/clone-repository.sh
And then clone-repository.sh looks like so
yum install -y git
if [ "$DEPLOYMENT_GROUP_NAME" == "group1" ]
then
git clone <Repo 1>
mv to the right place
etc
elif [ "$DEPLOYMENT_GROUP_NAME" == "group2" ]
then
git clone <Repo 2>
mv to the right place
etc
I was forced to install git otherwise I got an error. I also tried to add these lines
git config --global credential.helper '!aws codecommit credential-helper $#'
git config --global credential.UseHttpPath true
The CodeDeploy role has AWSCodeCommitPowerUser and AWSCodeDeployRole but impossible to git clone. I get the following error -> error: [stderr]: unable to access '': the requested URL returned error: 403
What's missing?
If you have another idea to solve my issue, I'll take it!
Thank you for you help!

AWS CodeDeploy Appspec.yml files are not being copied to destination

My files are properly copied to an EC2 instance for deployments, but they are not copied to the destination. I've got at appspec.yml file like this:
ubuntu#ip:~$ cat /opt/coded*/deployment-root/*/*/de*/appspec.yml
version: 0.0
os: linux
# I also tried with source: /
files:
- source: .
destination: /home/ubuntu/
file_exists_behavior: OVERWRITE
# notes
# ------------------------------------------------------------------------------------
# To learn more about hooks, visit the docs here:
# https://docs.aws.amazon.com/codedeploy/latest/userguide/reference-appspec-file-structure-hooks.html#appspec-hooks-server
hooks:
BeforeInstall:
- location: scripts/install_dependencies
timeout: 300
runas: root
ApplicationStart:
- location: scripts/start_server
timeout: 300
runas: root
ApplicationStop:
- location: scripts/stop_server
timeout: 300
runas: root
I've also tried setting source to source: / with the same result; nothing's copied
And I see my files are all in that /opt/coded* etc. path. But why isn't the CodeDeploy agent copying my files to the destination? The Download bundle hook is good, and the only error I get is in the ApplicationStart hook when I need to actually access my files.
Note: I'm using GitHub as my revision storage, not S3, if that helps
Edit: something that might be useful to know is in my install_dependencies script, I run something along the lines of:
cd /home/ubuntu/
pip install -r requirements.txt
where the pip installation is the line where it can't find the files
Reasoning: https://docs.aws.amazon.com/codedeploy/latest/userguide/reference-appspec-file-example.html#appspec-file-example-server
The order of the hooks. The files are copied before the AfterInstall hook.

How Can I deploy from Bitbucket to AWS to different instances and different folders?

I have two instances in AWS. One for production and one for homologation. I deploy automatically with CodeDeploy. I have two branches on BitBucket, the master and homolog. When I commit in the homolog deploy must go to the instance of homologation, and if I make a merge in the master deploy must be in the production stage.
To do the automatic deploy of Bitbucket to AWS there is a series of files that configure the deploy details. One of these files is the appspec.yml. According to AWS it is only possible to have an appspec.yml file.
This basic form file has the following structure:
version: 0.0
os: linux
files:
- source: /
destination: /var/www/html
hooks:
AfterInstall:
- location: deploy-scripts/install_dependencies.py
timeout: 300
runas: root
The problem is that for each instance I have a destination folder.
If I do the deploy on the homolog instance the destination folder should be var/www/html and for the production instance it should be var/www/html/test/
I tried to do it as below:
version: 0.0
os: linux
files:
- source: /
destination: deploy-scripts/destination.py
hooks:
AfterInstall:
- location: deploy-scripts/install_dependencies.py
timeout: 300
runas: root
That's the destination.py:
if os.environ['APPLICATION_NAME'] == 'ahimsa-store-homolog':
return '/var/www/html/'
elif os.environ['APPLICATION_NAME'] == 'ahimsa-store':
return '/var/www/html/teste/'
The above option does not work. How can I do this?
The files section of appspec.yml doesn't run scripts.
Move the required files to a temporary folder using the files section
Create a script that will move these files from the temporary location to the required destination depending on your requirements. As you suggested, using os.environ['APPLICATION_NAME'] for example.
Make sure your script sets correct file permissions after moving the files.
Include that script in the AfterInstall section, so the script can find the new files "installed" in the temporary location you choose. Also make sure it is before installing the dependencies!
Another option is to have a different appspec in each branch. It would make merging more difficult, but it could help.

bug in codedeploy afterinstall hook?

We have autoscaling set up for our symfony3 application. We are using aws codedeploy to deploy to autoscaling instances.
My appspec.yml file
version: 0.0
os: linux
files:
- source: /
destination: /usr/share/nginx/<some_dir>
hooks:
AfterInstall:
- location: post_deploy.sh
timeout: 180
runas: ubuntu
post_deploy.sh
#!/bin/bash
doc_root=/usr/share/nginx/<some_dir>
current_dir=$PWD
cd $doc_root
sudo -E composer install --no-interaction --no-dev --optimize-autoloader
cd $current_dir
and also exported environment variables for parameters.yml file
when we deploy revision, codedeploy succeed in deployment. But when i access my app through browser, nginx error log says:
PHP Fatal error: Uncaught exception 'Symfony\Component\DependencyInjection\Exception\ParameterNotFoundException' with message 'You have requested a non-existent parameter "database.host". Did you mean one of these: "database_host", "database_port"?
Strange thing is that when i run post_deploy.sh script manually by logging in to my server it executes well and no error afterwards.
I don't know how to deal with it.
Try to change database.host to database_host. that what's indicated in the message.
codedeploy doesn't preserve env var in spite of -E option.
So i pass the env var in command itself, like this
sudo SYMFONY_ENV=$SYMFONY_ENV SYMFONY__DATABASE__NAME=$SYMFONY__DATABASE__NAME SYMFONY__DATABASE__USER=$SYMFONY__DATABASE__USER SYMFONY__DATABASE__HOST=$SYMFONY__DATABASE__HOST SYMFONY__DATABASE__PORT=$SYMFONY__DATABASE__PORT SYMFONY__DATABASE__PASSWORD=$SYMFONY__DATABASE__PASSWORD COMPOSER_HOME=$COMPOSER_HOME composer install --no-interaction --no-dev --optimize-autoloader
worked for me.

Setting directory owner and permission with appspec.yml through Amazon Web Service CodeDeploy

I'm deploying a Node.js application through Codeship using the CodeDeploy AWS deployment system.
I am making use of the appspec.yml file to set the owner and permissions of one of the deployed directory.
I want to allow read/write for any files that will be created in a specified folder of the deployment. Files will be created by the web application once it starts running.
Currently my appspec.yml contains the following:
version: 0.0
os: linux
files:
- source: /
destination: /var/www/APPLICATION_NAME
permissions:
- object: /var/www/APPLICATION_NAME/tmpfiles
mode: 644
owner: ec2-user
type:
- directory
I found appspec.yml file really hard to deal with.
I have very big and complex folder structure and it's headache to try to set permissions with appspec.yml file. Because of this reason, I make use of "hooks" to call small bash script to set my permissions
Here is an example appspec.yml file that I have:
version: 0.0
os: linux
files:
- source: /
destination: /var/www
hooks:
AfterInstall:
- location: scripts/set-permissions.sh
Here is an example of set-permissions.sh file:
#!/bin/bash
# Set ownership for all folders
chown -R www-data:www-data /var/www/
chown -R root:root /var/www/protected
# set files to 644 [except *.pl *.cgi *.sh]
find /var/www/ -type f -not -name ".pl" -not -name ".cgi" -not -name "*.sh" -print0 | xargs -0 chmod 0644
# set folders to 755
find /var/www/ -type d -print0 | xargs -0 chmod 0755
If you have Access Control Lists (ACLs) enabled on your filesystem, you can use default ACLs on your directory to allow read/write permissions for owner/group/others on newly created files in that directory.
AWS CodeDeploy lets you specify ACLs for your files in appspec.yml. It can take any valid ACL entries that can be passed to setfacl [1]
For e.g, in your case to set read, write and execute permission for everyone on all newly created files you can do something like
version: 0.0 os: linux files:
- source: /
destination: /var/www/APPLICATION_NAME permissions:
- object: /var/www/APPLICATION_NAME/tmpfiles
mode: 644
acls:
- "d:u::rwx"
- "d:g::rwx"
- "d:o::rwx"
owner: ec2-user
type:
- directory
The permissions can be restricted by the application that creates the new files. You can also set default ACL mask to set mask bits to force certain permissions. For e.g, "d:m::rw" would mask the execute permission. You can explore more about ACL and masking here http://www.vanemery.com/Linux/ACL/POSIX_ACL_on_Linux.html
[1] http://linux.die.net/man/1/setfacl