I have a ubuntu on EC2 for which I have configured my appspec.yml to be executed with following config
version: 0.0
os: linux
files:
- source: /
destination: /home/ubuntu/source/testingCI_CD/
file_exists_behavior: OVERWRITE
hooks:
ApplicationStart:
- location: scripts/something.sh
runas: root
with something.sh being
echo "Some text here." > myfile.txt
The deployment works successfully passing all deployment lifecycle events but the bash file is not getting executed.
For me to executed the bash file I have to go into the folder and execute sudo bash something.sh. And this won't work if I don't use sudo and will result in insufficient permission. I have tried using 777 on the file and trying to execute bash something.sh which didn't work. I am worried if being unable to access the file with sudo is the reason codeDeploy is not executing my bash file.
Related
I have a Django web application that is deployed to AWS elastic beanstalk (Python 3.7 running on 64bit Amazon Linux 2/3.1.3). I am trying to run the following config file
files:
"/usr/local/bin/cron_tab.sh":
mode: "000755"
owner: root
group: root
content: |
#!/bin/bash
exec &>> /tmp/cron_tab_log.txt
date > /tmp/date
source /var/app/venv/staging-LQM1lest/bin/activate
cd /var/app/current
python manage.py crontab add
exit 0
container_commands:
cron_tab:
command: "curl /usr/local/bin/cron_tab.sh | bash"
This file placed in the .ebextentions folder. All other config files are working properly. However, this one is not working. Also, I have tried to run the container_commands code manually on SSH and it gives output such as below.
curl: (3) <url> malformed
I also checked the /tmp folder but there is no cron_tab_log.txt. I checked /usr/local/bin the cron_tab.sh is located there.
I just want this Django-crontab run after the deploy and it doesn't work. How can I handle this issue?
Curl is used for web url call not executing a script, I think you need to change the last line in your config file to be:
command: "sudo /usr/local/bin/cron_tab.sh"
AWS CodeDeploy is used for a simple WordPress application. Installed AWS codedeploy-agent on ubuntu 20.04 with help of the below script
#!/bin/bash
apt update
apt install ruby -y
gem install bundler
git clone https://github.com/aws/aws-codedeploy-agent.git /opt/codedeploy-agent
sudo chown -R root.root /opt/codedeploy-agent
sudo chmod 644 /opt/codedeploy-agent/conf/codedeployagent.yml
sudo chmod 755 /opt/codedeploy-agent/init.d/codedeploy-agent
sudo chmod 644 /opt/codedeploy-agent/init.d/codedeploy-agent.service
cd /opt/codedeploy-agent
bundle install --system
rake clean && rake
cp /opt/codedeploy-agent/init.d/codedeploy-agent /etc/init.d/
systemctl daemon-reload
systemctl start codedeploy-agent
systemctl enable codedeploy-agent
Using the below appspec.yml for code deployment. Its working fine with runas root
Questions :
How to run it as an ubuntu user, ?
Is any issue with while running as root user ?
....
appspec.yaml file
version: 0.0
os: linux
files:
- source: /
destination: /var/www/html/
overwrite: true
hooks:
BeforeInstall:
- location: scripts/before_install.sh
timeout: 300
runas: root
AfterInstall:
- location: scripts/setup_environment.sh
timeout: 300
runas: root
- location: scripts/after_install.sh
timeout: 900
runas: root
ApplicationStart:
- location: scripts/start_server.sh
timeout: 300
ApplicationStop:
- location: scripts/stop_server.sh
timeout: 300
ValidateService:
- location: scripts/validate_service.sh
timeout: 300
While runas ubuntu getting the below error.
Error code
ScriptFailed
Script name
scripts/setup_environment.sh
Message
Script at specified location: scripts/setup_environment.sh run as user ubuntu failed with exit code 4
LifecycleEvent - AfterInstall
Script - scripts/setup_environment.sh
[stderr]shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory
[stderr]shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory
[stderr]/opt/codedeploy-agent/deployment-root/44d6390b-485e-87ef-b50855bbf251/d-D0RTN7AR5/deployment-archive/scripts/setup_environment.sh: line 4: /var/www/html/.env: Permission denied
[stderr]sed: couldn't open temporary file /var/www/html/scripts/seTwGZAv: Permission denied
If you run it as ubuntu user it will not work due to lack of permissions which you are experiencing:
couldn't open temporary file /var/www/html/scripts/seTwGZAv: Permission denied
The reason is that /var/www/html/ is not accessible by ubuntu user. To make it work you would have to change its default permissions which is a bad practice.
Some things have to be executed as root, unless you want to start changing default configurations and permission model of ubuntu operating system.
As appspec.yml file and scripts are managed by you, there is not any security issue while running our script as root. What you'll write is what you'll get.
While using any non root user it is important to provide all the required permissions to that user. In most of the cases you will have to use sudo before each command and make sure your user is added to sudoers.
You need to make sure that
Your git is secure from any unauthorized changes.
CodeDeploy is only accessible to the trusted resources.
If these two things are checked, there's no way any anomalous command can run on your system
So, I'm setting up my first application that uses CodeDeploy (EC2 + S3) and I'm having a very hard time to figure out how to run the scripts after instalation.
So I defined an AfterInstall hook in the AppSpec file refering to my bash script file in the project diretory. When the commands in the script run I get the error stating the files could not be found. So I put an ls command before it all and checked the logs.
My script file is running in the CodeDeploy agent folder. There are many files there that I accidentally created when I was testing but I was expecting them to be in my project root folder.
--Root
----init.sh
----requirements.txt
----server.py
appspec.yml
version: 0.0
os: linux
files:
- source: ./
destination: /home/ubuntu/myapp
runas: ubuntu
hooks:
AfterInstall:
- location: init.sh
timeout: 600
init.sh
#!/bin/bash
ls
sudo apt install python3-pip
pip3 install -r ./requirements.txt
python3 ./server.py
So when ls is executed, it doesn't list the files in my project root directory. I also tried ${PWD} instead of ./ and it didn't work. It is copying the script file to the agent folder and running it.
Refer to this https://docs.aws.amazon.com/codedeploy/latest/userguide/reference-appspec-file-structure-hooks.html
This is written at the end of the above document
The location of scripts you specify in the 'hooks' section is relative
to the root of the application revision bundle. In the preceding
example, a file named RunResourceTests.sh is in a directory named
Scripts. The Scripts directory is at the root level of the bundle.
But apparently it refers to the paths in the appspec file only.
Could someone help? Is it correct? I MUST use absolute paths hard-coded in the script file?
Yes correct. The script doesn't execute in the destination folder, as you might expect. You need to hard code a reference the destination directory /home/ubuntu/myapp to resolve file paths in life cycle scripts.
Use cd to change the directory first:
cd /home/ubuntu/myapp
ls
sudo apt install python3-pip
pip3 install -r ./requirements.txt
python3 ./server.py
I have two problems deploying via AWS CodeDeploy.
I'm trying to deploy CodeCommit's code to an EC2 ubuntu instance.
At appspec.yml
version: 0.0
os: linux
files:
- source: /
destination: /home/ubuntu
hooks:
ApplicationStart:
- location: scripts/ApplicationStart.sh
timeout: 300
runas: ubuntu
There are several config files that I need to place at the right place in the application before starting pm2. I also assume since I set runas in appspec.yml as ubuntu, the bash script will at /home/ubuntu.
The my /home/ubuntu has
config/ backend/ frontend/
Looks like Code deploy won't overwrite the previous deployment so if I have backend/ and frontend/ folder at the directory, it will fail at Install stage.
In the ApplicationStart.sh
#!bin/bash
sudo cp config/config1.json backend/config/config1.json
sudo cp config/config2.json backend/config/environments/development/config2.json
sudo cp config/config3.json frontend/config3.json
sudo pm2 kill
cd backend
sudo npm install
sudo pm2 start "strapi start" --name backend
cd ../frontend
sudo npm install
sudo pm2 start "npm start" --name frontend
While the ApplicationStart stage, it gives me the following error.
LifecycleEvent - ApplicationStart
Script - scripts/ApplicationStart.sh
[stderr]bash: /opt/codedeploy-agent/path/to/deployment/scripts/ApplicationStart.sh: bin/bash:
bad interpreter: No such file or directory
I run the same bash file at the /home/ubuntu. It works fine.
Question 1.
- how to run BeforeInstall.sh without the error? Is there the path problems or something else I try to do but I am not supposed to do?
Question 2.
- How can I let code deploy to overwrite the previous deployment when there are already application folders in the directory (/home/ubuntu)?
- Do I manually delete the directory at BeforeInstall stage?
You're missing a slash before bin/bash in #!bin/bash.
It should be #!/bin/bash.
I am trying to setup CI using AWS CodeDeploy and CircleCI. Right now I am stuck at the step where AWS CodeDeploy should copy stuff into EC2 and run scripts. But somehow CircleCI tells me something is wrong. Does anyone know what might be happening? Thanks.
the appspec.yml is:
version: 0.0
os: linux
files:
- source: /
destination: /home/ubuntu
hooks:
BeforeInstall:
- location: scripts/setup.sh
timeout: 3800
runas: root
ApplicationStart:
- location: scripts/start.sh
timeout: 3800
runas: root
and setup.sh is:
#!/bin/bash
sudo apt-get install nodejs npm
npm install
in the above code I also tried only apt-get install nodejs npm but it's still nor working.
the error message in /var/log/aws/codedeploy-agent/codedeploy-agent.log is as follows:
2015-10-22 08:02:54 ERROR [codedeploy-agent(1314)]: InstanceAgent::Plugins::CodeDeployPlugin::CommandPoller: Error during
perform: InstanceAgent::Plugins::CodeDeployPlugin::ScriptError - Script at specified location:
./scripts/setup.sh run as user root failed with exit code 127 - /opt/codedeploy-agent/lib/instance_agent/plugins/codedeploy/hook_executor.rb:150:in `execute_script'
/opt/codedeploy-agent/lib/instance_agent/plugins/codedeploy/hook_executor.rb:107:in `block (2 levels) in execute'
......
Exit code 127 generally means that the OS couldn't find something required to execute the command. In this case it could be either the script wasn't at the expected path or /bin/bash doesn't exist (unlikely).
Check that the archive being produced by your build process is actually putting your scripts in the archive where your appspec expects them. scripts/setup.sh needs to be in that exact path within your archive.
You can also look at what the agent actually got by checking the deployment archive for your deployment: /opt/codedeploy-agent/deployment-root/deployment-group-id/deployment-id/deployment-archive to make sure the archive is being extracted correctly.
Do the following steps for the debugging:
in the CodeDeploy error log /var/log/aws/codedeploy-agent/codedeploy-agent.log there is a line that says Error during perform: InstanceAgent::Plugins::CodeDeployPlugin::ScriptError - Script at specified location: scripts/setup.sh failed with exit code 1. So from the error log I know the problems might be from this script.
In the above mentioned script setup.sh, put something like this at the beginning of the script:
exec 3>&1 4>&2
trap 'exec 2>&4 1>&3' 0 1 2 3
exec 1>/home/ubuntu/out.log 2>&1
This logs the entire error outputs for you.
Permission issues
It's also possible that EC2 failed to execute those scripts, you need to make sure those files have at least 755 permissions when copied to your instance. So you need to specify 755 file mode for your scripts.
How to change the File Mode on GitHub?
Also in appspec.yml you need can specify a runas directive. Could be ubuntu or root or whatever that gives you the correct permission.
miscellaneous
Some pitfalls on deploying like when you do sudo apt-get install nodejs there will be intermediate steps that ask if you want to install packages and used disk spaces and you have to type Y or N to proceed installation. those scripts would hang there and timeout resulting in failed deployment. So instead you do
sudo apt-get -y install nodejs npm
Or in your setup.sh script maybe you have
chmod -R 777 public
but it's possible CodeDeploy is executing this code in a folder that's different than your project root. So make sure all the paths are valid.