aws codedeploy long-running process - amazon-web-services

Aws codedeploy is being applied.
The appspec file is shown below.
version: 0.0
os: linux
files:
- source: script/install-file.sh
destination: /home/
hooks:
AfterInstall:
- location: script/install-file.sh
timeout: 120
runas: root
ApplicationStart:
- location: script/start-file.sh
timeout: 120
runas: root
I tried Succeeded until AfterInstall.
It is still pending in applicationStart.
AfterInstall installed Java files and set permissions.
chmod 755 ${file_HOME}/bin/install_api
chmod 755 ${file_HOME}/bin/install_web
Auto-run was set.
/bin/cp ${file_HOME}/bin/install_api /etc/init.d
/bin/cp ${file_HOME}/bin/install_web /etc/init.d
Chkconfig --add ib_api
Chkconfig --add ib_web
start-file.sh is below.
#!/bin/bash
# start InnerBeans
sudo service install_api start &
sleep 5
sudo service install_web start &
sleep 5

When calling background processes or daemons inside a LifeCycleEvent script codedeploy-agent (version OFFICIAL_1.0-1.1106_rpm) remains pending until 70 minutes timeout.
First, try removing & like this:
#!/bin/bash
# start InnerBeans
sudo service install_api start
#sleep 5
sudo service install_web start
#sleep 5
If it still failing, you probably have background or daemonized processes inside the init script so you need to redirect outputs stdout and stderr.
Try:
#!/bin/bash
# start InnerBeans
sudo service install_api start > /dev/null 2>&1
#sleep 5
sudo service install_web start > /dev/null 2>&1
#sleep 5
That second way worked for me.
I found the solution here:
https://forums.aws.amazon.com/thread.jspa?messageID=626766&#626766
and here
http://docs.aws.amazon.com/codedeploy/latest/userguide/troubleshooting-deployments.html#troubleshooting-long-running-processes

Related

Running GPU Monitoring on GCP in a container optimized OS

Title has most of the question, but more context is below
Tried following the directions found here:
https://cloud.google.com/compute/docs/gpus/monitor-gpus
I modified the code a bit, but haven't been able to get it working. Here's the abbreviated cloud config I've been running that should show the relevant parts:
- path: /etc/scripts/gpumonitor.sh
permissions: "0644"
owner: root
content: |
#!/bin/bash
echo "Starting script..."
sudo mkdir -p /etc/google
cd /etc/google
sudo git clone https://github.com/GoogleCloudPlatform/compute-gpu-monitoring.git
echo "Downloaded Script..."
echo "Starting up monitoring service..."
sudo systemctl daemon-reload
sudo systemctl --no-reload --now enable /etc/google/compute-gpu-monitoring/linux/systemd/google_gpu_monitoring_agent.service
echo "Finished Script..."
- path: /etc/systemd/system/install-monitoring-gpu.service
permissions: "0644"
owner: root
content: |
[Unit]
Description=Install GPU Monitoring
Requires=install-gpu.service
After=install-gpu.service
[Service]
User=root
Type=oneshot
RemainAfterExit=true
ExecStart=/bin/bash /etc/scripts/gpumonitor.sh
StandardOutput=journal+console
StandardError=journal+console
runcmd:
- systemctl start install-monitoring-gpu.service
Edit:
Turned out it was best to build a docker container with the monitoring script in it and run the docker container in my config script by passing the GPU into the docker container like shown in the following link
https://cloud.google.com/container-optimized-os/docs/how-to/run-gpus

AWS CodeDeploy not executing the bash file on my EC2 ubuntu instance

I have a ubuntu on EC2 for which I have configured my appspec.yml to be executed with following config
version: 0.0
os: linux
files:
- source: /
destination: /home/ubuntu/source/testingCI_CD/
file_exists_behavior: OVERWRITE
hooks:
ApplicationStart:
- location: scripts/something.sh
runas: root
with something.sh being
echo "Some text here." > myfile.txt
The deployment works successfully passing all deployment lifecycle events but the bash file is not getting executed.
For me to executed the bash file I have to go into the folder and execute sudo bash something.sh. And this won't work if I don't use sudo and will result in insufficient permission. I have tried using 777 on the file and trying to execute bash something.sh which didn't work. I am worried if being unable to access the file with sudo is the reason codeDeploy is not executing my bash file.

How to codedeploy appspec.yml runas ubuntu user

AWS CodeDeploy is used for a simple WordPress application. Installed AWS codedeploy-agent on ubuntu 20.04 with help of the below script
#!/bin/bash
apt update
apt install ruby -y
gem install bundler
git clone https://github.com/aws/aws-codedeploy-agent.git /opt/codedeploy-agent
sudo chown -R root.root /opt/codedeploy-agent
sudo chmod 644 /opt/codedeploy-agent/conf/codedeployagent.yml
sudo chmod 755 /opt/codedeploy-agent/init.d/codedeploy-agent
sudo chmod 644 /opt/codedeploy-agent/init.d/codedeploy-agent.service
cd /opt/codedeploy-agent
bundle install --system
rake clean && rake
cp /opt/codedeploy-agent/init.d/codedeploy-agent /etc/init.d/
systemctl daemon-reload
systemctl start codedeploy-agent
systemctl enable codedeploy-agent
Using the below appspec.yml for code deployment. Its working fine with runas root
Questions :
How to run it as an ubuntu user, ?
Is any issue with while running as root user ?
....
appspec.yaml file
version: 0.0
os: linux
files:
- source: /
destination: /var/www/html/
overwrite: true
hooks:
BeforeInstall:
- location: scripts/before_install.sh
timeout: 300
runas: root
AfterInstall:
- location: scripts/setup_environment.sh
timeout: 300
runas: root
- location: scripts/after_install.sh
timeout: 900
runas: root
ApplicationStart:
- location: scripts/start_server.sh
timeout: 300
ApplicationStop:
- location: scripts/stop_server.sh
timeout: 300
ValidateService:
- location: scripts/validate_service.sh
timeout: 300
While runas ubuntu getting the below error.
Error code
ScriptFailed
Script name
scripts/setup_environment.sh
Message
Script at specified location: scripts/setup_environment.sh run as user ubuntu failed with exit code 4
LifecycleEvent - AfterInstall
Script - scripts/setup_environment.sh
[stderr]shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory
[stderr]shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory
[stderr]/opt/codedeploy-agent/deployment-root/44d6390b-485e-87ef-b50855bbf251/d-D0RTN7AR5/deployment-archive/scripts/setup_environment.sh: line 4: /var/www/html/.env: Permission denied
[stderr]sed: couldn't open temporary file /var/www/html/scripts/seTwGZAv: Permission denied
If you run it as ubuntu user it will not work due to lack of permissions which you are experiencing:
couldn't open temporary file /var/www/html/scripts/seTwGZAv: Permission denied
The reason is that /var/www/html/ is not accessible by ubuntu user. To make it work you would have to change its default permissions which is a bad practice.
Some things have to be executed as root, unless you want to start changing default configurations and permission model of ubuntu operating system.
As appspec.yml file and scripts are managed by you, there is not any security issue while running our script as root. What you'll write is what you'll get.
While using any non root user it is important to provide all the required permissions to that user. In most of the cases you will have to use sudo before each command and make sure your user is added to sudoers.
You need to make sure that
Your git is secure from any unauthorized changes.
CodeDeploy is only accessible to the trusted resources.
If these two things are checked, there's no way any anomalous command can run on your system

AWS Codedeploy No such file or directory

I have two problems deploying via AWS CodeDeploy.
I'm trying to deploy CodeCommit's code to an EC2 ubuntu instance.
At appspec.yml
version: 0.0
os: linux
files:
- source: /
destination: /home/ubuntu
hooks:
ApplicationStart:
- location: scripts/ApplicationStart.sh
timeout: 300
runas: ubuntu
There are several config files that I need to place at the right place in the application before starting pm2. I also assume since I set runas in appspec.yml as ubuntu, the bash script will at /home/ubuntu.
The my /home/ubuntu has
config/ backend/ frontend/
Looks like Code deploy won't overwrite the previous deployment so if I have backend/ and frontend/ folder at the directory, it will fail at Install stage.
In the ApplicationStart.sh
#!bin/bash
sudo cp config/config1.json backend/config/config1.json
sudo cp config/config2.json backend/config/environments/development/config2.json
sudo cp config/config3.json frontend/config3.json
sudo pm2 kill
cd backend
sudo npm install
sudo pm2 start "strapi start" --name backend
cd ../frontend
sudo npm install
sudo pm2 start "npm start" --name frontend
While the ApplicationStart stage, it gives me the following error.
LifecycleEvent - ApplicationStart
Script - scripts/ApplicationStart.sh
[stderr]bash: /opt/codedeploy-agent/path/to/deployment/scripts/ApplicationStart.sh: bin/bash:
bad interpreter: No such file or directory
I run the same bash file at the /home/ubuntu. It works fine.
Question 1.
- how to run BeforeInstall.sh without the error? Is there the path problems or something else I try to do but I am not supposed to do?
Question 2.
- How can I let code deploy to overwrite the previous deployment when there are already application folders in the directory (/home/ubuntu)?
- Do I manually delete the directory at BeforeInstall stage?
You're missing a slash before bin/bash in #!bin/bash.
It should be #!/bin/bash.

gunicorn systemd startup not working with django inside a vagrant box

I translated this tutorial into a chef recipe. And so far everything apart from starting gunicorn (the right way) seems to be working.
For example when I shut down the machine after the initial setup and provisioning via vagrant halt and then start it up again with vagrant up - I always get an 502 Bad Gateway error.
Then I have to ssh into the box and run these commands manualy
sudo systemctl daemon-reload
sudo systemctl restart gunicorn
After that everything is working again.
What I dont understand is when I run sudo systemctl status gunicorn before I reload the daemon and restart gunicorn - it tells me that gunicorn is running.
Here is my gunicorn.service file contents that get written to /etc/systemd/sytem/gunicorn.service
[Unit]
Description=gunicorn daemon
After=network.target
[Service]
User=ubuntu
Group=www-data
WorkingDirectory=/vagrant_data
ExecStart=/vagrant_data/myprojectenv/bin/gunicorn --workers 5 --bind unix:/home/ubuntu/run/myproject.sock myproject.wsgi:application --reload
[Install]
WantedBy=multi-user.target
My projects folder structure is:
/home/ubuntu/myproject ls
manage.py myproject myprojectenv
/home/ubuntu/run ls
myproject.sock
I symlinked the myproject folder to vagrant_data which is setup to be the vm.synced_folder in my Vagrantfile.
this is all running on a ubuntu/xenial64 vagrant box.
UPDATE:
include_recipe 'locale'
include_recipe 'apt'
execute 'install requirements' do
command 'sudo apt-get install -y python3-pip python3-dev libpq-dev postgresql postgresql-contrib nginx'
not_if ('sudo dpkg -l | grep postgresql')
end
bash 'setup database and user' do
user 'postgres'
code <<-EOF
echo "CREATE DATABASE #{node['dbname']};" | psql
echo "CREATE USER #{node['dbuser']} WITH PASSWORD '#{node['dbpass']}';" | psql
echo "ALTER ROLE #{node['dbuser']} SET client_encoding TO 'utf8';" | psql
echo "ALTER ROLE #{node['dbuser']} SET default_transaction_isolation TO 'read committed';" | psql
echo "ALTER ROLE #{node['dbuser']} SET timezone TO 'UTC';" | psql
echo "GRANT ALL PRIVILEGES ON DATABASE #{node['dbname']} TO #{node['dbuser']};" | psql
EOF
not_if { `sudo -u postgres psql -tAc \"SELECT * FROM pg_database WHERE datname='#{node['dbname']}'\" | wc -l`.chomp == "1" }
end
execute 'install virtualenv' do
command 'sudo pip3 install virtualenv'
not_if ('sudo pip3 list | grep virtualenv')
end
link "/home/ubuntu/#{node['projectDir']}" do
to '/vagrant_data'
owner 'ubuntu'
group 'www-data'
end
directory '/home/ubuntu/run' do
owner 'ubuntu'
group 'www-data'
action :create
end
bash 'configure and install django' do
code <<-EOF
cd /home/ubuntu/#{node['projectDir']}
virtualenv myprojectenv
source myprojectenv/bin/activate
pip install django gunicorn psycopg2
django-admin.py startproject #{node['projectDir']} .
deactivate
EOF
not_if { ::File::exists?("/home/ubuntu/#{node['projectDir']}/#{node['projectDir']}")}
end
###############
# Note : In development set workers to 1 which will reload the code after each request
# in production set it to cores x 2 + 1 ... which would mostly result in 5 workers
##########
template '/etc/systemd/system/gunicorn.service' do
source 'gunicorn.erb'
owner 'root'
group 'root'
end
template '/etc/nginx/sites-available/myproject' do
source 'test.erb'
owner 'www-data'
group 'www-data'
end
execute 'link to sites-enabled' do
command 'sudo ln -s /etc/nginx/sites-available/myproject /etc/nginx/sites-enabled'
not_if { ::File.symlink?('/etc/nginx/sites-enabled/myproject')}
end
execute 'remove default host' do
command 'sudo rm /etc/nginx/sites-enabled/default'
only_if { ::File.exists?('/etc/nginx/sites-enabled/default') }
end
bash 'enable gunicorn' do
code <<-EOF
sudo systemctl daemon-reload
sudo systemctl start gunicorn
sudo systemctl enable gunicorn
EOF
#not_if { ::File.exists?('/home/ubuntu/run/myproject.sock')}
end
execute 'test nginx' do
command 'sudo nginx -t'
end
execute 'restart nginx' do
command 'sudo service nginx restart'
end
Does anyone know what I am doing wrong ?
UPDATE:
Still not working - after trying almost everything google had to offer.
Now I switched to a kind of workaround with the vagrant-triggers plugin and defined the needed commands for gunicorn in Vagrantfile.
config.trigger.after :up do
run_remote "sudo systemctl daemon-reload"
run_remote "sudo systemctl restart gunicorn"
end
That way I don't have to call vagrant up --provision every time I turn on the machine.
But still I would really like to know how to get that thing started the right way.