OpsWorks Django Deployment - django

I am trying to deploy a Django application using AWS OpsWorks. I'm brand spanking new to any sort of DevOps work, so I'm having considerable difficulties.
I am trying to use this cookbook to automate my deployments. I need Python3.4, so I modified a few things in the cookbook. Right now during the deploy hook, I am getting an error from the following code:
# install requirements
requirements = Helpers.django_setting(deploy, 'requirements', node)
if requirements
Chef::Log.info("Installing using requirements file: #{requirements}")
pip_cmd = ::File.join(deploy["venv"], 'bin', 'pip')
execute "#{pip_cmd} install --source=#{Dir.tmpdir} -r #{::File.join(deploy[:deploy_to], 'current', requirements)}" do
cwd ::File.join(deploy[:deploy_to], 'current')
user deploy[:user]
group deploy[:group]
environment 'HOME' => ::File.join(deploy[:deploy_to], 'shared')
end
else
Chef::Log.debug("No requirements file found")
end
The error reports:
STDERR: /opt/aws/opsworks/releases/20141216163306_33300020141216163306/vendor/bundle/ruby/2.0.0/gems/mixlib-shellout-1.4.0/lib/mixlib/shellout/unix.rb:147:in `chdir': No such file or directory - /srv/www/django/current (Errno::ENOENT)
I get that this code is trying to install requirements from my requirements.txt file, but what is up with the tmp directory and the current directory? Clearly there is no current directory created when I do my deployment. What is the file structure generally like for code being pulled into OpsWorks from a deploy? Moreover, how might I go about finding a fix for this error?
I've been reading through documentation on Chef, OpsWorks, KitchenCI, Berksfile, and other technologies for days just feeling swamped by everything in the world of DevOps. I just want to get my application running!
EDIT
Custom json is:
{
"deploy" : {
"django" : {
"django_settings_template" : null,
"django_settings_file" : "settings.py",
"django_collect_static" : "true",
"python_major_version" : "3.4",
"venv_options" : "--python=$(which python3.4) --no-site-packages",
"custom_type" : "django"
}
}
}

If there is no current directory it's because it wasn't created during the deploy. Your script actually does reference that directory.
This block of code below is where you are having an error. If you refer to the documentation on the execute resource block https://docs.chef.io/resource_execute.html you'll see that if you don't provide a command to the execute resource then the name of the block will be what command is executed. So in your case you are generating the path to your pip command via Ruby's file.join so pip_cmd should be something like /usr/bin/pip and then the name of the block is the command which should be something like
#Mind you I'm not sure if the /tmp dir is correct, but if not it might also be the chef tmp directory
/usr/bin/pip install --source=/tmp -r /srv/www/django/current/requirements
Now in the resource you have the cwd attribute, which is the "current working directory" or the directory in which the command is executed. So when this command is executed it's being executed in /srv/www/django/current
pip_cmd = ::File.join(deploy["venv"], 'bin', 'pip')
execute "#{pip_cmd} install --source=#{Dir.tmpdir} -r #{::File.join(deploy[:deploy_to], 'current', requirements)}" do
cwd ::File.join(deploy[:deploy_to], 'current')
user deploy[:user]
group deploy[:group]
environment 'HOME' => ::File.join(deploy[:deploy_to], 'shared')
end
Now without knowing a bit more about your deployment I can't really tell you how to fix it without knowing a bit more about your deployment. Can you post your actual cookbook code so we can see how you are using this cookbook to deploy your code?
{
'deploy': {
'django': {
'repository': 'Your github URL',
'revision': 'your revision number'
},
}
}

Related

Cannot chmod file on Openshift online v3 : Operation not permitted

I am migrating a Django application from Openshift v2 to v3 (In case you don't know, RedHat is shutting down v2 on September 30th, see: https://blog.openshift.com/migrate-to-v3-v2-eol/)
So, I am following this blog post to help me: https://blog.openshift.com/migrating-django-applications-openshift-3/ . I am new to all these Docker / Kubernetes concepts the new version is build upon.
I was able to make some progress : I managed to get a successful build of my app. Yet it crashes at deployment time:
---> Running application from script (app.sh) ...
/usr/libexec/s2i/run: line 42: /opt/app-root/src/app.sh: Permission denied
Indeed, app.sh has lost its x permission. I log into the failing container as debug and see it:
> oc debug dc/<my app>
> (app-root)sh-4.2$ ls -l /opt/app-root/src/app.sh
-rw-rw-r--. 1 default root 127 Sep 6 21:20 /opt/app-root/src/app.sh
The blog posts states "Ensure that the app.sh file is executable by running chmod +x app.sh.", which I did on my local repo. Whatever, I want to do it again directly in the pod, but it doesn't work:
(app-root)sh-4.2$ chmod +x /opt/app-root/src/app.sh
chmod: changing permissions of ‘/opt/app-root/src/app.sh’: Operation not permitted
So, how can I set the x permission to app.sh ? Thank you
Without looking into more details, any S2I builder image will gladly use your custom supplied run script to start the application in an alternative way.
Create .s2i/bin/ (mind the dot) in your source code directory, place the run script into it and rebuild the app in OpenShift - it will automatically use your custom run script upon deployment.
This is the preferred way of starting applications using custom commands in OpenShift.
Regarding your immediate problem, there is a very simple reason why you can not change the permissions of the script: you were trying to modify the permissions in the deployed pod, and not the builder pod. Deployed pods run using different UIDs, usually somewhere in the range of 100000000, and definitely do not match the file ownership as generated by the build. Hence permission denied.
The root cause of your problem (app.sh losing executable permissions) must be in the way the build process installs those files, and indeed looking at the /usr/libexec/s2i/assemble script in the base image does seem to reveal the culprit. The last two lines are:
# set permissions for any installed artifacts
fix-permissions /opt/app-root
If you wanted to change this part of the build instead of using a custom run script, I suggest you then create .s2i/bin/assemble in your project's source code and make it look sort of like this:
#!/bin/bash
echo "Running stock build:"
${STI_SCRIPTS_PATH}/assemble
echo "Fixing the mess:"
chmod 755 /opt/app-root/src/app.sh
This will fix whatever the stock build process does to file permissions, and will do it using the same UID as the rest of the build, so file ownership shouldn't be an issue.
as I stumbled upon this issue myself I've found a way to resolve it.
You have to make your file app.sh executable and push it in your repo as such.
If git does not track this modification as it did for me, you have to use: git update-index --chmod=+x app.sh for it to work.

Laravel showing error from /var/app/ondeck after deployment aws elasticbeanstalk

After having successfully deployed my laravel 5.4 app using AWS Elasticbeanstalk, Laravel throws InvalidArgumentException in FileViewFinder exception saying View [index] not found when i access my root route.
Here is a snippet of the stack trace.
in FileViewFinder.php line 137
at FileViewFinder->findInPaths('index', array('/var/app/ondeck/resources/views')) in FileViewFinder.php line 79
at FileViewFinder->find('index') in Factory.php line 128
at Factory->make('index', array(), array()) in helpers.php line 914
at view('index') in HomeController.php line 24
As you can see, Laravel it trying to find the view from /var/app/ondeck which as far as i know is the temporary location during deployment not after deployment. I've search everywhere and I can't seem to find a similar issue.
Any help on it will be appreciated.
After battling with it, I managed to use a post deployment script /opt/elasticbeanstalk/hooks/appdeploy/post/artisan_clear_cache.sh to run php artisan config:cache. Laravel caches the view path to point to ondeck directory so after deployment the cache needs to be cleared. Unfortunately this method is not documented. So I worked around it this way.
I had to create the directory that will contain the post deploy hook because it didnt exists by default. Running mkdir /opt/elasticbeanstalk/hooks/appdeploy/post will do.
I had to set the permissions for Laravel to write to storage. Running chmod -R 755 /var/app/current/storage will do.
Finally i had to put artisan command within the .sh.
This way the script will automatically run post deployment and clear the cache.
Note that all of this has to be done in a .config file in the .ebextensions directory within your project root. More details on that here

AWS Code Deploy Error on Before Install Cannot Solve

So I am attempting to setup CodeDeploy for my application and I keep getting an error during the BeforeInstall part of the deployment. Below is the error.
Error Code UnknownError
Script Name
Message No such file or directory - /opt/codedeploy-agent/deployment-root/06100f1b-5495-42d9-bd01-f33d59fb5deb/d-NL5K1THE8/deployment-archive/appspec.yml
Log Tail
I assumed this meant the YAML file was in the wrong place. However it is in the root directory of my revision. I have tried using a simple AppSpec file like so instead of a more complex one.
## YAML Template.
---
version: 0.0
os: linux
files:
- source: /
destination: /home/ubuntu/www
More or less since this is a first deployment I want it to add all files in the revision to the public directory on the web server.
I am tearing my hair out over this and I feel it is a simple issue. I have the IAM policies and roles correct and I have CodeDeploy setup and running on my instance I am trying to deploy to.
It seems to think you had a successful deploy at some point.
Go into /opt/codedeploy-agent/deployment-root/deployment-instructions/ and delete all the files in there. Then it won't look for this last deploy.
I just had this SAME problem and I figured it out! Make sure your AppSpec file has the right EXTENSION! I was using yaml and not yml, now everything works perfectly.
I made it work like this:
I had a couple of failed deployments for various reasons.
The thing is that CD keeps in the EC2 instance and in the path /opt/codedeploy-agent/deployment-root/​ a folder named by the ID of the failed deployment [a very long alphanumeric sting] .
Delete this folder and create a new deployment [from the aws UI console] and redeploy the application. This way the appspec.yml file that is in the wrong place will be deleted.
It should now succeed.
Extra Notice:
CD does not rewrite files [that have not been created by it's specific deployment]
CodeDeploy does not deploy in a folder that there is already code[files] as it does not want to interfere with different CD deployments and/or other CI/CD tools [like Jenkins].
It only deploys in a path that has already deploy code with the specific deployment.
You can empty the folder where your deployment want to happen and redeploy your code via CD.
When you login to the host, do you see the appspec.yml file in the directory there? If not are you positive it has been checked in with the rest of your deployed code?
Just encountered this issue too. In my case, the revision zip file extracts into a directory when deployed. Because of that /opt/codedeploy-agent/deployment-root/xxx/xxx/deployment-archive contains the parent directory of my revision files (instead of the actual revision files).
The key is to compress your revision without the parent directory. In mac terminal,
cd your-app-directory-containing-appspec
zip -r app.zip .

Elastic Beanstalk and Cron tasks not sure if its working with Yii

I am having a problem with getting a simple cron task set up on Elastic Beanstalk. I have found some of the other questions on here useful, but i still can't seem to get the cron to execute. I am unsure if it is an AWS issue, or if the script itself is not executing. The script is set up inside YII as a Console Command. I am not finding any PHP errors, and the ec2 instance is loaded without errors. Here is what i have done so far:
I have created a folder on the root of my application called .ebextensions.
Within that folder i have created a configuration file with the contents
# Installing dos2unix in case files are edited on windows PC
packages:
yum:
dos2unix: []
container_commands:
01-command:
command: dos2unix -k cron_setup.sh
02-command:
command: chmod 700 cron_setup.sh
03-command:
command: "cat .ebextensions/cron_task.txt > /etc/cron.d/cron_task && chmod 644 /etc/cron.d/cron_task"
# leader_only prevents problems when EB auto-scales
leader_only: true
the file cron_task.txt exists inside the .ebextensions folder with the contents
# The newline at the end of this file is extremely important. Cron won't run without it.
* * * * * /bin/php /var/www/html/crons.php test > /dev/null
Crons.php is a file at the root of the application that includes the Yii framework
defined('YII_DEBUG') or define('YII_DEBUG',true);
// including Yii
require_once(dirname(__FILE__).$yii.'/yii.php');
// we'll use a separate config file
$configFile=dirname(__FILE__).'/protected/config/cron.php';
// creating and running console application
Yii::createConsoleApplication($configFile)->run();
the config/cron.php file is a setup file for the framework, includes database connection and model inclusions, etc
and the cron script being referenced in the cron_task.txt file is a console command that looks like this
class TestCommand extends CConsoleCommand {
public function run($args) {
$message = new Crontasks();
$message->timestamp = date("Y-m-d H:i:s");
$message->message = "test";
$message->insert();
}
}
here i am just trying to get a record into the database to prove the cron was executed successfully. And i can't seem to get a record added.
The problem is, i don't know where this is failing. I am not getting any instance errors. And i took a snapshot log and cant seem to find any relevant errors in there either. Should php errors be logged here? OR do i have to set it up myself to log errors? The problem, i am also having trouble getting into ec2 via SSH. I am getting a permission denied (public key) error!! Even though i have set up the security group/key pair and using the correct public DNS for the instance!
If anyone can see anything obvious is what im doing wrong here, please let me know! Otherwise could you give any advice on where to look for any errors that might be preventing this cron task to execute? Many thanks!!

Why doesn't my custom recipes run on AWS OpsWorks?

I've created a GitHub repo for my simple custom recipe:
my-cookbook/
|- recipes/
|- appsetup.rb
I've added the repo to Custom Chef Recipes as https://github.com/my-github-user/my-github-repo.git
I've added my-cookbook::appsetup to the Setup "cycle".
I know it's executed, because it fails to load if I mess up the syntax.
This is my appsetup.rb:
node[:deploy].each do |app_name, deploy|
script "install_composer" do
interpreter "bash"
user "root"
cwd "#{deploy[:deploy_to]}/current"
code "curl -sS https://getcomposer.org/installer | php && php composer.phar install --no-dev"
end
end
When I log into the instance by SSH with the ubuntu user, composer isn't installed.
I've also tried the following to no avail (A nodejs install):
node[:deploy].each do |app_name, deploy|
execute "installing node" do
command "add-apt-repository --yes ppa:chris-lea/node.js && apt-get update && sudo apt-get install python-software-properties python g++ make nodejs"
end
end
Node doesn't get installed, and there are no errors in the log. The only references to the cookbook in the log just says:
[2014-03-31T13:26:04+00:00] INFO: OpsWorks Custom Run List: ["opsworks_initial_setup", "ssh_host_keys", "ssh_users", "mysql::client", "dependencies", "ebs", "opsworks_ganglia::client", "opsworks_stack_state_sync", "mod_php5_apache2", "my-cookbook::appsetup", "deploy::default", "deploy::php", "test_suite", "opsworks_cleanup"]
...
2014-03-31T13:26:04+00:00] INFO: New Run List expands to ["opsworks_initial_setup", "ssh_host_keys", "ssh_users", "mysql::client", "dependencies", "ebs", "opsworks_ganglia::client", "opsworks_stack_state_sync", "mod_php5_apache2", "my-cookbook::appsetup", "deploy::default", "deploy::php", "test_suite", "opsworks_cleanup"]
...
[2014-03-31T13:26:05+00:00] DEBUG: Loading Recipe my-cookbook::appsetup via include_recipe
[2014-03-31T13:26:05+00:00] DEBUG: Found recipe appsetup in cookbook my-cookbook
Am I missing some critical step somewhere? The recipe is clearly recognized and loaded, but doesn't seem to be executed.
(The following are fictitious names: my-github-user, my-github-repo, my-cookbook)
I know you've abandoned the cookbook but I'm almost 100% sure it's because you don't have a metadata.rb file in the root of your cookbook.
Your cookbook name should not contain a dash. I had the same problem, replacing by '_' solved it for me.
If those commands are failing silently, it could be that your use of && is obscuring a failure.
As for add-apt-repository, that is an interactive command. Try using the "--yes" option to answer yes by default, making it no longer interactive.
If you do not execute your command successfully, you will not find the files in the current directory. Check inside the last release folder to see if it had been put there.
It maybe prudent to check if you got the right directory etc setup by changing the CWD to : /tmp