How to start web server in aws cloudformation? - amazon-web-services

I have installed odoo webserver using cloudformation. But need to start services manually. How can I start my odoo webserver using cloudformaton?
I tried calling the script which starts odoo webserver by passing following command through Userdata.
"UserData":{ "Fn::Base64" : "#!/bin/bash sudo /etc/init.d/odoo-server start "}
But received following error
/bin/bash: sudo /etc/init.d/odoo-server start : No such file or directory

it looks like the first line you posted is commented out with the #:
"UserData":{ "Fn::Base64" : "#!/bin/bash sudo /etc/init.d/odoo-server start "}
It would also help if you posted the full UserData section so we can see what commands are run before that.
What ami are you using? what other resources are spun up in your template? the more info the better.

How did you install Odoo using CloudFormation? Can you share the CloudFormation template? Without that information, it'll be difficult to help you, but I'll still try pointing you in the right direction.
You don't need sudo in a UserData script since that script is always executed with sudo behind the scenes.
Look at the contents of /var/log/cloud-init-output.log on the webserver. It'll contain the console output for your UserData script execution.

Related

Unable to execute a step on a running EMR

I have an EMR cluster 5.28.1 running in AWS but I forgot to install from python libraries as part of the bootstrap action. Now that the cluster is running, I was simply attempting to add a step via the EMR console. Here are my settings
JAR: s3://us-east-1.elasticmapreduce/libs/script-runner/script-runner.jar
Main class: None
Arguments: s3://xxxx/install_python_libraries.sh
Unfortunately, I get the following error.
Cannot run program "s3://xxxxx/install_python_libraries.sh" (in directory "."): error=2, No such file or directory
I am not sure what I am doing wrong. The shell script looks like this.
#!/bin/bash -xe
# Non-standard and non-Amazon Machine Image Python modules:
sudo pip-3.6 install boto3
sudo pip-3.6 install xmltodict
I also tried this by simply using 'command-runner.jar' but I get the same error. Can you please help me figure out the problem so I do this via the console? I would like to install the libraries on all nodes - master and core.
Thanks
The issue is the xxx.sh files EOL/carriage return type.
In other words, if it is Windows ("\r\n") then it will not work and return the ./ file not found error.
Convert it to unix type ("\n") using something like notepad++ and it will run fine.
(In notepad++ edit>EOL Conversion>Unix(LF) hit save and try again)

Is it possible to run an Ansible playbook from a Chef AWS/Opworks cookbook?

I try to figure out if it's possible to create a Chef cookbook that ssh into an Ansible server and run some Ansible cookbook from AWS Opworks on the current node
I think of a script that I can put in a execute like this :
define :foobar_magento2_deploy do
release_path = node[:app_release_path]
execute 'Ansible playbook' do
command "ssh -i key ansible-server 'ansible-playbook arg1 arg2'"
end
end
Do you think it's possible ? Is there some caveats ? Hints ?
Edit from #coderanger answer:
define :foobar_magento2_deploy do
release_path = node[:app_release_path]
execute 'Ansible playbook' do
command "git clone ansible-playbook"
command "cd ansible-playbook"
command "ansible-playbook -l localhost playbook.yml"
end
end
So a couple of things:
OpsWorks Stacks is dangerously out of date and using it should be considered highly suspect.
I don't actually recognize that define block thing in there, maybe that's an older OpsWorks syntax?
You can definitely run an Ansible playbook from Chef code, but I would probably go a little simpler than you have there. Probably just run ansible-playbook locally and aim it at localhost.

AWS EMR bootstrap action as sudo

I need to update /etc/hosts for all instances in my EMR cluster (EMR AMI 4.3).
The whole script is nothing more than:
#!/bin/bash
echo -e 'ip1 uri1' >> /etc/hosts
echo -e 'ip2 uri2' >> /etc/hosts
...
This script needs to run as sudo or it fails.
From here: https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-plan-bootstrap.html#bootstrapUses
Bootstrap actions execute as the Hadoop user by default. You can execute a bootstrap action with root privileges by using sudo.
Great news... but I can't figure out how to do this, and I can't find an example.
I've tried a bunch of things... including...
running as Hadoop and adding 'sudo' to each of the 'echo' statements in the script
using a shell script to copy and chmod the above ('echo' statements with no 'sudo') and running local copy using run-if bootstrap that calls 1=1 sudo bash /home/hadoop/myDir/myScript.sh
hard coding the whole script as a one-liner into a run-if bootstrap action
I consistently get:
On the master instance (i-xxx), bootstrap action 2 returned a non-zero return code
If i check the logs for the "Setup hadoop debugging" step, there's nothing there.
From here: https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-overview.html#emr-overview-cluster-lifecycle
summary emr setup (in order):
provisions ec2 instances
runs bootstrap actions
installs native applications... like hadoop, spark, etc.
So it seems like there's some risk that since I'm mucking around as user Hadoop before hadoop is installed, I could be messing something up there, but I can't imagine what.
I think it must be that my script isn't running as 'sudo' and it's failing to update /etc/hosts.
My question... how can I use bootstrap actions (or something else) on EMR to run a simple shell script as sudo? ...specifically to update /etc/hosts?
I've not had problems using sudo from within a shell script run as an EMR bootstrap action, so it should work. You can test that it works with a simple script that simply does "sudo ls /root".
Your script is trying to append to /etc/hosts by redirecting stdout with:
sudo echo -e 'ip1 uri1' >> /etc/hosts
The problem here is that while the echo is run with sudo, the redirection (>>) is not. It's run by the underlying hadoop user, who does not have permission to write to /etc/hosts. The fix is:
sudo sh -c 'echo -e "ip1 uri1" >> /etc/hosts'
This runs the entire command, including the stdout redirection, in a shell with sudo.

AWS EMR script-runner access error

I'm running emr-5.12.0, with Amazon 2.8.3, Hive 2.3.2, Hue 4.1.0, Livy 0.4.0, Spark 2.2.1 and Zeppelin 0.7.3 on 1 m4.large as my master node and 1 m4.large as core node.
I am trying to execute a bootstrap action that configures some parts of the cluster. One of these includes the line:
sudo sed -i '/zeppelin.pyspark.python/c\ \"zepplin.pyspark.python\" : \"python3\",' /etc/alternatives/zeppelin-conf/interpreter.json
It makes sure that the Zeppelin uses python3.4 instead of python2.7. It works fine if I execute this in the terminal after SSH'ing to the master node, but it fails when I submit it as a Custom JAR step on the AWS Web interface. I get the following error:
ed: can't read /etc/alternatives/zeppelin-conf/interpreter.json
: No such file or directory
Command exiting with ret '2'
The same thing happens if I use
sudo sed -i '/zeppelin.pyspark.python/c\ \"zepplin.pyspark.python\" : \"python3\",' /etc/zeppelin-conf/interpreter.json
Obviously I could just change it from the Zeppelin UI, but I would like to include it in the bootstrap action.
Thanks!
It turns out that a bootstrap action submitted throug the AWS EMR web interface is submitted as a regular EMR step, so it's only run on the master node. This can be seen if you click the 'AWS CLI export' in the cluster web interface. The intended bootstrap action is listed as a regular step.
Using the command line to launch a cluster with a bootstrap action bypasses this problem, so I've just used that.
Edit: Looking back at the web interface, it's pretty clear that I was adding regular steps instead of bootstrap actions. My bad!

Limiting Code Deploy revisions with max_revisions value is not working

I am attempting to limit the quantity of successful code deploy revisions that are preserved on the EC2 instances by editing the codedeployagent.yml file’s max_revisions value. I have currently set the value to :max_revisions: 2.
I believe that the issue I am having is due to the method that I am setting the file value. I am attempting to set the value by deploying it with the code deploy package. To do this I have created a custom codedeployagent.yml file locally at the following location:
etc/codedeploy-agent/conf/codedeployagent.yml
In my appspec.yml file I am specifying the installation location of this file by the following lines:
- source: etc/codedeploy-agent/conf/codedeployagent.yml
destination: /etc/codedeploy-agent/conf
I have found that this errors out when I attempt to deploy due to the script already being in place. To work around this, I have added a script that hooks on BeforeInstall with my appspec.yml that will remove the script prior to installing the package:
#!/bin/bash
sudo rm /etc/codedeploy-agent/conf/codedeployagent.yml
Okay, so after this I have ssh’d into the server and sure enough, the :max_revisions: 2 value is set as expected. Unfortunately, in practice I am seeing many more revisions than just two being preserved on the ec2 instances.
So, to go back to the beginning of my question here… Clearly this workaround is not the best way to update the codedeployagent.yml file. I should add that I am deploying to an auto scaling group, so this needs to be a solution that can live in the deployment scripts or cloud formation templates rather than just logging in and hardcoding the value. With all this info, what am I missing here? How can I properly limit the revisions? Thanks.
Have you restart the agent after updating the config file? Any new configurations won't work until you restart the agent.
You may try one of below approaches.
Take an AMI of an instance where you already modified max_revisions to 2 and update ASG's Launch configuration with this AMI, so that scale out instance will also have this config.
Add this config in your userdata section while creating launch configuration
Command to add in userdata section
"UserData" : { "Fn::Base64" : { "Fn::Join" : ["", [
"#!/bin/bash -xe\n",
"# Delete last line and add new line \n",
"sed '$ d' /etc/codedeploy-agent/conf/codedeployagent.yml > /etc/codedeploy-agent/conf/temp.yml\n",
"echo ':max_revisions: 2' >> /etc/codedeploy-agent/conf/temp.yml\n",
"rm -f /etc/codedeploy-agent/conf/codedeployagent.yml\n",
"mv /etc/codedeploy-agent/conf/temp.yml /etc/codedeploy-agent/conf/codedeployagent.yml\n",
"service codedeploy-agent restart\n"
]]}}
As per reference, max_revisions applies for applications per deployment group. So it keeps only 2 revisions under /opt/codedeploy-agent/deployment-root/<deployment_group_id>/. If ASG is associated with multiple applications, codedeploy will store 2 revisions of each application in its deployment_group_id directory.